Model Management

The previous article went through the Kronk SDK: a batching engine, a conversation cache, and a pile of workarounds behind a very small public API. All of it rested on three assumptions: that the model file was already on disk, that somebody had chosen a context size that fits in your VRAM (the GPU’s own memory), and that if you asked for a second model there’d be room for it.

So rather than tour the code layer by layer, let’s follow the models themselves through the system: how they sit on your disk, how one gets there in the first place, where their configuration comes from, how Kronk works out whether your machine can hold one, and what happens when a request finally asks for one that isn’t loaded.