Reliability and Small Models

Large models with 2 billion or more parameters (such as Gemma 4 E4B) tend to be reliable in tool calling in agentic systems. But as one tries smaller and smaller models, reliability drops. Sub-1B models can behave in unexpected ways; some additional logic is needed when using them.

In an MCP server, it is often necessary to repair or correct tool calls done by these smaller models. This became more necessary as I started tested a 350B parameter model (LFM 2.5 350M).

I also started using a prompt-checking system:

• The agent reports the last user prompt back as a parameter to the MCP server.

• The MCP server uses code to check things in the original prompt (validating or finding information that the model might have missed).

So in essence we have a double-processing system. The model is supposed to process the user prompt, but then the MCP server does it again just to improve reliability and find things the model might have missed. It is sort of like cheating—but the smallest models are less reliable, and cannot be trusted to find all important details from an input.