How to evaluate a local LLM before using it in production

A model can perform well on public benchmarks and still fail at the work we want it to perform. Production workloads include formats, tools, messy data, latency constraints, structured outputs and operational consequences that a benchmark does not represent.

The useful question is therefore not “which model is best?”, but “does this model + runtime + workload combination satisfy the contract we need?”

A useful test starts from an observable contract: allowed inputs, expected output, available tools, time limits, acceptable errors and conditions that must stop the flow.

Without those boundaries, an apparently correct result can become a false authorization.

Quantization, context window, prompt format, tool calling, available memory and runtime libraries can change model behaviour. The same model can therefore produce different outcomes in different environments.

A useful test records configuration, version, input, output, timing and conditions so the result can be repeated and compared.

A successful test demonstrates a capability under specific conditions. It does not prove that the model can operate without limits on different inputs, new data or additional tools.

For sensitive workloads it is useful to separate qualification from authorization: demonstrate the capability first, then authorize a narrow and observable operating boundary.

This approach can make smaller models useful when they satisfy a specific contract, while preventing a large model from receiving critical work simply because it performs well on average.

The goal is not to find the best model in general. It is to know what a model can do, in which environment, with what evidence and within which limits.