Set Ollama Modelfile Parameters for a Repeatable Local Evaluation

A model evaluation is easier to interpret when the base model, prompt, and sampling settings stay fixed. Use a dedicated Ollama variant with explicit parameters for the comparison.

Worked example

Sampling defaults for one evaluation variant

FROM llama3.2:latest
PARAMETER seed 48
PARAMETER temperature 0.1
PARAMETER top_p 0.8
SYSTEM Summarize each test shipment in one sentence without changing the ID or item count.

Steps

  1. Choose an installed base model and record the tag and model ID from ollama list. Put the sample settings in a Modelfile, replacing the tag if needed.
  2. Run ollama create warehouse-eval -f Modelfile. Run the same synthetic prompt twice with warehouse-eval from fresh sessions and save the results.
  3. Change one parameter for a second named variant. Use the same prompt and compare the results using a fixed rubric, such as whether output preserves the shipment identifier and count.

What to check

  • Record the Ollama version and exact input alongside the output. The same seed can help repeat a run with a fixed setup, while a model update or runtime change can alter results.
  • A per-request option can override a model default. Inspect the API request during the comparison if you are calling the variant from an app.

Prepare both local evaluation variants

References