Benchmark Ollama with more than tokens per second
Separate model loading, prompt processing and generation to learn where your local inference server actually spends its time.
In brief
- Ollama’s generate response exposes timing and token counts.
- Run both cold and warm trials using the same weights, prompt, context and output budget.
What to check
Ollama’s generate response exposes timing and token counts. Compute generation throughput as eval_count × 1,000,000,000 ÷ eval_duration when the duration is positive. That rate describes generated tokens and excludes other work the request may perform. Record load time and prompt evaluation time separately, and measure elapsed time from the client.
Put it to work
Run both cold and warm trials using the same weights, prompt, context and output budget. For streamed requests, capture the delay before the first visible output and the time to completion. Report the number of runs and the spread, not just the fastest result. For a voice or chat application, those client-side delays often matter more than peak throughput.
Keep the benchmark conditions comparable
Record the model digest, effective context and processor split alongside each result. Fix the test corpus and output limit, and describe whether the weights were already loaded. Use multiple runs at each setting. Report failed trials separately from successful ones, because a fast result on the runs that fit can conceal a configuration that is unreliable with the full workload.
Read the stages as separate signals
If cold requests are slow and warm requests are acceptable, investigate loading and model residency. If delay rises with prompt length, look at prompt evaluation and effective placement. If generated tokens arrive steadily but the interface waits to display them, inspect client buffering. Compare request elapsed time with server measurements using the same clock units. Keep the benchmark as a repeatable test, so a later model upgrade can be assessed on the same evidence.
Sources and verification
Based on primary documentation reviewed on 3 November 2026. Practical test procedures are editorial guidance. Performance depends on the model, hardware and configuration.
- Generate a response Ollama · Primary document
- List running models Ollama · Primary document
- Context length Ollama · Primary document
Prepared with AI assistance from the linked sources. This is not a claim of independent field reporting or human review. How we publish · Report a correction
Keep up with homelab
Explore the rest of the local Ollama guide series.
RSS requires an email newsletter opt-in.
