Why a model that fits can still run out of VRAM
Model weights are only part of the memory budget. Test the prompt lengths your applications actually need before sizing the GPU.
In brief
- The context window determines how much text the model can work with at once.
- Start with a realistic prompt and a modest context, then repeat with your longest expected input.
What to check
The context window determines how much text the model can work with at once. Ollama documents that increasing context increases memory requirements. On a constrained GPU, a model that runs smoothly on a short prompt can partially offload to the CPU when the context allocation grows. Inspect the processor split and allocated context with ollama ps.
Put it to work
Start with a realistic prompt and a modest context, then repeat with your longest expected input. Record peak memory and time to first token. Leave room for output, the system prompt and any retrieved passages. If the larger setting pushes the workload into CPU offload, reduce context or compare a smaller model on the same task.
Calculate a first memory budget
Separate model weights, runtime buffers, context cache and memory already occupied by other processes. As an illustrative budget, a 16 GiB card with 10 GiB assigned to weights and 2 GiB reserved for runtime and other workloads has about 4 GiB left for context and additional headroom. Those figures are assumed inputs. Measure the actual allocation under load and leave room for fluctuations and competing requests.
Increase context one step at a time
Use a test set with a short conversation, a typical document and a maximum-length document. Raise context in controlled steps and verify the effective setting from the running-model response. Keep the model digest and output budget fixed. Check whether the useful answer improves at each step and watch for CPU placement or a spike in first-token latency. A context setting that fits an idle machine may fail when other users submit prompts at the same time.
Sources and verification
Based on primary documentation reviewed on 3 November 2026. Practical test procedures are editorial guidance. Performance depends on the model, hardware and configuration.
- Context length Ollama · Primary document
- List running models Ollama · Primary document
- Modelfile Reference Ollama · Primary document
Prepared with AI assistance from the linked sources. This is not a claim of independent field reporting or human review. How we publish · Report a correction
Keep up with homelab
Explore the rest of the local Ollama guide series.
RSS requires an email newsletter opt-in.
