Choosing a GPU for Ollama starts with the driver
Check the exact GPU and driver pairing before buying used hardware or assuming a newer card will work with an older system image.
In brief
- Ollama’s hardware list includes the Tesla P40, V100 and RTX 5070, but they belong to different compute capability generations.
- Compare candidates with the same model, quantization, prompt and context.
What to check
Ollama’s hardware list includes the Tesla P40, V100 and RTX 5070, but they belong to different compute capability generations. Check the minimum driver requirement for the exact card and backend. Record the installed driver and verify that the operating system can see the GPU before changing the model or runtime settings.
Put it to work
Compare candidates with the same model, quantization, prompt and context. A card that fits a larger model can still be a poor fit for interactive work if its generation speed or cooling demands miss your target. For a used server accelerator, check chassis airflow and power connector requirements against the manufacturer’s documentation before purchase.
Test the candidate with a fixed workload
Record card model, driver, Ollama version, model digest, quantization and allocated context in a worksheet. Start with a known small local model and one client. While generation is active, compare the reported GPU allocation with the driver’s memory reading. Then repeat with the model and prompt size you actually want. The running-model API gives you a digest and reported VRAM size, so a later test can distinguish a model change from a driver or configuration change.
Check the host before adding a second card
Draw the intended slot and cooling layout. Compare slot spacing, physical clearance, available lanes, connector wiring and airflow with the host and card manuals. A second card changes the chassis heat load and can obstruct intake on an adjacent card. Repeat the original one-card test after installation, then test the two-card workload. Compare end-to-end response time, stability and wall energy. Use a full workload run before making a purchasing decision.
Sources and verification
Based on primary documentation reviewed on 3 November 2026. Practical test procedures are editorial guidance. Performance depends on the model, hardware and configuration.
- Hardware support Ollama · Primary document
- List running models Ollama · Primary document
- Docker Ollama · Primary document
Prepared with AI assistance from the linked sources. This is not a claim of independent field reporting or human review. How we publish · Report a correction
Keep up with homelab
Explore the rest of the local Ollama guide series.
RSS requires an email newsletter opt-in.
