Verify GPU access before running Ollama in Docker
A healthy container can still be running inference on the CPU. Check GPU access independently from the web interface and model.
In brief
- Ollama documents separate paths for NVIDIA, ROCm and Vulkan.
- Verify the host driver first, then test device access inside the container and load a small model.
What to check
Ollama documents separate paths for NVIDIA, ROCm and Vulkan. On NVIDIA hosts, configure the Container Toolkit and explicitly expose the GPU to the container. Keep downloaded weights in a persistent volume so that replacing the container does not require downloading them again. Publish the HTTP port on loopback if only a local application needs access.
Put it to work
Verify the host driver first, then test device access inside the container and load a small model. Check the running model’s processor placement while it is answering. Pin the image version and record the host driver version. If a later upgrade causes a regression, those two details help identify whether the problem follows the image or the host.
Verify the host and container separately
On an NVIDIA host, first verify that the host driver recognizes the card. Then check that the container runtime can make the same device available. Use a model small enough to fit comfortably in memory and check its placement after the first successful request. A responsive HTTP endpoint establishes that the service started. The running-model record and device telemetry establish whether the GPU is doing the work.
Make an upgrade reversible
Keep the model volume separate from the application container. Write down its mount point, image tag, GPU selection and port binding before replacing anything. Test the replacement with a short prompt and verify that the same model digest appears without another download. Preserve the previous image reference until the upgrade passes. If the server responds but placement changes, compare runtime device access before changing inference parameters.
Sources and verification
Based on primary documentation reviewed on 3 November 2026. Practical test procedures are editorial guidance. Performance depends on the model, hardware and configuration.
- Docker Ollama · Primary document
- Hardware support Ollama · Primary document
- List running models Ollama · Primary document
Prepared with AI assistance from the linked sources. This is not a claim of independent field reporting or human review. How we publish · Report a correction
Keep up with homelab
Explore the rest of the local Ollama guide series.
RSS requires an email newsletter opt-in.
