Set an explicit context length, run a representative task and check what Ollama actually loaded. A model file fitting on your GPU does not prove that your required context and concurrent requests will fit or run fast enough.
If a local model becomes slow during a long conversation, first determine whether context, concurrency or another loaded model changed the memory requirement. Buying a larger card can be appropriate, but an avoidable allocation should not decide that purchase for you.
This is a documentation-based configuration and purchasing guide, reviewed 8 November 2026. Commands are examples to adapt to your installed version and model. We have not run a hardware benchmark or validated a model/GPU performance matrix for this guide.
Start with the task you need to finish
Write down the prompt length, required answer length, acceptable waiting time and number of simultaneous requests. Count system instructions, tool definitions, retrieved documents and conversation history as part of the request. A short test prompt does not exercise a long document or an agent’s accumulated history.
Keep one example that represents a busy session and one that represents normal use. Decide how you will judge the answer before changing settings. Otherwise, a fast answer that discarded necessary information can look like a successful optimization.
| What you observe | First check | What would justify more capacity? |
|---|---|---|
| A longer conversation slows down | Actual context allocation and CPU/GPU split | The context your task needs still cannot meet your timing target |
| One request works, concurrent requests struggle | Parallelism and other loaded models | Required simultaneous work exceeds the usable configuration |
| Model loads but answers are unusably slow | Offloading, device support and actual timing | A verified bottleneck that a different device or deployment addresses |
| A larger context gives no task benefit | Whether the task needs the additional history | A demonstrated improvement worth its memory and operating cost |
Set num_ctx explicitly
Check the installed client version and the model you intend to use:
ollama --version
ollama show llama3.2 --modelfile
Replace llama3.2 with your installed model. Record its tag and digest when available; a mutable tag can change. The Modelfile reference documents PARAMETER num_ctx for context configuration. A larger allocation does not extend the model’s supported capability beyond its own limits.
In an interactive ollama run session, the FAQ documents this setting:
/set parameter num_ctx 8192
For a repeatable request, pass the context in options to the native API. This example performs one local generation if you run it; it requires the named model to be available:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [{"role": "user", "content": "Summarize the supplied notes."}],
"stream": false,
"options": {"num_ctx": 8192}
}'
See the native chat API for its request and usage fields. Set the value in the application making the request too; a client with its own settings can invalidate an assumption based only on how the server was started.
For a standalone server process, the context guide documents a server default using OLLAMA_CONTEXT_LENGTH:
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
Use that form when you manage the server in that terminal. An already running desktop app or system service needs its own environment configuration and restart; starting another server is not a way to reconfigure the existing one. Follow the platform-specific instructions in the FAQ and preserve other existing service settings.
Verify allocation instead of relying on a default
The current context guide describes VRAM-dependent defaults: 4k below 24 GiB, 32k from 24 to below 48 GiB, and 256k at 48 GiB or more. Other official pages still show different generic defaults. Those statements should not be treated as a single promise for every installed release, backend or model.
Use the explicit value above, then run a request and inspect:
ollama ps
On versions exposing it, check CONTEXT for the allocation and PROCESSOR for the CPU/GPU split. Keep the command output with the request settings. If your version lacks the context column, inspect server logs and the documented controls for that release rather than assuming the requested value took effect.
100% GPU describes the reported loading arrangement. It is not a measured throughput result or proof that the application preserved every document. Check the answer and timing separately. The troubleshooting guide explains where to find logs on each platform.
Separate weights, context cache and other memory
A useful memory budget distinguishes model weights, the attention key/value cache and runtime overhead. A download size is only one input. Available device memory can also be reduced by the desktop, other applications and other loaded models.
For conventional attention models, more retained tokens and simultaneous sequences can increase KV memory. Exact requirements depend on architecture, precision, backend and allocation behavior; a universal GB-per-1,000-tokens rule would conceal those differences. Nor does doubling the context imply doubling total model memory, since the weights remain another component.
Use the GPU model directory to explore explicitly labeled estimates for its supported model configurations, or the Apple Silicon directory for unified-memory options. An estimate there is a starting budget, not evidence that this Ollama request will meet a speed target.
If the configuration no longer fits as intended, reduce one variable and repeat the same task. Comparing results is more informative than changing context, quantization and parallelism together: you need to know which trade-off helped and what quality or capacity it cost.
Treat parallelism and cache precision as trade-offs
The FAQ explains that parallel request processing increases context-related memory demand. Begin a diagnostic comparison with one request at a time. Then test the concurrency your workload actually needs; do not infer production capacity from one successful chat.
Cache precision and model-weight precision are separate choices. Ollama documents OLLAMA_KV_CACHE_TYPE values including f16, q8_0 and q4_0, with quantized KV cache depending on Flash Attention. Hardware, backend and model support need checking. Reduced cache precision can affect answers, especially at longer context, so compare representative outputs before retaining a setting.
Do not change a shared server’s global configuration merely to make an isolated example fit. Record the users and workloads that depend on that process, plan the restart, and retest their requirements. A smaller allocation that breaks the required number of sessions is not a complete solution.
Decide whether to tune, upgrade, rent or use an API
Keep the existing machine when the required task works within an explicit configuration and acceptable waiting time. Measure its actual energy use and your maintenance time before turning those into a cost estimate.
Consider more memory or a different machine when a documented task requirement remains blocked after configuration checks. Match that bottleneck to the proposed device. More memory may address capacity while leaving a throughput or software-support problem unresolved. Include whole-system costs and realistic resale value if purchasing hardware.
Consider renting a GPU when you need additional capacity for bursts or a project with uncertain duration. Include setup, downloads, retained storage and time left running. Our Serverless versus Pod guide helps compare the effect of request timing.
Consider a hosted API when its capabilities and service conditions meet the task without the local operating burden. Verify the model, output quality, data requirements and billable token mix. Use the API cost calculator with the route you would actually buy.
For the financial comparison, continue with API, rented GPU or your own hardware?. Treat accepted work as the common denominator. A low electricity bill does not by itself make a slow or unsuitable local setup the lowest-cost way to finish the task.
Keep a configuration record
Before spending, save the installed version, model identifier, explicit context setting, concurrency, cache precision, allocation output, representative request, timing and answer assessment. Include the device and software environment, so another run can be compared meaningfully.
Repeat the busy-session example after upgrades or model changes. If the result changes, update the budget using the new evidence. This record is more useful for the next hardware decision than a generic claim that a particular number of model parameters always fits on a particular card.