Local LLM Runtime Optimization Notes

Deploying Large Language Models (LLMs) locally is fundamentally a runtime engineering problem, not simply a model selection problem. Many developers focus on downloading a smaller model, expecting lower memory consumption and faster inference. In practice, the model file is only one component of the runtime.

During inference, the execution engine continuously allocates memory for attention caches, prompt processing, execution buffers, scheduling, and hardware-specific optimizations. Consequently, optimizing a local LLM requires understanding how the runtime manages memory, compute resources, and data movement rather than only changing the model itself. Recent work on efficient transformer inference consistently identifies runtime optimizations — particularly KV cache management, attention optimization, and context reduction — as the primary factors affecting latency and memory consumption on edge devices

Runtime Configuration

Runtime configuration defines how the inference engine executes the model after it has been loaded into memory. It does not modify the neural network itself; instead, it controls the execution environment, including memory allocation, scheduling, attention implementation, caching strategy, concurrency, and prompt size.

# Context window (largest RAM multiplier)
num_ctx=4096 # Reduce KV cache memory
# Model randomness
temperature=0.05 # Deterministic outputs
# Model lifetime
keep_alive="2m" # Unload after idle
# Flash Attention
OLLAMA_FLASH_ATTENTION=1 # Reduce attention memory and improve throughput
# KV Cache Quantization
OLLAMA_KV_CACHE_TYPE=q8_0 # ~50% smaller KV cache than f16
# Runtime scheduler
OLLAMA_MAX_LOADED_MODELS=1 # Only one model resident
OLLAMA_NUM_PARALLEL=1 # Disable parallel inference
# RAG
MAX_CONTEXT_TOKENS≈2500 # Context budget
TOP_K=20 # Retrieve
RERANK=4 # Send only best chunks

Model Weights (GGUF)

The GGUF file contains only the trained neural network parameters stored on disk. It is not the total runtime memory. During inference, additional memory is allocated for the KV Cache, execution buffers, Metal unified memory allocations, and prompt processing. Therefore a 2.3 GB model consuming 5–6 GB RAM is expected behavior rather than inefficient implementation. Runtime optimization should target execution memory rather than only model size.

FAQ - Ollama

Runtime Memory =
Model
+ KV Cache
+ Runtime Buffers
+ Metal Buffers
+ Prompt

Context Window (num_ctx)

The context window defines the maximum number of tokens the transformer may process in one inference request. It is not equivalent to answer length; it represents the model’s working memory. Since the KV Cache stores information for every processed token, increasing the context window increases memory consumption almost linearly. For Retrieval-Augmented Generation (RAG), retrieval quality is usually more important than maximizing context length.

2048 ctx → Small KV Cache → Low RAM
4096 ctx → Balanced → Recommended
8192 ctx → Large KV Cache → High RAM

Reducing num_ctx from 8192 → 4096 is generally the largest memory optimization with minimal impact on well-designed RAG systems. oai_citation:1‡Ollama - -

KV Cache

During inference, every generated token must attend to all previous tokens. Recomputing attention repeatedly would make inference prohibitively slow. Instead, transformers cache intermediate attention states called the Key-Value Cache (KV Cache).

Prompt
→ Compute Attention
→ Store KV
→ Reuse KV
→ Generate Next Token

The KV Cache is therefore a runtime execution cache, analogous to Redis for databases or Spark cache for distributed computation. It greatly accelerates inference but becomes the largest memory consumer for long prompts. Modern inference research largely focuses on KV Cache optimization because it dominates memory usage in long-context inference.

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head

Flash Attention

Flash Attention is not a different attention algorithm; it computes the same mathematical result while reorganizing memory access patterns to minimize data movement between GPU memory and compute units. This reduces temporary allocations, lowers memory bandwidth requirements, and improves throughput without affecting model quality.

Standard Attention
Read → Copy → Compute → Copy
Flash Attention
Read → Compute → Discard

Enable Flash Attention whenever supported because it significantly reduces memory consumption for long contexts.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Model Quantization

Model quantization compresses the neural network weights stored on disk and loaded into memory.

FP16
↓
Q8
↓
Q6
↓
Q5
↓
Q4_K_M
↓
Q3_K_M

Lower precision reduces memory requirements but eventually degrades reasoning quality.

Q4_K_M remains one of the best quality-to-memory trade-offs for local inference.

KV Cache Quantization

KV Cache quantization is independent from model quantization. It compresses only the runtime attention cache rather than the model weights.

KV Cache
FP16
↓
Q8_0

Using q8_0 approximately halves KV cache memory with only a very small quality reduction for most workloads, making it one of the highest-impact runtime optimizations currently available.

keep_alive

Loading a model from disk is expensive.

Ollama therefore keeps models resident in memory after inference.

Request
→ Load
→ Generate
→ Keep Loaded

Setting

keep_alive="2m"

keeps the model available for two minutes after the last request, reducing startup latency. Setting

keep_alive=0

unloads the model immediately after inference, minimizing memory usage at the cost of slower subsequent requests. Choose the value according to workload characteristics rather than always maximizing responsiveness.

Retrieval-Augmented Generation (RAG)

The objective of RAG is not to maximize context length but to maximize the relevance of information provided to the model. Large prompts frequently compensate for poor retrieval quality rather than improving reasoning. Current approach:

Retrieve
→ Large Context
→ LLM

Preferred approach:

Retrieve (20)
→ Rerank
→ Best 4
→ LLM

Sending fewer but more relevant chunks reduces inference cost while often improving answer quality because the model spends its attention budget on higher-value information.

Token Budget

Transformers allocate memory in tokens, not characters. Prompt construction should therefore use a fixed token budget rather than arbitrary character limits.

System Prompt ≈ 500
Retrieved Context≈2500
User Query ≈ 200
Answer ≈ 700
Reserve Remaining

This produces predictable runtime memory and avoids unnecessary KV Cache growth.

Persistent Knowledge

Repeatedly processing identical documents is inefficient. Expensive AI tasks should execute once during ingestion and produce reusable artifacts.

Upload
→ Parse
→ Chunk
→ Embeddings
→ Metadata
→ Entities
→ Timeline
→ Store

Subsequent inference consumes structured artifacts rather than raw documents, reducing both latency and repeated computation.

Incremental Processing

Knowledge extraction should be incremental rather than rebuilding the entire corpus.

New Document
→ Parse
→ Chunk
→ Embeddings
→ Update Index
→ Update Artifacts

Only modified knowledge should be recomputed, following the same design principles used in incremental ETL systems.

Runtime Optimization Priority

1. Reduce Context Window
2. Improve Retrieval
3. Enable Flash Attention
4. Enable KV Cache Quantization
5. Limit Loaded Models
6. Cache AI Artifacts
7. Incremental Processing
8. Consider Smaller Models

In modern local LLM systems, runtime architecture has a greater impact on performance than model selection. Recent work on Flash Attention, KV cache compression, and adaptive KV quantization consistently shows that optimizing execution memory yields larger practical gains than simply replacing the model with a smaller one.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness