Conventional setup
KV cache keeps consuming GPU memory, so larger models and longer context can trigger OOM.
Run larger models with less VRAM
AI SSD offloads LLM KV cache to high-speed storage with the Herdsman local inference engine, so 8GB and 16GB GPUs can run larger models reliably.
Four real hardware platforms / six mainstream models / AI SSD and standard SSD comparison
BENCHMARK / VERIFIED DATA
x405
Peak hot Prefill speedup
Compared with a standard SSD on the same platform and model
x407
KV cache reuse speedup
Repeat inference reuses cache already stored on SSD
16GB
Run 122B models
Validated where standard SSD runs reached OOM
8GB
Run 35B models
Longer context on low-VRAM hardware
How it works
Move memory pressure to fast storage
Every new model token reads the preceding KV cache. That cache grows linearly with context length and is a major consumer of VRAM.
→
AI SSD offload
Movable KV cache is stored on high-speed SSD storage to free scarce GPU memory.
→
Warm cache reuse
Repeated prompts reuse stored KV data to reduce Prefill wait time.
Break the VRAM wall
Turn a VRAM capacity problem into an SSD bandwidth problem so low-VRAM hardware can carry larger models and contexts.
Faster warm inference
On repeat prompts, KV cache is ready on SSD and Prefill avoids most recomputation.
Long-context ready
At 128K and 256K context, standard SSD commonly reaches OOM while AI SSD remains stable.
VRAM constraint
Why AI SSD is needed
Consumer and laptop GPUs commonly have 8-32GB of VRAM, while a 120B model can require over 60GB for weights alone.
Intel 358H 32G (16GB VRAM)：Qwen3.6-35B, gpt-oss:120b, Qwen3.5-122B, and Gemma4:26B reached OOM with standard SSD. All four run with AI SSD.
NVIDIA RTX 5060 (8GB VRAM)：gpt-oss:20b and Qwen3.6-35B reached OOM with standard SSD. AI SSD enables 35B models on 8GB VRAM.
Long-context inference：At 128K and 256K context, KV cache expansion often causes standard SSD runs to reach OOM. AI SSD raises the context ceiling through offload.
In short
On the same hardware, models that standard SSD cannot run can run with AI SSD.
For the same model, AI SSD warm inference can be several to hundreds of times faster.
The measured line charts below show the difference.
Measured results
Benchmark highlights
Every figure below comes from the measured data in this report.
Peak hot Prefill x405
On AMD 395 128G with Gemma4:26B (128K / 64K input), AI SSD hot Prefill TPS is about 405 times standard SSD.
KV reuse acceleration x407
On the same system, warm Prefill reusing SSD KV cache is up to about 407 times faster than cold Prefill.
Peak cold Decode x6.4
On Intel 358H 32G with Gemma4:26B, AI SSD cold Decode TPS is about 6.4 times standard SSD.
Large models run through
122B runs on 16GB VRAM and 35B runs on 8GB VRAM, where standard SSD could not start.
Test platforms
Four hardware platforms and six mainstream models
gpt-oss:20b, Gemma3:27B, Qwen3.6-35B, gpt-oss:120b, Qwen3.5-122B, Gemma4:26B
AMD 395 128G
96GB VRAM / 7 models / Radeon 8060S / 128GB SSD cache
Intel 358H 64G
32GB VRAM / gpt-oss:20b / Intel Ultra X7 358H / 85GB SSD cache
Intel 358H 32G
16GB VRAM / 6 models / standard SSD OOM on 120B and 122B models
NVIDIA RTX 5060
8GB VRAM / 2 models / standard SSD OOM on 20B and 35B models
Explore the complete benchmark
Four line-chart metrics: Cold Prefill TPS, Hot Decode TPS, Cold TTFT, and Hot TTFT. AI SSD uses a cyan solid line and standard SSD a gray dashed line, with OOM clearly marked.
Line charts / platform switcher / maximum measured context for each model