Early research prototype LLM systems Open benchmark

Risk-Routed Heterogeneous KV Memory

An investigation of KV-cache compression as fidelity routing: preserve exact-critical spans in full KV precision and store lower-risk background context with low-bit quantized KV.

Ahmet Rıfat Öztürk Researcher

StatusResearch prototype

Models testedQwen2.5 0.5B / 1.5B

ContextUp to 16K

FidelityFull, 4-bit, 3-bit

Implementation caveat

The current implementation quantizes and dequantizes FP16 tensors as an inference-time proxy. It does not implement packed 3-bit or 4-bit KV storage kernels.

Review the experiment Explore the benchmark

Experiment overview

Fidelity routing instead of only token eviction

Long-context Transformer inference is constrained by KV-cache memory and bandwidth. Many compression methods decide which tokens to retain or apply one compression level uniformly. This experiment tests a heterogeneous policy: exact-critical tokens and a local window remain in full KV precision, while lower-risk background tokens use low-bit quantized KV.

Research question

What memory representation should each token use?

A low-attention token can later become exact-answer critical. Evicting an identifier, date, quoted value, negation, secret marker, or key-value fact can cause exact recall to fail. Uniform quantization retains all tokens but assigns brittle exact spans and filler background the same fidelity.

Exact-critical tokens ± a local window → Full KV. Remaining background tokens → low-bit Quantized KV.

Interpretation

Key findings

Routing can protect exact recall.

On Qwen2.5-0.5B diagnostics, routed low-bit policies often outperform quantization-only baselines at similar effective memory.

3-bit can be too aggressive.

The Qwen2.5-1.5B 8K proxy shows severe degradation under the current 3-bit setup.

Uniform 4-bit is a strong baseline.

The current heuristic router does not consistently beat uniform 4-bit quantization on the larger-model diagnostic.

Proxy memory is not measured packed memory.

Effective KV estimates describe intended packed storage, not the GPU peak memory of the FP16 proxy.

Experimental design

Method at a glance

Full-fidelity spans: Identifiers, dates, code strings, quoted values, negations, secret markers, and key-value facts
Background: Remaining context represented by simulated 3-bit or 4-bit KV
Models: Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct
Contexts: 512 through 16,384 tokens across diagnostic runs
Evaluation: Forced-choice exact-recall cases with accuracy, gold NLL, and margin reporting
Memory metric: Estimated effective packed KV memory, separate from observed GPU proxy memory

Selected diagnostics

Results remain model- and fidelity-dependent

0.5B · up to 2K: 4-bit routed: 0.967 accuracy at 0.264 effective memory; 4-bit quant-only: 0.922 at 0.250
0.5B · up to 16K: 3-bit routed: 0.822 accuracy; 3-bit quant-only: 0.644
0.5B · 16K slice: 3-bit routed: 0.933 accuracy; 3-bit quant-only: 0.733
1.5B · 8K: Uniform 4-bit remains stronger than the current heuristic router in accuracy

Open research

Code, experiment logs, and benchmark data

Proxy implementation, results, paper draft, and experiment notesGitHub · Source repository Public exact-recall benchmark datasetHugging Face · Dataset

Scope

Limitations

The implementation is an inference-time fidelity proxy, not a production packed-KV kernel.
Effective memory estimates are not measured GPU peak-memory reductions.
The current risk router is heuristic and does not consistently outperform strong uniform-quantization baselines.
Results are diagnostic and limited to the tested models, contexts, prompts, and forced-choice protocol.
Larger-model validation shows that conclusions do not transfer uniformly across scale or bit width.

Reuse and citation

Cite the experiment repository

@software{risk_routed_kv_memory_2026,
  title  = {Risk-Routed Heterogeneous KV Memory},
  author = {Öztürk, Ahmet Rıfat and contributors},
  year   = {2026},
  url    = {https://github.com/Ahmet2001/-risk-routed-heterogeneous-kv-memory}
}