Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit

Recent work on KV cache quantization, culminating in TurboQuant, has approached the Shannon entropy limit for per-vector compression of transformer key-value caches. We observe that this limit applies to a strictly weaker problem than the one that actually matters: compressing the KV cache as a sequence. The tokens stored in a KV cache are not arbitrary floating-point data -- they are samples from

[takayamaki]

KVキャッシュの各エントリをどう圧縮するかはTurboQuantでほぼシャノン限界であり、これ以上はKVキャッシュがseqenceなのを利用して予測delta符号化するしかない、という話を定式化して主張した論文