Continuous batching from first principles

TL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput. If you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes a while for the first word of the response to appear, and then words appear one-by-one on your screen with (hopefully) a regular and fast-paced frequency. That'

[misshiki]

LLM のスループットを最大化するために、①過去トークンを再計算しないKVキャッシュ、②可変長プロンプトに対応する事前入力のチャンク化、③動的スケジューリングによる効率的なバッチ生成の3つを組み合わせる。