You think your model generates tokens slowly because the GPU is busy doing math? It isn't. During decode, your GPU is mostly sitting there, waiting on memory. Every token the model writes, it looks back at every token before it, so it stores the keys and values for each one, the KV cache. Token 1000? Read 1000 entries. Token 10,000? Read 10,000. The cache grows with your context, and it gets re-read from memory on every single token. The math finishes fast. The read doesn't. So DeepSeek asked a different question. Not how do we compute attention faster? How do we store less of it? Multi-head latent attention squeezes those full-size keys and values down into one small shared latent vector. That tiny vector is the only thing cached. When attention actually runs, the model decompresses it back up to full keys and values on the fly. Which sounds backwards. You just added work. That up-projection costs real compute. But decode is starved for bandwidth, not compute. You're spending the resource you have to spare to buy back the one you don't. Deepseq V2 cut its KV cache by 93.3%, and generation throughput went up 5.76 times. Same attention. Same quality. They just stopped storing what they could rebuild. Mistral, GLM5, and Sarvam have all shipped it since.
Machine-transcribed from the published render, so acronyms may be cased as ordinary words.