>_TheQuery
← All Articles

Google's TurboQuant Cuts AI Memory 6x

By Addy · March 28, 2026

On March 24, 2026, Google Research published a blog post about a compression algorithm.

The post was understated. There was no product keynote. No model launch. No new subscription tier. Just a research write-up about what happens when you attack one of the least glamorous but most expensive bottlenecks in modern AI systems: memory.

The algorithm is called TurboQuant. And for anyone paying attention to inference economics, it was immediately obvious why the result mattered.

Google says TurboQuant can compress KV cache memory by at least 6x while preserving downstream accuracy across long-context benchmarks, and can quantize the cache down to 3 bits without requiring training or fine-tuning. On H100 GPUs, Google also reports up to an 8x speedup in computing attention logits relative to 32-bit unquantized keys.

That is not a small optimization. It is a direct attack on one of the most expensive assumptions in long-context AI.

The Memory Problem Everyone Eventually Hits

When a transformer model generates text, it does not recompute the entire conversation from scratch on every token. It stores prior keys and values in a running memory structure: the KV cache.

That cache is what makes modern chat feel responsive. It is also what makes long-context serving expensive.

The size of the KV cache grows with both model size and context window length. As context windows stretch from 32K to 128K to 1M tokens, the memory burden shifts away from weights alone and toward the cached activations required to keep attention working efficiently. Google describes this explicitly as a bottleneck between HBM and SRAM on accelerators and across distributed clusters.

That is why TurboQuant matters. If you can shrink cache memory by 6x or more without sacrificing model quality, you do not just save RAM. You change the economics of serving long-context models.

What Google Actually Showed

The strongest claims in the story come directly from Google's own blog post and the TurboQuant paper.

Google evaluated TurboQuant across LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval using open-weight Gemma and Mistral models. In the blog post, Google says TurboQuant achieved perfect downstream results on needle-in-a-haystack tasks while reducing KV memory by at least 6x. The paper similarly reports quality neutrality at 3.5 bits per channel and only marginal degradation at 2.5 bits per channel.

The implementation story is also important. TurboQuant is data-oblivious and online: it does not require calibration, retraining, or per-model adaptation before deployment. That makes it much easier to imagine as infrastructure rather than as an academic curiosity.

This is the real significance of the result. A compression technique is interesting. A drop-in compression technique is operationally important.

How TurboQuant Works

TurboQuant uses a two-stage design.

First comes PolarQuant, which applies a random rotation to the input vectors and then performs high-quality scalar quantization on the transformed coordinates. The key idea is that this avoids the usual overhead of storing full-precision quantization constants for each block, which is one of the reasons traditional vector quantization loses some of its practical efficiency.

Then comes QJL, short for Quantized Johnson-Lindenstrauss. QJL applies a 1-bit quantization step to the residual error left by the first stage. In the paper, the authors show that this second stage corrects the bias problem that appears when mean-squared-error-optimal quantizers are used for inner-product estimation. That matters because attention mechanisms depend on accurate inner products.

The result is a compression pipeline that stays extremely small in bit-width while preserving the geometric structure attention actually needs.

Why The Market Reacted

Memory stocks sold off after the announcement. Reporting from The Wall Street Journal and MarketWatch linked the move to fears that software advances like TurboQuant could flatten the demand curve for AI memory.

That reaction makes sense. The current infrastructure thesis for long-context AI assumes that larger context windows require proportionally more high-bandwidth memory. More context means more cache. More cache means more expensive hardware. If software starts compressing that cache aggressively with little or no quality penalty, the linear relationship between context growth and memory demand starts to weaken.

That does not mean memory demand disappears. It means the market has to take software efficiency more seriously than it wants to.

This is the same pattern showing up across AI again and again. First the industry assumes scale will solve the problem. Then software finds a way to move the bottleneck.

The Honest Limits

The result is important. It is not magic.

The public benchmarks in Google's post use Gemma and Mistral models, and the paper's KV-cache experiments top out in the open-model regime rather than the very largest production systems. That means the strongest version of the claim, that compression stays effectively lossless at much larger scales, is still an inference rather than a demonstrated fact.

The 8x speedup is also specific: it applies to attention-logit computation on H100 GPUs, not to total end-to-end generation latency. Real serving gains will depend on how much of the full inference stack is dominated by attention versus everything around it.

And despite the amount of excitement around the result, Google's blog post links papers, not a production library. The research is real. The turnkey deployment story is not official yet.

TurboQuant vs. KVTC

TurboQuant is not the only serious KV-cache compression result in circulation.

KVTC, a separate 2025 paper associated with NVIDIA researchers, reports much higher compression ratios, up to 20x, while staying within roughly 1 point of baseline accuracy across a broader model range that includes models up to 70B parameters. But KVTC uses a calibration step based on PCA-style feature decorrelation, which makes it less frictionless operationally than TurboQuant's data-oblivious design.

That makes the tradeoff clear. TurboQuant is not necessarily the most aggressive compressor. It may be the most deployable one.

For teams serving many models across different architectures, that distinction matters. Zero-setup infrastructure often beats theoretically stronger infrastructure that requires extra per-model work.

What Changes If This Holds

If TurboQuant's results hold as the technique moves into larger production systems, the effect compounds quickly.

Inference providers get more concurrent long-context sessions per accelerator. Smaller teams get access to longer contexts without paying frontier-lab hardware costs. Edge and prosumer deployments get more breathing room before memory becomes the limiting factor. And hardware investors have to confront an uncomfortable possibility: some portion of the AI memory boom can be offset by software.

That is why this did not feel like a niche compression paper.

Google did not announce a new model. It announced that one of AI's fastest-growing cost centers may be more compressible than the market assumed.

That is the kind of result everyone notices, whether they admit it immediately or not.

Sources:

Previously on TheQuery: The Model That Thinks With 12B Parameters but Knows Everything a 120B Model Knows - the efficiency thesis this result extends.