>_TheQuery
← All Articles

Subquadratic Says It Broke the Transformer Ceiling

By Addy · May 6, 2026

Every AI model you have used through a modern chat interface since 2017 has been paying a tax.

Not a financial one. A mathematical one. When a transformer model processes text, it computes a relationship score between every token and every other token in the sequence. Read that again. Every token against every other token. If you double the input, the work does not double. It quadruples. If you triple the input, the work grows ninefold. This is what it means for attention to scale quadratically - and it is the architectural constraint that has quietly shaped every context window limit, every pricing tier, every "we support up to one million tokens" claim from frontier labs since the transformer architecture was published in the original Attention Is All You Need paper in 2017.

A million tokens sounds like a lot. It is roughly 750,000 words. It is more than most humans will read in a week and more than enough to swallow a long novel, a full diligence packet, or a meaningful slice of a codebase. The reason it became the ceiling is not that someone decided a million tokens was enough. It is that the compute required to go further made the economics collapse. Doubling to two million tokens quadruples the attention cost. Getting to twelve million tokens on standard transformer attention would require compute so large that the cost per query would make the model commercially unviable for almost every real workload.

On May 5, 2026, a Miami-based startup called Subquadratic walked out of stealth and said it had solved the problem.

Who Built This and Why It Matters That They Did

Subquadratic was founded by Justin Dangel as CEO and Alex Whedon as CTO. Whedon's background is the detail that makes the claim harder to dismiss than it might otherwise be: Subquadratic's own launch materials describe him as a former Meta software engineer who later served as Head of Generative AI at TribeAI, where he led more than 40 enterprise AI implementations. That does not make the benchmark numbers true. It does mean the company is not arriving at the problem from a purely theatrical place.

The $29 million seed round includes Justin Mateen, who co-founded Tinder, Javier Villamizar from the former SoftBank Vision Fund, Grant Gittlin of Lasagna, Jaclyn Rice Nelson of Coalition Operators, and early investors in Anthropic, OpenAI, Stripe, and Brex. The company says its research team includes 11 PhD researchers and research engineers with backgrounds from Meta, Google, Oxford, Cambridge, ByteDance, Adobe, and Microsoft.

The company was previously known as Aldea. Earlier hiring pages described Subquadratic, fka Aldea, as focused on speech and language models. The pivot to long-context architecture appears recent and deliberate. The technical architecture behind SubQ - Subquadratic Sparse Attention, or SSA - is now the reason for the company's existence in its current form.

The core claim of SSA is linear scaling. Not quadratic. Linear. Where standard transformer attention quadruples its work when the context doubles, SSA is designed to scale at roughly the same rate as the input grows. Subquadratic says SSA is 52.2 times faster than dense attention with FlashAttention-2 at one million tokens on Nvidia B200s. At twelve million tokens, the company claims the architecture reduces attention compute by almost 1,000 times compared with frontier transformer models. The model is positioned as processing the equivalent of roughly 120 books in a single context window while costing far less than dense-attention frontier models at comparable long-context workloads.

That is the claim. The rest of the story is how much of it has been verified.

The Number That Stops You Mid-Scroll

There is one benchmark comparison in SubQ's launch coverage that, if independently reproduced, makes everything else secondary.

On RULER 128K - the long-context benchmark used to test whether models can retrieve and use information buried deep in a long context - SubQ 1M-Preview scored 95.0% accuracy. Claude Opus 4.6 scored 94.8% in Subquadratic's comparison. Subquadratic's launch coverage put the cost for that run at 8perquery,comparedwithapproximately8 per query, compared with approximately 2,600 for Claude Opus.

One percentage point of accuracy difference. More than three hundred times the cost.

That cost comparison is not independently reproducible yet because Subquadratic has not published normal self-serve pricing. But if the number holds under independent evaluation, it does not just change how developers think about long-context models. It changes the economics of the entire enterprise AI stack.

MRCR v2 is where the launch story needs more precision than some early summaries gave it. Subquadratic's own launch post cites a research result of 83 on MRCR v2, but its third-party verified production score for SubQ 1M-Preview is 65.9. The same table lists GPT-5.5 at 74.0 and Opus 4.6 at 78.3. Launch coverage emphasized the 83 research result against GPT-5.5's 74.0 score; VentureBeat and Subquadratic's technical post both note the verified production score is 65.9.

That distinction matters. The honest version is not "SubQ is already the verified best retrieval model at any context length." The honest version is sharper and more useful: SubQ's research result would beat GPT-5.5 on MRCR v2, while the named production preview trails the strongest published frontier baselines on that benchmark but claims a very different cost curve.

On SWE-Bench Verified, SubQ reports 81.8% - slightly above Claude Opus 4.6's 80.8% in the company's launch comparison and in the same band as models that have spent years optimizing for software engineering tasks. Subquadratic's own technical post also lists Opus 4.7 at 87.6%, which means the safest reading is competitive, not cleanly dominant. VentureBeat also noted that SWE-Bench should be treated carefully because harness choices can matter as much as the model.

On needle-in-a-haystack retrieval at the full 12 million token window, launch coverage reports SubQ at 92.1%. This is the test that breaks many long-context systems: hide a specific sentence in a massive document and ask the model to find it. 92.1% accuracy at that scale, if independently reproduced, would be the clearest demonstration that the 12 million token window is functional rather than nominal.

The phrase doing the work there is if independently reproduced.

The Difference Between a Context Window and a Functional Context Window

This distinction matters, and most coverage still glosses over it.

Every frontier lab in 2026 advertises a context window. Some models accept 128,000 tokens. Some accept one million. A few demos and research systems go beyond that. What these numbers describe is the maximum input the model will accept. What they do not describe is how well the model reasons across all of that input.

Think of it like working memory in a human. A person can technically hold a ten-hour meeting transcript in front of them and read every word. That does not mean they can reliably answer a question about something said in hour three while tracking the implication of something said in hour eight. Attention degrades with distance. Information buried in the middle of a long context is harder to retrieve than information near the beginning or end. The "lost in the middle" problem - where models fail to use information in the center of long contexts - has been documented across major model families.

MRCR v2 is designed to test this. It places reference information at multiple locations in a long context and asks the model to find and reason across all of them. A model that scores 65.9% on the verified SubQ 1M-Preview run is not magically perfect. But a model that can maintain retrieval quality while making million-token contexts much cheaper would still be a material improvement, because enterprise failure rates compound into real errors.

The claim SubQ is making is not just that it accepts more tokens. It is that the tokens it accepts can actually be used at a cost curve dense attention cannot match. SSA's linear scaling means the model's attention mechanism is designed to avoid paying for every possible token-pair relationship as the context grows. If the routing works, the model can preserve long-range retrieval without the quadratic bill that forces dense-attention systems into compression, retrieval, or selective context loading.

That is the real architectural bet.

The Magic.dev Problem

Before writing the rest of this article as a victory lap, the Magic.dev precedent deserves its own paragraph.

In August 2024, Magic announced LTM-2-mini, a 100 million token context model, and said its sequence-dimension algorithm was roughly 1,000 times cheaper than Llama 3.1 405B's attention mechanism at a 100 million token context window. Magic also announced a $320 million round, bringing its total funding to roughly half a billion dollars. The announcement generated significant coverage. As of early 2026, VentureBeat notes there is no public evidence that LTM-2-mini has been deployed outside Magic's own environment.

The parallels to SubQ's launch are uncomfortable enough that they need naming directly. Both companies claim massive context windows. Both claim roughly 1,000x efficiency gains. Both launched with benchmark numbers that require more independent third-party testing before the market should treat them as settled facts.

There are also differences. Subquadratic has published a technical explanation of SSA and has third-party verified numbers for SubQ 1M-Preview on at least some benchmarks. But the full model card is still listed as coming soon, the weights are closed, and the product access is private beta. The company says its API exposes the full 12 million token context window, while the named model in the launch materials is SubQ 1M-Preview and the most concrete verified benchmark table is at one million tokens.

The AI research community's response to SubQ split roughly along those lines: skeptics invoked Theranos and Magic.dev; enthusiasts pointed to the specificity of the SSA mechanism and the fact that sparse attention itself is not science fiction. AI researcher John Rysana pushed back on the Theranos framing directly, arguing that the work looks like subquadratic attention done well rather than vaporware. Others questioned the early-access gating, benchmark selection, and gap between the 83 research MRCR result and the 65.9 verified production score.

Independent evaluation will settle this. Until it does, SubQ's launch numbers are the most interesting self-reported claims in AI this month - which is a category with a mixed track record.

What Happens If the Architecture Is Real

Set aside the skepticism for a moment and reason from the premise: what does the world look like if SSA, or something architecturally equivalent, becomes the standard for how language models handle long contexts?

The first consequence is direct: every use case that currently requires RAG becomes simpler. Retrieval Augmented Generation exists because models cannot hold entire knowledge bases in context. You chunk the documents, embed the chunks, retrieve the relevant ones at query time, and inject them into a shorter context window. It is an engineering workaround for a fundamental architectural limitation. A model that can hold twelve million tokens and retrieve accurately across all of them does not eliminate the need for search in every system, but it changes RAG from mandatory plumbing into an optional optimization.

The chunking strategy, embedding model, vector database, retrieval logic, reranking step, and context packing layer stop being the default architecture for every long-document workflow. They become choices.

The second consequence is for agents. The failure mode that GitHub's squash bug incident exposed, that the vibe coding analysis in this publication covered last week, that every multi-step agent deployment eventually hits, is context degradation. An agent working on a large codebase over a long session accumulates context until the relevant information from early in the session is no longer reliably accessible. KAIROS, the background memory consolidation feature found in Anthropic's leaked Claude Code source, exists to solve this problem by compressing and reorganizing context before it degrades. A model with twelve million tokens of reliable context does not need that style of memory consolidation as urgently. The agent can simply hold more of the session in active memory without the same quality drop.

The third consequence is enterprise document processing. Legal review, financial due diligence, medical records analysis, regulatory compliance - these are the workloads where context length is often the binding constraint, not raw model intelligence. A model that can process an entire contract portfolio, a company's full financial history, or a patient's complete medical record in a single context window, accurately and cheaply, does not just make existing workflows cheaper. It makes previously uneconomical workflows viable.

That is why the benchmark numbers matter. Not because RULER is destiny. Because cost curves decide what developers can afford to build.

The Open Source Consequence Nobody Is Talking About

The conversation around SubQ has focused on the benchmark numbers and the Magic.dev comparison. The implication that has received less attention is the one with the longest tail.

SubQ itself is not open weight. The company does not plan to open-source the weights in the near term, though launch coverage says it may offer training tools for enterprise post-training. But if subquadratic attention works as claimed and the architecture becomes reproducible through public technical writing, independent replication, and open-source implementation work, then the efficiency gains do not have to stay inside SubQ's API. They can propagate into every open-weight model that adopts a similar mechanism.

This month alone, TheQuery has covered Qwen3.6-27B matching Claude Opus 4.5 on three benchmarks at a fraction of the cost, DeepSeek V4-Pro pushing competitive programming performance under an open license, and Ant Group's Ling-2.6 family using hybrid attention ideas to cut token waste in agentic workloads. Open-source efficiency research is already compressing what was once a hardware and capital advantage into a software one.

SSA, if validated, is the next compression. A Qwen, DeepSeek, or Ling-style model trained with SSA-like attention would inherit the context scaling properties without the quadratic tax. A 27B dense model that already matches frontier reasoning on selected benchmarks, running on subquadratic attention, becomes a different class of tool: a model that can process millions of tokens reliably at a cost developers can actually use.

The implication for inference cost is direct. Cheap inference has already been the story of the past twelve months. DeepSeek, Kimi, Qwen, and other open or open-weight models have pushed per-token prices below frontier closed-model pricing. But long context is where the old architecture bites hardest. A generation of open models built on subquadratic attention and running on increasingly commoditized hardware would push inference costs lower specifically for the workloads that are currently most expensive and most constrained.

The developer in a market where $25 per million output tokens is not an abstraction but a real cost barrier does not just get access to cheaper versions of the same capability. They get access to a fundamentally different class of capability - long-context reasoning at scale - that has been economically inaccessible regardless of model quality, because even cheap models eventually run into the quadratic wall.

The Ceiling Was Always Mathematical

The history of computing is a history of constraints that seemed permanent until they were not.

The speed of light was supposed to limit how fast transistors could switch. Dennard scaling would eventually fail. Memory bandwidth would become the binding constraint before compute did. Every one of these ceilings held until someone found an architectural workaround that made the constraint less relevant for practical purposes.

The quadratic attention constraint is nine years old. It has shaped the entire large language model era: context window limits, pricing tiers, engineering workarounds for long documents, RAG pipelines, agent memory management systems. The industry has built an enormous amount of infrastructure around this constraint because nobody knew how to remove it without sacrificing model quality.

SSA is not the first attempt to remove it. Mamba, RWKV, Hyena, DeepSeek Sparse Attention, and a half-dozen other efficient architectures have been published and evaluated. None has fully replaced transformer attention at frontier scale while also delivering the efficiency gains the theory predicts. The common finding is that subquadratic attention often trades quality for efficiency in a way that makes the tradeoff unattractive for production use, or keeps enough dense attention around that the quadratic cost remains load-bearing.

SubQ's claim is that SSA closes that quality gap. The RULER result, the MRCR research score, the one-million-token verified score, and the SWE-Bench Verified number are the evidence offered for that claim. Some are third-party verified. Some are research results. None has yet been broadly replicated by the wider community.

That is the right level of skepticism. The architecture is described publicly, but the full model card and broader evaluations are still pending. The weights are closed. The product is in private beta. The 12 million token claim is reported, but the most concrete public benchmark table is still centered on SubQ 1M-Preview.

If the quality gap is genuinely closed - if subquadratic attention at frontier scale produces the benchmark numbers SubQ claims while preserving the cost curve - then the constraint that has defined AI for nine years is no longer structural. It becomes a choice. Labs that keep training on dense quadratic attention would be choosing to pay the tax when an alternative exists.

That transition will not happen overnight. Retraining frontier models is expensive. Existing infrastructure is optimized for transformer attention. The ecosystem of kernels, serving stacks, eval harnesses, and hardware assumptions built around dense attention does not disappear because a Miami startup published a technical post.

But ecosystems do change. The open-source community that takes new model architectures and gets them running locally within days will not wait for frontier labs to bless SSA before experimenting with it. If the architecture works at smaller scales, the open community will implement it, optimize it, and try to ship it in llama.cpp and vLLM before most enterprise AI teams have finished reading the original launch post.

The ceiling was always mathematical. Mathematics does not negotiate. When someone finds the proof, the ceiling is gone - regardless of what the incumbents built underneath it.

SubQ may or may not be that proof. Independent evaluation will tell us. But the question it is forcing the industry to answer - whether nine years of quadratic scaling was a fundamental constraint or an unsolved engineering problem - is the right question, and it is being asked at exactly the right moment.

Sources:

Previously on TheQuery: A 55GB Open Model Just Beat Claude Opus on Three Benchmarks and DeepSeek V4 Is Almost at the Frontier. The Price Is Not.