Architectures of Frontier AI: A Map of the Choices That Compound

The published frontier looks more uniform than it is. Read the recent technical reports from the major labs and the surface architecture is roughly the same: transformer decoder, attention with grouped-query or sliding window, mixture-of-experts at the feedforward layer, rotary position embeddings, context windows in the high hundreds of thousands of tokens. The choices that distinguish one lab's frontier model from another are mostly invisible to the public, and the choices that the field as a whole has converged on are stronger commitments than the lack of public debate suggests.

This paper does three things. It maps the visible architectural decisions. It separates the ones that compound across capability dimensions from the ones that are mostly orthogonal. And it argues that the architectural decision space is wider than the published frontier indicates, and that the lab betting on a different slice is doing the highest-EV research currently available.

The argument is not that the consensus architecture is wrong. It is that the consensus has been adopted with much less deliberation than the published technical reports suggest, and that the alternatives that were not pursued are not weaker than the alternatives that were. They are just less well-explored. The path-dependence of the field's architectural commitments is real, increasing every quarter, and producing a narrower frontier than the underlying decision space would admit.

The decisions that are visible

Mixture of experts versus dense

Every recent frontier model is sparse. The argument is straightforward: at scale, MoE buys you capacity per active FLOP. The trade is engineering complexity in training and serving, and inference latency variance that dense models do not have. The consensus is settled. Whether it should be is less clear.

The interesting question is not MoE versus dense at fixed parameter count. It is MoE versus dense at fixed inference cost. There is published work suggesting the answer here is more nuanced than the current consensus assumes, and the labs that have looked carefully are not all reaching the same conclusion. The serving-cost calculus that drove the field toward MoE at the frontier-scale tier does not necessarily hold at the inference profiles that the next generation of reasoning-heavy serving requires, and the labs that built MoE-first inference stacks may find themselves with infrastructure that does not match where capability is actually moving.

There is a second-order consideration that the field has under-discussed. MoE routing decisions are themselves a learned behavior, and the routing decisions made during pre-training are coupled to the training corpus in ways that make later corpus changes more disruptive than they would be for a dense model. The architectural commitment to MoE is therefore a commitment to a particular relationship between architecture and substrate, and changing one without changing the other is harder than it looks. This is one reason labs that have made the MoE commitment at frontier scale are reluctant to substantially change pre-training composition between generations.

Attention variants

Sparse attention, sliding-window attention, grouped-query attention, multi-query attention, flash-attention. Most of these are engineering decisions about throughput and memory more than capability decisions. The exception is sliding-window attention, which has real effects on long-context behavior that the published evals do not capture cleanly.

The sliding-window failure mode is structural and predictable. The model can attend to a recent window of context with full resolution, and to older context with degraded resolution or not at all. Tasks that require integrating information across the full context window degrade in ways that needle-in-a-haystack benchmarks do not surface, because needle-in-a-haystack tasks are dominated by retrieval of a single salient item rather than integration across the full attention surface. The published long-context evals score sliding-window models nearly as well as full-attention models. The behavioral gap shows up in production, in tasks the eval suite is structurally unable to measure.

Context length design

Context length is the most visibly competitive number in the spec sheet. It is also the place where the architectural decision is most coupled to the eval design. A model with a one-million-token context window does not necessarily do anything useful with all those tokens. The needle-in-a-haystack evals everyone runs do not surface the failure modes.

There is a recurring pattern where a lab releases a model with a context window an order of magnitude larger than the previous generation, scores well on the standard long-context evals, and is found in production to degrade substantially on tasks that require coherent reasoning across the full context. The pattern is so consistent that taking spec-sheet context length at face value has become a reliability hazard for downstream users. The serious question for any frontier model with a million-token context is not whether it can retrieve a needle; it is whether it can integrate across the full context in ways that the integration-heavy tasks of real work actually require.

Training-data substrate

The decisions here are mostly opaque. Public technical reports describe corpus composition at the level of marketing categories: code, science, web, books, mathematics. The actual decisions about source weighting, synthetic data composition, and filter design are not public. The capability differences between frontier models that show up in domain-specific evaluation are probably more attributable to this layer than to the visible architectural choices.

The under-discussed fact is that the corpus composition for the current generation of frontier models is substantially synthetic. Lab-A trains on output curated from Lab-B's previous-generation model. Lab-B does the same in reverse. The cross-pollination has been happening for at least two generations, and the resulting model behavior is shaped by a synthetic substrate whose composition is determined by the previous generation of models that were themselves trained on a partly synthetic substrate. The recursive nature of this process is one of the largest unaddressed sources of model-behavior risk in the current frontier.

Post-training stack

Instruction-tuning data composition, RLHF objective design, constitutional methods, refusal training, tool use training. The post-training stack is doing more work than is publicly acknowledged. The model you ship is mostly a product of post-training decisions made on top of a base model whose pre-training composition is itself opaque.

Compute spent on post-training at frontier labs has grown from a small fraction of pre-training compute to a comparable or larger fraction. The shift is structural, not transient. The marginal capability return on additional pre-training compute is decreasing relative to the marginal return on additional post-training compute, and capital allocation across labs has been adjusting toward post-training-heavy training stacks. The published technical reports describe this shift indirectly when they describe it at all, but the trend across the field is clear and the capital allocation behind it is substantial.

Reasoning at inference

Inference-time reasoning is the most consequential architectural shift of the past eighteen months. The model is given an extended budget of compute per query, used to construct intermediate reasoning steps that are sometimes returned to the user and sometimes hidden inside the inference process. The capability gain from this shift is not subtle. On hard reasoning benchmarks, the same base model with extended inference-time reasoning outperforms the same base model with standard inference by margins that are uncomfortably large for the previous generation of compute economics to make sense of.

Inference-time reasoning is not yet a first-class architectural element. It is grafted on top of architectures that were designed for standard inference, with the reasoning loop implemented in the serving stack rather than the model itself. The frontier capability that would come from designing the model itself for inference-time reasoning, with the architecture and the training objective and the reasoning loop co-designed from the start, has not yet been demonstrated by any major lab in the visible literature. This is one of the largest opportunities in the current decision space.

The architectural decisions the field is treating as settled are mostly path-dependent and reversible. The path dependence is real and increasing every quarter.

What compounds

Attention design and context length design are coupled. A decision about attention sparsity changes what the model can do across the full context length, and the same attention variant performs differently at 128k tokens than at 1M. These two choices compound because they jointly determine the long-range cognitive surface, and a misalignment between them produces models that look long-context on the spec sheet and do not behave long-context in production.

Training-data substrate and post-training stack are coupled. A base model trained on a particular corpus admits a particular post-training trajectory. The labs that change their corpus composition find that their post-training recipe also has to change. This pair compounds because the substrate determines the prior that the post-training is shaping, and post-training that worked well on one substrate may fail to converge well on a different one.

Inference-time reasoning compounds with the post-training stack in ways that are still being mapped. A model trained to produce explicit reasoning during post-training behaves differently when run with an extended inference-time reasoning budget than a model trained without that explicit reasoning emphasis. The labs that design the post-training pipeline with inference-time reasoning in mind from the start are building models that compound across both surfaces. The labs that bolt reasoning on at the inference layer after training is complete are getting some of the gains and missing others.

Mixture-of-experts versus dense is mostly orthogonal once you scale past about ten billion parameters. The choice has real implications for engineering and serving, but the capability differences it produces are first-order recoverable through other means. This is one of the few visible architectural decisions that does not compound across capability dimensions in the way the field assumes.

What the published frontier is not exploring

Mixture-of-depths. Recurrent context compression. State-space hybrids. Continuous reasoning at inference time as a first-class architectural element rather than a post-training graft. Two-stream architectures where retrieval is part of the forward pass rather than appended to the prompt. Some of these are explored in published research. None of them are in the visible frontier.

The lab betting on one of these alternative slices, with serious capital and patient time horizons, is doing the highest-EV research currently available. The capability ceiling of the consensus architecture is closer than the current spec-sheet competition suggests, and the next generation of frontier capability is likely to come from a different architectural commitment than the one the field has converged on.

The barrier to alternative architectural exploration is not technical. The barrier is infrastructural. The consensus architecture is now backed by a deep stack of tooling, serving infrastructure, training infrastructure, and operational knowledge that compounds independently of the capability question. A lab choosing to bet on an alternative architecture has to either build that infrastructure from scratch or operate at a disadvantage relative to labs that inherit it. This is the kind of barrier that is invisible in the technical conversation about architectural choice and decisive in the practical conversation about what gets built.

The window is closing

The window in which an alternative architectural bet can catch up is closing, not because the bet does not work, but because the consensus architecture is accumulating tooling and infrastructure investment that compounds independently of the capability question. A lab choosing to bet on, say, a state-space architecture today is choosing to also build the kernels, the parallelism strategies, the serving stack, the debugging tools, the evaluation harnesses, and the institutional knowledge that the consensus architecture has spent four or five years building. This is doable. It is also a substantially larger investment than the architectural choice alone suggests, and the labs that do not move soon will find the gap large enough that the architectural choice is effectively locked in by the cost of the supporting infrastructure.

Conclusion

The architectural decision space at the frontier is wider than the published technical reports suggest. The visible convergence is not because the alternatives have been explored and rejected. It is because the alternatives have not been explored at scale. The decisions that the field is treating as settled are mostly path-dependent and reversible, but the path dependence is real and increasing every quarter. The window in which an alternative architectural bet can catch up is closing, not because the bet does not work, but because the consensus architecture is accumulating tooling and infrastructure investment that compounds independently of the capability question.

For the field as a whole, the takeaway is that more architectural diversity at the frontier would be more productive than the current narrow convergence. The labs that bet on alternative architectures should be supported. The capital that goes into more pre-training scaling on the consensus architecture might be better spent on infrastructure for one of the alternative slices. The strongest research bets in frontier AI over the next three years are probably the ones that look maverick today and that the major labs are not pursuing because the consensus architecture is good enough and the cost of betting differently is high.