In short

The transformer architecture, though revolutionary, has become a research bottleneck trapping AI in a local minimum, echoing co-inventor Llion Jones and Ilya Sutskever's view that scaling alone won't reach AGI. This essay surveys three emerging successors: text diffusion models that generate whole sequences at once; Continuous Thought Machines using neural synchronisation for adaptive compute; and Nested Learning (HOPE) with multi-speed memory enabling continual learning.

The transformer launched the modern AI boom, and the argument of this essay is that it may now be holding it back.

When even one of the architecture’s own co-inventors suggests we’re stuck in a local minimum, it’s worth asking what comes next. The piece surveys three serious contenders, each attacking a different transformer weakness: text diffusion models, which generate a whole sequence at once and so can plan rather than blurt one token at a time; Continuous Thought Machines, which let the model spend more compute on harder inputs; and Nested Learning (the HOPE architecture), which builds in multiple speeds of memory to enable genuine continual learning.

None has dethroned the transformer yet. But this is a guided tour of where the frontier is actually moving, and a useful corrective to the assumption that “more scale” is the only road ahead.

Key takeaways

  • Transformers have become a bottleneck, confining AI research to a local minimum despite their success.
  • Text diffusion models generate entire sequences simultaneously, improving planning and parallelisation.
  • Continuous Thought Machines use neural synchronisation for adaptive, variable computation per input.
  • Nested Learning (HOPE) introduces multi-speed memory layers enabling continual, on-the-job learning.
  • Interim fixes include sparse and gated attention; 2026 may bring mainstream post-transformer text models.

Read the full piece

This is a summary. Read the complete essay, with all the sources and argument, on Substack.

Read on Substack →

Frequently asked questions

Why are transformers considered a bottleneck for AI?
Their success masks fundamental problems, trapping research in a local minimum. Figures like co-inventor Llion Jones and Ilya Sutskever argue scaling transformers alone cannot reach AGI.
What are text diffusion models?
AI models that generate an entire text sequence simultaneously by denoising, rather than producing tokens one at a time, enabling better planning and parallelisation.
What architectures might replace the transformer?
Three contenders: text diffusion models, Continuous Thought Machines, and Nested Learning (HOPE), each addressing planning, adaptive compute, or continual learning.

People & ideas in this piece

Llion JonesIlya SutskeverYann LeCunText diffusion modelsContinuous Thought MachinesNested Learning / HOPEDeepSeekSparse attentionCatastrophic forgetting

Topics: How AI Actually Works