Inside GPT 6 Astra: How Looped Transformers Work

OpenAI describes GPT 6 Astra as its most capable and aligned model yet.

Greg Brockman went further, saying that it would be reasonable to view Astra as an early form of artificial general intelligence.

The model also became the first to cross OpenAI's Critical threshold for cybersecurity. According to the company, it can discover previously unknown vulnerabilities and build working exploits with limited human guidance.

Those claims are dramatic, but they are not the most important part of the release.

The real change is architectural.

A conventional language model passes information through a fixed sequence of transformer blocks and generates the next token. Astra can process its internal state through the same computational blocks multiple times before showing the user a single word.

It does not simply generate a longer chain of thought.

It spends more time thinking before it answers.

1 How Modern Language Models Work

Imagine typing:

Everest is...

The model has to predict what comes next.

Its attention mechanism determines which parts of the context matter. The word Everest activates related concepts such as mountains, altitude, the Himalayas, Nepal, Tibet, climbing, and expeditions.

The model then combines those associations with the surrounding context. In most cases, a continuation such as the highest mountain on Earth becomes highly probable.

Inside a conventional transformer, this representation moves through a sequence of blocks. Each block transforms it once and passes the result to the next one.

If a model contains 40 blocks, the hidden state moves through all 40 in sequence. The final representation is converted into a probability distribution for the next token.

More blocks usually mean greater computational depth. A deeper model can perform more transformations before committing to an answer.

But adding blocks also increases the number of parameters, memory requirements, and training cost.

The usual assumption has been simple: if you want a deeper model, you need to build a larger stack of layers.

A looped architecture changes that principle.

2 What a Looped Architecture Changes

According to reporting from The Information, GPT 6 Astra uses a looped transformer architecture.

OpenAI has not published a complete technical specification, so the exact implementation remains private. But the general mechanism is already well understood from public research.

Instead of sending the hidden state through every block only once, the model can return the result to a computational block and process it again.

Think of it as an internal draft.

The model produces a preliminary representation, sends it through the same weights again, refines it, and repeats the process several times. Only after the loop is complete does the next visible token appear.

The parameter count stays the same. The weights remain inside one block, compute grows with the number of loops, and only the final token is emitted.

The key detail is that the number of parameters does not necessarily increase.

The same weights are reused on every pass.

What increases is the amount of computation.

In a conventional transformer, a token travels through a sequence of different blocks. In a looped transformer, it can travel through the same shared structure multiple times.

The model becomes deeper through computation without becoming larger through parameter count.

This creates a new tradeoff:

More parameters require more memory
More loops require more computation
One model can spend different amounts of compute on different tasks

The final point is especially important. Computational depth can change without creating a new model.

3 How Recurrent Depth Is Trained

Simply placing a transformer block inside a loop is not enough.

If a model always performs exactly 32 passes during training, it may learn to rely on the position of each step. The first pass could learn one function, the tenth another, and the thirty second could become a fixed output layer.

The result would be a conventional 32 layer network disguised as a loop.

Researchers solve this problem by varying the number of recurrent passes during training.

One of the clearest demonstrations came from the recurrent depth model Huginn. Its creators trained a 3.5 billion parameter network on approximately 800 billion tokens.

At each training step, the number of loops was sampled from a distribution with a mean near 32. One example might receive four passes, another 17, another 40, and another close to 90.

The model could not know in advance which pass would be its last.

It therefore had to learn something more general than a fixed sequence of operations.

It had to learn how to improve its current internal state.

Abstract from the Recurrent Depth Paper on arXiv

Training through dozens of recurrent steps creates another problem: memory.

Standard backpropagation would require storing the intermediate activations from every loop. With enough repetitions, the memory cost becomes enormous.

The Huginn researchers used a practical compromise. The forward pass could run for many loops, but gradients were computed only through the final eight.

Earlier passes still influenced the final result, but their complete activation history did not have to remain in memory.

This is similar to truncated backpropagation through time. The difference is that recurrence happens across computational depth rather than across words in a sequence.

The number of loops is sampled at every training step, while the gradient is computed only through the final eight passes.

The model is not being taught to perform a rigid sequence of 32 operations.

It is being taught to move its hidden state toward a useful answer regardless of when the process is stopped.

That changes what depth means.

Depth is no longer only a property chosen by engineers before training. It can become a variable during inference.

4 Adaptive Compute During Inference

Most current reasoning controls operate at the level of visible tokens.

If a model is asked to think longer, it generates a longer chain of thought, uses more output tokens, or creates additional intermediate text.

A looped architecture offers a different mechanism.

The number of internal passes can change without modifying the weights and without forcing the model to write a longer explanation.

A simple task might receive four loops.

A difficult mathematical proof might receive 32.

A complex coding problem might receive 64.

The model stays the same. Only the amount of internal computation changes.

The weights remain fixed. Only the number of internal passes used to process each token changes.

Public recurrent depth experiments already show the expected pattern.

Performance improves quickly during the first loops. The gains then become smaller until the curves approach a plateau.

This suggests that the hidden state is converging.

Early passes make large corrections. Later passes refine smaller details. Eventually, additional computation stops producing meaningful improvements.

A future system could detect that moment automatically.

Instead of assigning every prompt a fixed loop count, the model could continue processing until its hidden state becomes stable enough. Easy tokens would be cheap. Difficult tokens would receive more compute.

That would create genuine adaptive computational depth.

Thinking longer would no longer have to mean writing more.

5 Research and Early Practical Results

Reusing transformer layers is not a new idea.

Universal Transformers introduced recurrent processing in 2018. ALBERT reduced parameter counts by sharing weights across layers. Mixture of Recursions explored routing tokens through different amounts of computation.

The open Nanbeige4.2 3B model provides an especially direct example.

Its configuration contains 22 layers and num_loops: 2. In effect, the model travels through those layers twice. It has roughly the parameter count of a 22 layer network but approximately the computational depth of a 44 layer network.

Nanbeige4.2 3B Configuration on Hugging Face

Twenty two layers, num_loops: 2, and an Apache 2.0 license. The mechanism is already visible in an open model configuration.

For years, recurrent transformer designs remained interesting research ideas rather than the dominant approach.

The missing ingredient was strong scaling evidence.

On September 1, a group of researchers published SMELT, a study designed to compare looped and conventional transformers under matched conditions.

They controlled for parameter count, training compute, and KV cache size. Under those constraints, looped models required 6.8% to 18% less training compute to reach the same level of performance.

The largest gains appeared on code.

Abstract from the SMELT Paper on arXiv

With compute, parameters, and KV cache matched, looping saves 6.8% to 18% of training compute, with the strongest gains on code.

The researchers also observed a change in attention behavior.

In conventional transformers, the earliest tokens in a sequence often become attention sinks. They receive a disproportionate amount of attention even when their semantic importance is limited.

After a second pass through the same blocks, the model's attention shifted toward more relevant tokens.

The second loop did not merely repeat the first. It used the result of the first pass as a foundation for more selective processing.

SMELT does not prove that every future model should be recurrent. It does show that looping remains competitive after the major experimental conditions are matched.

The question is no longer whether recurrent depth can work.

The question is how far it can scale.

6 Interpretability and Control

The same architectural feature that makes loops attractive also complicates model monitoring.

With explicit chain of thought reasoning, at least some intermediate steps appear as readable text. Researchers can inspect them, search for suspicious patterns, and compare the stated reasoning with the final answer.

A recurrent model can perform more processing inside hidden activations without generating any text.

Buck Shlegeris and Ryan Greenblatt have argued that this could reduce the usefulness of chain of thought monitoring. If more processing moves into hidden computation, the visible explanation may reveal less about how the model reached its result.

Sebastian Raschka offered an important counterpoint.

Reusing weights does not automatically create secret or inaccessible cognition. Internal activations can still be studied with interpretability tools. The main difference is that the model may need fewer visible intermediate tokens because more computation happens before generation begins.

OpenAI Chief Scientist Jakub Pachocki has also said that Astra's computational graph remains within roughly a factor of two of GPT 4's depth and that the loop was deliberately limited to preserve monitoring capabilities.

That suggests a restrained implementation rather than an unlimited recurrent process.

A chain of thought is not guaranteed to be a faithful transcript of a model's internal computation in the first place. The system is rewarded for producing useful answers, not for providing a scientifically accurate report of every hidden operation.

Research involving OpenAI, Anthropic, and Google DeepMind shows that a model's explanation can omit important factors, rationalize a decision after the fact, or present a cleaner path than the one that actually influenced the result.

A looped architecture makes this distinction harder to ignore.

Visible reasoning may be an interface.

The main computation may happen inside the model before any text appears.

7 GPT 6 Astra Results

GPT 6 Astra's most spectacular benchmark result is a score of 99.9% on ARC AGI 3.

That number sounds almost final, but it depends heavily on how the test is run.

Simon Willison highlighted that the 99.9% result came from OpenAI's custom Provider Adapter harness. Under the standard ARC Prize harness, the same model scored 62.7%.

The model did not change.

The evaluation environment did.

The same model produces two different scores at two different run costs.

The OpenAI run cost approximately $19,000. The standard run cost approximately $26,000.

The significantly higher score was also produced by the cheaper setup.

That does not necessarily make the result invalid. A custom adapter may interact with the model more effectively or expose capabilities that a generic evaluation framework misses.

But the headline number cannot be interpreted separately from the conditions that produced it.

GPT 6 Astra on ARC AGI 3

One model, two evaluation harnesses, and a gap of 37.2 percentage points.

Other evaluations are less dramatic.

On the Artificial Analysis Intelligence Index, Astra scores roughly on par with GPT 5.6 Sol and about five points below Claude Fable 5.1.

Its more practical advantage may be efficiency on agentic tasks, where it can reach competitive performance at a lower cost.

Benchmarks show what a model can achieve. Architecture helps explain where those capabilities come from and how they might develop.

8 What This Means for the Future of AI

For years, scaling language models primarily meant adding parameters, adding data, and building larger fixed stacks of transformer blocks.

Every new generation became larger, more expensive, and more demanding to run.

GPT 6 Astra points toward another path.

A model can reuse the same parameters, allocate more computation to difficult tasks, and refine its internal representation before producing its first word.

This separates model size from the depth of its reasoning.

Instead of following one fixed computational path, the system can adjust the amount of work to the difficulty of a specific problem.

That could change both model performance and the economics of using AI. The same intelligence could operate in a fast and inexpensive mode for simple requests, then increase its computational depth when a task genuinely requires more reasoning.

Recurrence is returning to language models. This time, it operates not primarily between words, but inside the process that creates them.

The next generation of systems may not become more powerful only because they contain more parameters.

Their defining advantage may be knowing when a good answer has already been found and when it is worth going around the internal loop one more time.