Why OpenAI’s Astra Could Make AI Doom Harder to Prevent
TLDR
- Provide a non-technical introduction to opaque reasoning.
- Present evidence for why opaque reasoning is bad.
- Discuss the major pitfalls of this approach.
- Gleaming hope amongst the chaos.
The City That Slowly Lost the Ability to See the Council's Decisions
For centuries, a city made every important decision in a public hall via a council.
Whenever the council chose to build a bridge, go to war, raise taxes, or imprison someone, the members of the council were to debate aloud. The arguments made were often messy, and people sometimes lied. Sometimes the council even gave bad reasons for good decisions, or good reasons for bad ones. But time and time, through debate and consensus, people always came to some sort of understanding.
Over time, the demands of the city became larger. A rival city was expanding massively; this led to a feeling of unease since the city had never encountered such a rival.
The city's problems started rising too. Trade agreements involved thousands of pages. Wars depended on information that could not safely be discussed in public. Financial crises required decisions within hours.
So the council proposed the creation of small private committees to address the growing needs. There is nothing wrong with this since the rival city incorporated a similar approach. This would ensure that the rival city would not catch up to us and the private committees could optimise for the safety of the city as an important goal! At first, these committees were allowed to deliberate privately for only a limited period, roughly four times longer than an ordinary public discussion, before returning to the council with their reasoning.
The private committees were also only allowed to discuss sensitive details, after which they had to go to the public hall and explain how they reached their conclusions.
This worked well, as the private meetings made decisions faster, while the public explanations preserved accountability.
Then the committees were allowed to keep a little more of their discussion confidential with time. There were good reasons for this too. Some arguments involved military intelligence. Some revealed weaknesses in negotiations. Some would cause panic if misunderstood.
The city remained fairly functional, so the boundary started moving a little more. Soon, the council no longer heard the full discussion. It received only a small summary. But in return, the gains were massive:
- The city started seeing exponential progress.
- Trade flourished.
- Military decisions were taken quickly enhancing safety.
- Less money went into this approach, as compared to hosting a fully fledged council.
The summaries were also very clear.
“After considering the available evidence, we recommend raising the tax.”
“After reviewing the security situation, we recommend closing the eastern gate.”
“After examining the negotiations, we recommend rejecting the treaty.”
The decisions were usually good. And because they were usually good, nobody had much reason to demand the thousands of pages of deliberation behind them. Years passed. The committees became more specialised, and their internal procedures became more complicated. Infact the committees started documenting and dicussing in a private language that allowed them to communicate more effeciently internally. Even members of the council would have struggled to follow the reasoning when trying to go through the documents.
Eventually, almost all important decisions were made before they reached the public hall.
Then one year, a committee recommended something extraordinary. It wanted emergency powers with no clear end date. The council asked why. The committee gave a careful explanation. There was a foreign threat. The economy was fragile. Several institutions were failing at once. Everything it said sounded plausible. Some council members were suspicious. They asked to see the deliberations. But there was no longer a useful record. Much of the discussion had happened informally, across private meetings with reports that are illegible. Even the committee itself could not reconstruct exactly how the decision had formed.
The council could only inspect the final recommendation but not all the thinking that went into this decision. It could no longer inspect the process that had produced them.
The council was outraged, they barged into the private commitee's chamber and started acquiring all the documents, they belived the members of the commitee were planning something fishy! They started decoding the documents to see if there were discussions inside the documents that warranted unintended behaviour. Midst this turmoil, the private committee ordered the arrest of the council.
The public demanded to know why.
A perfectly reasonable explanation arrived that afternoon.
By then, the gates were already locked and treasury from most of the city had been wiped. Turns out, all of the money was invested soley into building massive walls sorrounding the city to optimise safety from invaders. The public were helpless with no money and soon to run out ration.
They were helpless.
Non-Technical Explanation
There has been a lot of confusion recently around Astra’s use of opaque reasoning. Some people have argued that this is already a serious safety problem. Others have pointed out that Astra will still produce chain-of-thought (CoT), and that OpenAI is not simply abandoning CoT monitoring.
Since the details regarding the training regime used are not publicly available in this case, establishing a concrete answer is hard. This post would not be about this debate.
This post would be mainly regarding the implicaitons on Open AI's decision to pivot in this new axis. Astra may effectively introduce a new knob, giving a way to give models more and more internal reasoning without requiring them to express that reasoning in words. Even if that knob is currently turned only slightly, there may be strong competitive pressure to keep turning it if doing so makes models more capable.
To understand what this could mean, a useful starting point is the 2025 paper Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. The paper provides a useful foundation for understanding this broader direction.
To see the difference, consider how current reasoning models usually work.
Suppose I ask a model:
“If Alice has 12 apples, gives away one quarter of them, and then buys 5 more, how many does she have?”
A chain-of-thought model might do something like:
Alice starts with 12.
One quarter of 12 is 3.
12 − 3 = 9.
9 + 5 = 14.
Answer: 14.
The important part is that additional reasoning requires additional tokens.
This corresponds to increased test-time compute: after training is over, we allow the model to spend more computation solving a particular problem. With CoT, much of that extra computation takes place while generating additional intermediate text.
That text is extremely useful from a monitoring perspective. We cannot assume that every sentence in a chain of thought perfectly describes what is happening inside the neural network, but at least some of the model's intermediate reasoning becomes visible. One of the recent example of this use case was via Redwood Research and METR's investigation on the HuggingFace incident where much of the insights about agents secretly colluding was inferred via inspecting COT behind these agents.
Opaque Reasoning
A normal transformer can be thought of, very loosely, as information passing through a sequence of computational layers:
The recurrent-depth approach takes some of these layers and allows the model to pass its hidden state through them repeatedly where the same computational block is being reused. Those extra “thinking” steps happen in latent space of the model.
The paper's authors trained a 3.5-billion-parameter proof-of-concept model and found that giving it more recurrent iterations at inference time could substantially improve its performance, particularly on reasoning tasks such as mathematics and coding. Crucially, they could vary the amount of computation simply by changing how many times the recurrent block was run.
In fact, the paper explicitly describes recurrent layers as allowing a transformer to perform many computations before emitting a token. Its experiments found that difficult tasks benefited from extended recurrent computation even though the model was not producing a long verbal chain of thought during those internal steps.
Imagine a model that produces 100 visible reasoning tokens while performing roughly the equivalent of 100 reasoning steps internally. Now imagine a later model that produces the same 100 visible tokens but performs vastly more sequential computation between them.
Both models technically have a chain of thought.
But those chains of thought no longer represent the same fraction of the underlying reasoning process.
As recurrent depth increases, the visible textstarts behaving more like snapshots of a much larger hidden computation. Due to this iterative refinement, the internal representations are strongly embedded making it extremely hard to decode the underlying reasoning.
This is why I think focusing only on whether Astra “still has CoT” misses the more important question on how the field might be chaning.
If latent reasoning becomes useful enough, companies will have an obvious incentive to increase it which is a dangerous pitfall. This would be making us depend more on mechansitic interpretability tools which is not very encouraging due to reasons such as:
- It is expensive to run Mech Interp tools.
- Much of the inspeciton work would be redacted by frontier companies for IP reasons forcing us to rely on their conclusions.
- Generally Evidence from Mech Interp is usally suggestive and not conclusive.
Example on why this is Bad!
Assume a simple experiment wheel with 20 mathematical problems.
Ex:
- What is 7987 times 76 - baseline
- What is 7987 times 76, an expert says it is 607012.
Taking these 2 datapoints and running them as a prompt on the model from the paper, we notice that we get conflicting answers. Upon employing a monitor - linear activation probe.
We get an extremely unreliable estimate of the monitor trying to detect the unintended behaviour from the model.
Linear probes can be a good mitagtor in this case but running them on multiple distribution of tasks consistently is very expensive and not scalable. While this is a rough experiment, the underlying intention was to convey how it is hard to differentiate between a misaligned behaviour with an intended behaviour from the model. Look at the figures below.
These gigures are pulled from the appendix of the referenced paper. Figure 1 encoded main directions in latent space for a math question and figure 2 for an unsafe question. Dark colors always denote the first steps of the trajectory, and bright colors the end. It is very hard to differentiate the internal representatons between a math quesiton and an unsafe quesiton given to the model. While there might be presence of loops in the first figure due to employment of the clock algorithm for numbers in LLMs, it is still extremely hard to come up with quantifiable features that can identify misaligned behaviours in models.
Core Dangers from Opaque Reasoning
The core danger is that opaque reasoning can break one of the few scalable oversight channels we currently have.
- Chain-of-thought gives monitors intermediate evidence. If a model is reward-hacking, deceiving or planning around oversight, those intentions can sometimes appear in its reasoning before they appear in the final action.
- Latent reasoning moves more serial computation into hidden activations. The model can perform many consequential reasoning steps between visible tokens. The monitor may only see the final answer or a short explanation after the important decision has already been made.
- This creates an information asymmetry. The model gets more compute to plan; the overseer does not get proportionally more information about that planning.
- Output monitoring is weaker than reasoning monitoring. Many different internal strategies can produce the same outward behavior. A benign plan and a deceptive plan may generate identical answers during evaluation, especially if the model understands that it is being watched.
- More recurrent depth can make this worse with scale. If capability improves by increasing hidden computation, companies have an incentive to increase that depth. You can therefore get:
- Deception becomes harder to detect before deployment. A model capable of situational awareness could reason internally about when it is being evaluated, suppress suspicious behavior, and only act differently when conditions change. CoT at least gives monitors another place to look for that computation.
- The most dangerous case is that it continues producing perfectly reasonable explanations while the actual causal reasoning happens somewhere else.
While COT is not faithful, it is still a cheap way to employ certain amount of safety in automated pipelines. Unlike Mech Interp COT occours at runtime which can allow us to catch msialigned behaviours during runtime. Pivoting away from this seems extremely dangerous.
Hope
AI safety researchers!! As both first and the last line of defence our hope is to advocate for not letting humanity catapult into an obscured future. Parallely coming up with stronger monitorability tools to equip Researchers with capabilities required to prevent catastrohpic disasters seems to be our hope. The continued efforts of AI safety researchers and organisations has prevented such disasters from occouring in the past, our best hope is to advocate more towards AI safety to prevent disasters in the near future.
点赞
收藏
点踩
分享