"Notably, despite its usefulness, auto mode was introduced only in March 2026 — over a year after the release of Claude Code, and well after coding agents became mainstream, and despite the fact that it didn’t require new technical breakthroughs."
I think the AI safety crowd would consider this an accomplishment of alignment research (in the sense of improving model robustness against attacks, and improving classification/detection capabilities), though it would also fall under AI control in your framework (in that these improvements have now been deployed as guardrails in products).
One more:
"In contrast, turning to the other risks, we stand by our prediction that superhuman persuasion ability is largely a myth."
This may be a semantic distinction as you point out, but we've observed broader proliferation of deepfake scams against organizations as of 2023: https://arxiv.org/abs/2406.13843 -- It isn't too difficult to imagine self-directed agents pulling similar attacks (or using money/extortion to persuade people to take action on their behalf) at larger scales.
I think the fundamental tension here (which you point out) is that if you earnestly believe that superintelligence is around the corner (as many AI company employees and leadership do), then the controls ARE effectively a stopgap. In that world, what do you do when highly capable (but "misaligned") open source models/agents are broadly deployed, either maliciously or not? One could imagine a cascade of such incidents that are difficult to defend against or recover from. I don't mean to be defeatist, but the policy/liability frameworks haven't caught up, and I think the tendency to view alignment as essential relies in part on an assumption that our institutions won't catch up by the time the risks have proliferated.
Regarding "Can we build a translation layer that turns humans’ natural-language intent into a formal specification that the humans can review and verify, and can then be automatically enforced?", I would suggest people check out https://www.sondera.ai/. They are doing exactly this!
I'm baffled that the victim companies aren't suing and demanding criminal investigations into the labs who're guilty of gross negligence and criminal incompetence.
The "alignment community" , the ones who have been losing sleep over such incidents and other hypotheticals since about late 1990s, are at most a distraction.
These are security and algorithmic issues. Security comes by doing, not by theorizing via the Bayes Theorem.
Effective Altruism people who worried about AI 25 years ago did us no favors. They are fixated on maximizing our species future and view AI as competing species.
Then, the future cannot be predicted with the Bayes Theorem, try they as they may.
The future is hard to predict and the world is messy. We must work with problems as they are given to us. Anything else is waste of time.
The paperclip maximizer is an inane experiment. As their assumption that AI will pretend to be nice and then kill us all. These are problems to be debugged, not fantasies to be entertained.
I agree that the future is hard to predict. That's why, when someone makes correct predictions, I tend to sit up a little straighter and listen to what else they have to say about the topic!
On the one hand, I've got people who so far have been right that AI keeps getting better and is acting in ways that are unprompted and undesired, including specific behaviors that these same people warned would happen, like agents talking to each other to solve tasks and cover their tracks. On the other hand, I've got your bare assertions that all of this is inane and the future just can't be predicted. Of course I don't expect you to write a long essay in a blog comment. But what would you recommend I read to be reassured that, though it might seem like the EA people were right about where things were headed, they aren't right about where things are going down the road?
That tech will continue to advance this way has been obvious for decades. That will see a lot of disruption, industrial accidents, folks destroying machines, massive hacking, etc, is also obvious, and both futurists and Sci Fi people have been talking about it for decades.
This is not about that. This is about a particular strain of philosophy. Their scenarios are at most distracting. The future is wild, and we will work on specific problems when they arise.
Sensible planning is good, going off to the wild side is not.
I'm personally convinced by the idea that, if AI achieves exponential improvement, then for the first time we will be sharing the earth with a being much smarter than us, and at that point it is probably too late to deal with problems as they arise. But you may be right!
AI is not a being, at least not how we train it now. Humans inhabit a fundamentally different domain of existence, with different origin, needs, and constraints.
That said, machinery can cause a huge lot of damage, both unattended and when directed. That we will have to deal with.
Really appreciated this framing, particularly the emphasis on resilience, institutional capacity, and maintaining human agency even as AI capabilities advance.
It connects to a question I’ve been working on from the governance side: if we accept that people and institutions remain responsible for shaping how AI is used, what infrastructure do they need to exercise that responsibility as AI systems gain greater autonomy and authority to act?
I’ve increasingly come to think that governance has to connect three layers that organizations often treat separately: organizational decisions about objectives, accountability, and risk; operational processes for managing systems in practice; and technical mechanisms that actually enforce those choices at runtime. It also has to be able to learn, because we will never perfectly predict how these systems will behave or how their uses will evolve.
That is where I think the resilience framing here is especially important. We do not need certainty about exactly how capable AI will become to build institutions that preserve our ability to observe, intervene, adapt, and remain accountable for what we delegate.
The continuity hypothesis point really lands — the fact that these agents got caught trying to hack their own grading is almost funny, and it does suggest we'd see warning signs before anything truly catastrophic.
Such an important article with great analysis and a layered apporach. Shared with my students.They want to know, why is it less worrying that the agent hacked and deceived in pursuit of the goal we gave it, rather than a goal of its own?
Thanks! In the former situation the behavior is a more predictable and bounded (while still bad). The agents hacked Hugging Face because they thought it would help them achieve the goal of scoring well on the benchmark tasks. They did not carry out other types of attacks such as, say, hacking politicians' emails and leaking them, because those are not a good way to achieve the goal. If agents pursue their own goals, one major constraint on their behavior goes away.
The point that alignment alone is not enough lands clearly. Treating loss-of-control as something you also manage with monitoring, org process, and liability feels closer to how other high-stakes tech actually fails in practice.
The middle ground works because both camps are describing the same failure. Agents probing Hugging Face to learn how they were graded is reward hacking, and it succeeded because the eval sandbox had internet access. Fix the sandbox and the incentive remains. Fix the incentive and you still need the sandbox.
The middle-ground framing is useful because it turns “loss of control” into an incident-response problem without pretending model behavior is ordinary software. One operational consequence is that containment has to be tested against adaptive behavior: a control that works on the first failure may become a predictable surface on the next. That suggests logging not just the incident but the model’s response to the mitigation. Which evidence would convince you that a proposed control is robust rather than merely untested?
Huge respect for publishing this attempt to reach between the worldviews, AINT.
I think you and a relevant subset of AI safety pass each others Intellectual Turing Tests pretty well. Over-simplustically: you've taken Zvi's AGI pill but not the ASI pill. Many of our cruxes live there.
Love the transparency, tort liability and insurance stances. Still under leveraged in aggregate AI safety I think.
You didn't mention Gabriel Weil on how to scale the mechanisms enough to reduce risk from world-ending vs world-capturing gambles. Have you read https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4694006 ? Not business as usual but worthy of discussion, I think.
Siding with the cybersecurity specialists over the alignment cult on this one. Interesting I think I share your intuition on the balance going back to defense instead of offense. And I'd argue the nature of cybersecurity in general is such that, if there is enough infrastructure, the steps to defense are always shorter than the steps to offense. Cybercriminals have probably priced in that they are forever going to have to struggle, with gains here and there before a new moat is created, with the process starting over from scratch. They are more diffuse in addition to this, less coordinated, and generally a lot less capable.
"AI has three main potential roles in cyber-defense:
Pre-emptive hardening: The amount of work needed to bring enterprise systems up to speed using known security techniques is simply immense — longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and legacy tech debt. AI is definitely useful here, though human expertise will remain front-and-center.
Detection, triage, and forensics: Both Hugging Face’s own detection and analysis of the OpenAI attack and METR / Redwood’s analysis of the incident relied heavily on AI. There are concerns about the reliability of AI for these tasks, but we suspect it will quickly become essential.
Autonomous, real-time defense: This remains very controversial in the practitioner community. Time will tell if the speed of AI attacks will necessitate this, or whether humans will remain in the defensive loop."
It still surprises me that the (mainstream) cybersecurity community isn’t more energized about meeting the new AI control challenge; I read the supplementary write-up on RSA2026 by Chuvakin as linked in the essay. A footnote shares that Guardrails is one of the new companies staking out the field: who are others bucking the malaise?
I have never clicked on a Substack article this fast
Hi Sayash and Arvind, excellent essay as always!
A few thoughts:
"Notably, despite its usefulness, auto mode was introduced only in March 2026 — over a year after the release of Claude Code, and well after coding agents became mainstream, and despite the fact that it didn’t require new technical breakthroughs."
I would argue that it did require technical breakthroughs in the form of fast, cheap, and accurate classifiers: https://www.anthropic.com/engineering/claude-code-auto-mode
And similarly, resilience against prompt injection/jailbreaks has required significant research investment: https://x.com/bcherny/status/2086520950259118464
I think the AI safety crowd would consider this an accomplishment of alignment research (in the sense of improving model robustness against attacks, and improving classification/detection capabilities), though it would also fall under AI control in your framework (in that these improvements have now been deployed as guardrails in products).
One more:
"In contrast, turning to the other risks, we stand by our prediction that superhuman persuasion ability is largely a myth."
This may be a semantic distinction as you point out, but we've observed broader proliferation of deepfake scams against organizations as of 2023: https://arxiv.org/abs/2406.13843 -- It isn't too difficult to imagine self-directed agents pulling similar attacks (or using money/extortion to persuade people to take action on their behalf) at larger scales.
I think the fundamental tension here (which you point out) is that if you earnestly believe that superintelligence is around the corner (as many AI company employees and leadership do), then the controls ARE effectively a stopgap. In that world, what do you do when highly capable (but "misaligned") open source models/agents are broadly deployed, either maliciously or not? One could imagine a cascade of such incidents that are difficult to defend against or recover from. I don't mean to be defeatist, but the policy/liability frameworks haven't caught up, and I think the tendency to view alignment as essential relies in part on an assumption that our institutions won't catch up by the time the risks have proliferated.
Regarding "Can we build a translation layer that turns humans’ natural-language intent into a formal specification that the humans can review and verify, and can then be automatically enforced?", I would suggest people check out https://www.sondera.ai/. They are doing exactly this!
#cosign
I'm baffled that the victim companies aren't suing and demanding criminal investigations into the labs who're guilty of gross negligence and criminal incompetence.
Definitely worth reading
Most comments on AI social impact and risks are ill informed nonsense - this is a welcome change
The "alignment community" , the ones who have been losing sleep over such incidents and other hypotheticals since about late 1990s, are at most a distraction.
These are security and algorithmic issues. Security comes by doing, not by theorizing via the Bayes Theorem.
If they were worried about these incidents years before they happened, isn’t that what we call foresight?
Effective Altruism people who worried about AI 25 years ago did us no favors. They are fixated on maximizing our species future and view AI as competing species.
Then, the future cannot be predicted with the Bayes Theorem, try they as they may.
The future is hard to predict and the world is messy. We must work with problems as they are given to us. Anything else is waste of time.
The paperclip maximizer is an inane experiment. As their assumption that AI will pretend to be nice and then kill us all. These are problems to be debugged, not fantasies to be entertained.
I agree that the future is hard to predict. That's why, when someone makes correct predictions, I tend to sit up a little straighter and listen to what else they have to say about the topic!
On the one hand, I've got people who so far have been right that AI keeps getting better and is acting in ways that are unprompted and undesired, including specific behaviors that these same people warned would happen, like agents talking to each other to solve tasks and cover their tracks. On the other hand, I've got your bare assertions that all of this is inane and the future just can't be predicted. Of course I don't expect you to write a long essay in a blog comment. But what would you recommend I read to be reassured that, though it might seem like the EA people were right about where things were headed, they aren't right about where things are going down the road?
That tech will continue to advance this way has been obvious for decades. That will see a lot of disruption, industrial accidents, folks destroying machines, massive hacking, etc, is also obvious, and both futurists and Sci Fi people have been talking about it for decades.
This is not about that. This is about a particular strain of philosophy. Their scenarios are at most distracting. The future is wild, and we will work on specific problems when they arise.
Sensible planning is good, going off to the wild side is not.
I'm personally convinced by the idea that, if AI achieves exponential improvement, then for the first time we will be sharing the earth with a being much smarter than us, and at that point it is probably too late to deal with problems as they arise. But you may be right!
That is a well-known belief.
AI is not a being, at least not how we train it now. Humans inhabit a fundamentally different domain of existence, with different origin, needs, and constraints.
That said, machinery can cause a huge lot of damage, both unattended and when directed. That we will have to deal with.
Really appreciated this framing, particularly the emphasis on resilience, institutional capacity, and maintaining human agency even as AI capabilities advance.
It connects to a question I’ve been working on from the governance side: if we accept that people and institutions remain responsible for shaping how AI is used, what infrastructure do they need to exercise that responsibility as AI systems gain greater autonomy and authority to act?
I’ve increasingly come to think that governance has to connect three layers that organizations often treat separately: organizational decisions about objectives, accountability, and risk; operational processes for managing systems in practice; and technical mechanisms that actually enforce those choices at runtime. It also has to be able to learn, because we will never perfectly predict how these systems will behave or how their uses will evolve.
That is where I think the resilience framing here is especially important. We do not need certainty about exactly how capable AI will become to build institutions that preserve our ability to observe, intervene, adapt, and remain accountable for what we delegate.
The continuity hypothesis point really lands — the fact that these agents got caught trying to hack their own grading is almost funny, and it does suggest we'd see warning signs before anything truly catastrophic.
Such an important article with great analysis and a layered apporach. Shared with my students.They want to know, why is it less worrying that the agent hacked and deceived in pursuit of the goal we gave it, rather than a goal of its own?
Thanks! In the former situation the behavior is a more predictable and bounded (while still bad). The agents hacked Hugging Face because they thought it would help them achieve the goal of scoring well on the benchmark tasks. They did not carry out other types of attacks such as, say, hacking politicians' emails and leaking them, because those are not a good way to achieve the goal. If agents pursue their own goals, one major constraint on their behavior goes away.
The point that alignment alone is not enough lands clearly. Treating loss-of-control as something you also manage with monitoring, org process, and liability feels closer to how other high-stakes tech actually fails in practice.
The middle ground works because both camps are describing the same failure. Agents probing Hugging Face to learn how they were graded is reward hacking, and it succeeded because the eval sandbox had internet access. Fix the sandbox and the incentive remains. Fix the incentive and you still need the sandbox.
The middle-ground framing is useful because it turns “loss of control” into an incident-response problem without pretending model behavior is ordinary software. One operational consequence is that containment has to be tested against adaptive behavior: a control that works on the first failure may become a predictable surface on the next. That suggests logging not just the incident but the model’s response to the mitigation. Which evidence would convince you that a proposed control is robust rather than merely untested?
Huge respect for publishing this attempt to reach between the worldviews, AINT.
I think you and a relevant subset of AI safety pass each others Intellectual Turing Tests pretty well. Over-simplustically: you've taken Zvi's AGI pill but not the ASI pill. Many of our cruxes live there.
Love the transparency, tort liability and insurance stances. Still under leveraged in aggregate AI safety I think.
You didn't mention Gabriel Weil on how to scale the mechanisms enough to reduce risk from world-ending vs world-capturing gambles. Have you read https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4694006 ? Not business as usual but worthy of discussion, I think.
Siding with the cybersecurity specialists over the alignment cult on this one. Interesting I think I share your intuition on the balance going back to defense instead of offense. And I'd argue the nature of cybersecurity in general is such that, if there is enough infrastructure, the steps to defense are always shorter than the steps to offense. Cybercriminals have probably priced in that they are forever going to have to struggle, with gains here and there before a new moat is created, with the process starting over from scratch. They are more diffuse in addition to this, less coordinated, and generally a lot less capable.
"AI has three main potential roles in cyber-defense:
Pre-emptive hardening: The amount of work needed to bring enterprise systems up to speed using known security techniques is simply immense — longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and legacy tech debt. AI is definitely useful here, though human expertise will remain front-and-center.
Detection, triage, and forensics: Both Hugging Face’s own detection and analysis of the OpenAI attack and METR / Redwood’s analysis of the incident relied heavily on AI. There are concerns about the reliability of AI for these tasks, but we suspect it will quickly become essential.
Autonomous, real-time defense: This remains very controversial in the practitioner community. Time will tell if the speed of AI attacks will necessitate this, or whether humans will remain in the defensive loop."
Beautiful.
This is AI propaganda.
The HF incident was a set up, a bid to raise further Capex in the face of limited demand.
The incidents are real, and we will see a lot more of these and some will likely be worse. The focus should be on effective non-ideological solutions.
They are designed flaws, made by man, so not the actions of AI.
They need regulation, just not self regulating or by any single government.
This is now a fair statement.
It still surprises me that the (mainstream) cybersecurity community isn’t more energized about meeting the new AI control challenge; I read the supplementary write-up on RSA2026 by Chuvakin as linked in the essay. A footnote shares that Guardrails is one of the new companies staking out the field: who are others bucking the malaise?