An eminently reasonable position. Aside from their failure to think seriously about non-catastrophic risks, it also seems to me the AI safety crowd are largely unconcerned with what I would term the mundane problems of AI, i.e. that this technology has the potential to make the quotidian business of life worse in innumerable small ways.
Shouldn't we balance that against this technology has the potential to make the quotidian business of life better in innumerable small ways? Which, truthfully, has been my experience so far.
Not for the people who experience that everyday impact. And this is not a zero sum game, we can extend the tent to tackle both types of problems instead of rejecting people who come with different concerns like societal impact or ethics.
if you expect AI to eliminate the people who experience that everyday impact in the next two years, then yes, that everyday impact is less important. and yes, it may very well be zero-sum. a lot of these "mundane risks" can be addressed by just devoting some compute from a more capable AI-- which, in turn, actively makes the x-risk worse.
If one concern is 1000x more important than the other, I suspect the people holding the view I mentioned would agree we can give the lesser concern 1/1001 of the resources to resolve.
Not really, the fact that there’s a small chance a car might explode doesn’t mean we shouldn’t worry about equipping it with seatbelts and turn signals?
Additionally, it seems to me that if one is worried about “alignment“ (which this post does a pretty good job dismantling, but let’s assume it is a problem) it seems far less likely to me that a society that has seen its economic and political systems degraded by this technology is likely to avoid the catastrophic problems either. The systems that have been hollowed out by AI‘s mundane problems are very unlikely to muster an appropriate response to catastrophic problems. Someone who takes catastrophic risk seriously should see mundane risk as a step down the wrong path.
The economic and political systems probably won’t be degraded by AI to that large of an extent, if it happens within 2 years. Diffusion is a real deal, and so is fast takeoff.
And the explicit belief of those people is that the probability is not “a small chance”, but overwhelmingly likely. The “small chance” is muddling through.
I'm definitely in the camp of strong collaboration between cybersecurity and AI safety groups. I do have to take issue with the characterization that the cybersecurity community doesn't think catastrophic risk is even worth thinking about.
Many cybersecurity practitioners, including myself, entered the field to address these very risks.
Inspired by folks like Peter Neumann, who identified the need for security as "especially critical for systems whose failure would result in extreme risk to the public"* in 1985, I knew that we had a mission to try to reduce those risk. (I subscribed to RISKS Digest when I went to college in 1991...)
Dan Geer is a prolific writer and scholar on this type of risk who I worked with at @stake. In 2007, he published the controversial paper regarding monoculture resulting in catastrophic risk. **
You did highlight some of the networking concentration risk, but I've been involved in lots of hardening projects on some of the most sensitive protocols (such as BGP) where we clearly saw the catastrophic risk angle if parts of the critical infrastructure were not appropriately hardened.
Finally, we have established numerous incident response and sharing organizations - starting with FIRST in 1990 in order to better respond when and if a particularly dangerous issue arose. In fact, some of the "limited impact" you mentioned regarding worms was limited because those practitioners had prepared to respond to these types of incidents.
Thanks for writing this -- it is really wild to me, with how early AI is as a technology, that our policy circles are hearing from one loud and vocal subset. I suspect, come 6 months or so, when the AI Safety communities predictions haven't all come true, that a lot of policy and other power circles will wake up and expand their search a bit.
It's a cycle we've kind of seen before, where the AI Safety community gets very close to power, but then it's timed wrong, so there's a big walk back and skepticism. I think this oscillation is indicative of some failure in their culture. See y'all soon IRL!
Indeed, the "AI safety" movement who claim to have their day in the sun now, are an ideological movement with a very lengthy legendarium. What we are seeing is industrial accidents, not "AI wants your atoms", as Yud used to claim.
The good news is that as the issues are becoming real, rather than some p(doom), so are the solutions.
"What would you expect to see, at the current capability level, if the doomer position were correct?"
I'll take a stab. A well-articulated doomer position is an "industrial takeoff" by 2032 or so, where AI becomes completely self-sustaining. I asked Ajeya Cotra if it meant "full manufacturing of AI hardware" and she said "yes".
So here's what I'd expect to see by now: an autonomous robot that can diagnose and replace a clogged slurry tube in a machine that grinds wafers in a fab.
You would expect a gap of 4 years between that and takeoff?
That seems way too high on its face. The task you are describing is notoriously difficult for current AI, both because it requires manipulating real-world environments (where we have far less useful training data) and because it’s a far less verifiable domain than translation, programming, mathematics, and cyberhacking.
If we get to the point where AI can succeed at a task like that, we will have overcome the overwhelming majority of the hurdles preventing AI from being deployed in all domains (as per the way these techniques grow, through the Bitter Lesson). A gap of 4 years AFTER that seems far too long.
It's far more likely we obtain industrial takeoff BEFORE this task ever gets accomplished, and the capability for it occurs after it has disempowered those that stand in the way of its goals.
I'm not trying to pull rank here, but I have to ask... do you work in robotics? or manufacturing in general?
The reason I ask is this: once this capability has been demonstrated, it's still some years away from mass production and reliable deployment, which I would expect a manufacturing expert to know. Also, this is just an example of a key missing capability, that I think would harbinger a possible industrial takeover. The robot may be able to replace the slurry line, but still unable to figure out why the yield from fab building 18 is going down due to an edge pattern of failures, get the relevant equipment fixed, which may be far more complex than a single line, etc. etc.
In short, my plumber robot is necessary, but not at all sufficient, for an industrial takeover. When this robot shows up, the industrial takeover may be five or still fifty years away; without it, it is some unknown long time away.
The problem with the doomer position is that they start with some valid premise, then they build a giant castle on top of it. The higher up you go their deduction chain, the more you lose grip with reality.
Truth is, nobody really knows. We can't theorize these things. Willfully malignant AI, abstract rare paths through the decision space that kill us all while respecting all the rules, is something not as much of concern as more pedestrian type of problems. We'll address all these as we go.
You did not really answer my question, which is fine.
Instead, you boldly asserted that catastrophic risk is "not as much of concern as more pedestrian type of problems". What is that based on? Even assuming your frame of "nobody really knows" (which is incorrect, but for purposes of this dialogue, it's fine), how would you know enough to assert that?
That is based on common sense. You work with the cards you are given, one day at a time.
I did address your question. Current issues do not validate the doomer positions. One can be a very focused pragmatic kind of person and still see tools cause problems and likely it will get worse.
What Doom folks say is possible. Not likely. Not most pressing. I also don't buy the self-recursive improvement that will make us lose control in a nanosecond, superintelligence we can't fathom, and other wild extrapolations. We shall live and see.
"despite the fact that credible causal pathways haven’t been articulated (and attempts to do so are unconvincing to say the least)."
It is remarkable to me that you persist in saying this, despite all the evidence to the contrary. You are academics; presumably you have learned to do a literature search at some point in your careers?
Joe Carlsmith published a piece in 2022, titled "Is Power-Seeking AI an Existential Risk?" You can find it at arXiv:2206.13353. Richard Ngo submitted "The Alignment Problem from a Deep Learning Perspective" to ICLR 2024. "Gradual Disempowerment" is from 2025. "Classification of Global Catastrophic Risks Connected with Artificial Intelligence" by Turchin and Denkenberger is from 2020. Kaj Sotala's "Disjunctive Scenarios of Catastrophic AI Risk" from 2018, published in "Artificial Intelligence Safety and Security" analyzes this directly. "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (2008, in Global Catastrophic Risks, Oxford UP), by Yudkowsky, details the more "sci-fi" version of it. Bostrom details a specific takeover path in chapter 6 of Superintelligence.
Seriously, what are we doing here? Those sentences don't pass a sniff test, and any reviewer would call them out immediately. I don't expect your audience to be familiar with all (or even any) of this, but you are the ones staking out a concrete position, supposedly based on knowing what you are talking about. You have a duty to your readers to do better.
What do you mean by “empirical”? If you mean “are they studying real-life events where powerful AI killed all of humanity”, no, they are not. If you wait until that has happened to see if it will happen, by that point you are dead.
When the Manhattan Project worked on designing the nuclear bomb, there was a concern that releasing it could cause a chain reaction so powerful it would ignite the entire atmosphere and kill everyone on Earth. Scientists did hard work on computing the likelihood of this (even though it had never occurred before) and wrote the LA-602 report on it.
That’s what it means to work hard in advance for a potentially devastating problem you will face later on. The good news for them was that they had a much better understanding of the science of nuclear reactions than we do now the science of alignment. That’s bad news for us.
And there is work in this field comparable in rigour, scientific grounding, and data to the physicists doing calculations about the effects of the atom bomb?
That’s the biggest problem we have. We understand so little of alignment as a science that we don’t even have an established paradigm, in the Kuhnian sense.
That is an (extremely strong) argument for slowing down and doing research, not for speeding up ahead. Especially given the evidence we have already seen about real life misbehavior from models.
It's a little difficult to build a bigger tent with the very people who build the AI to consider you a "low quality data source". The people who built this thing get to decide the metrics of whose cultures are superior and inferior to train the models on. What data they upscale.
That means there just is no tent for the people the models think are inferior. I don't really see how that design flaw gets solved. But it's not like I would not be very amused to hear solutions. I'm tired of the hallucinations, the delusion, the biases, and the tone policing because an EAs politics are programmed into my chatbot.
“The cybersecurity community has the right skillset to do that, but unfortunately they don’t seem to find this even worth thinking about.” As someone who works in the cybersecurity space, Professor Narayanan, I completely agree. We need to fundamentally rethink how to “do” security with the new amplified risk that AI agents pose! I sometimes feel alone in voicing these concerns. Super frustrating.
As far as I can tell there is no sincere AI safety movement with much influence. There is at best a call from frontier labs to help write their own regulations (that’s not how you get real safety), ideas to temporarily slow development before speeding up again, suggestions to curb open-weights competition in the US market (see Anthropic), and pretty much no talk whatsoever from the labs or hyperscalers about mitigating AI’s near-term environmental and social damages.
I know there are safety experts out there who want all the rights things, but they do not appear to have a seat at the table.
I've read several similar calls to cybersecurity action in the past couple of weeks, but they are oddly light on references to any concrete initiatives in that direction. Some people should do something, but who are the people we think should be stepping up? Academics? Government figures? The people who give talks about funky fun attacks at DEFCON? NIST? As far as I know, there is not even some temporary think tank trying to write a white paper and get a bunch of security people to sign on and publicize it. Why is that?
(Or, if something like that is actually happening, please point me to it!)
L'analogie avec la preparedness pandemique est eclairante. En entreprise, on retrouve exactement ce biais : les budgets vont a la remediation post-incident, jamais a la prevention. Le cadre big-tent a le merite de rassembler des profils qui, individuellement, ne se seraient pas mobilises sur la securite IA. La question est de savoir si l'approche ISO 42001 peut offrir ce socle commun que l'article appelle de ses voeux, sans tomber dans le piege de la dilution des priorites.
i dont know. could ‘AI as normal technology’ become a new form of complacency? You're right to challenge the certainty of doom, but what evidence do we have that existing institutions can contain the risks? It's a tricky question.
"AI is an amplifier of existing systemic risks" is the sentence I would hand to every hospital board. Medicine is where big-tent safety is easiest to see: the harms are mundane, measurable and already here, from a widely deployed sepsis model that caught 7 in 100 of the cases it was built to find to clinicians whose skills degrade when the tool is usually right. That second harm is Kasirzadeh's accumulative risk in clinical form, and no ban on superintelligence touches it.
That question extends to enterprise learning too. Periodic, low-stakes simulations without the assistant could show whether people still recognize an exception and know when to ask for help. Faster work today can hide a loss of judgment that only becomes visible when the tool is wrong.
The point about cybersecurity having no culture of modeling systemic or catastrophic risks is a good one. Lloyd's refusing to insure severe state-backed attacks is a striking data point
Everything about this piece is so reasonable it begs the question of how anyone could find fault with it. There’s certainly an element of identity with extreme doomers; finding identity with others fixated on not easily addressable existential issues while overlooking within reach immediate concerns visible to anyone. But that is a now common trait of our politics.
Is there really an EA group saying mundane concerns don't matter? I haven't seen that. I have seen reactions to be dismissed or minimized. I think this piece itself dismisses and minimizes. Seems rather fair to react to that. I've seen statements that alignment is "the most important". But this seems fair too. A big tent should have room for different priorities. It shouldn't waste time debating them if the goal is shared recognition, but a lack of debate doesn't require silence.
I think both are very important, thus I've written about both. But I definitely are more immediately concerned with resilience and cybersecurity.
Maybe I haven't been paying attention and you are being attacked for being concerned with those. But I think this newsletter has sometimes launched the first volley, so I wonder if you're merely observing the return.
Your pluralism point stuck with us, because the x-risk framing feels like a conversation one layer above where the risks live. Summits and declarations are the institution layer; cyber, manipulation, resilience are infrastructure and interface problems. We've been arguing at GlobalStack that the race is shifting from who builds the smartest models to who can audit and control them ("Whose Stack Will the World Run On?", September 21).
An eminently reasonable position. Aside from their failure to think seriously about non-catastrophic risks, it also seems to me the AI safety crowd are largely unconcerned with what I would term the mundane problems of AI, i.e. that this technology has the potential to make the quotidian business of life worse in innumerable small ways.
Shouldn't we balance that against this technology has the potential to make the quotidian business of life better in innumerable small ways? Which, truthfully, has been my experience so far.
Both are real. I would just want someone checking the small failures too, since they never get a headline.
It seems to me that the small and large failures are getting an awfully lot of headlines these days.
True, and the small ones are often the better read since they show where the cracks start.
If you expect AI to become superhuman in the next two years, the mundane problems become far less relevant.
Not for the people who experience that everyday impact. And this is not a zero sum game, we can extend the tent to tackle both types of problems instead of rejecting people who come with different concerns like societal impact or ethics.
Support queues and hiring filters are where the small stuff shows up first. Those rarely make headlines but they touch a lot of people.
if you expect AI to eliminate the people who experience that everyday impact in the next two years, then yes, that everyday impact is less important. and yes, it may very well be zero-sum. a lot of these "mundane risks" can be addressed by just devoting some compute from a more capable AI-- which, in turn, actively makes the x-risk worse.
If one concern is 1000x more important than the other, I suspect the people holding the view I mentioned would agree we can give the lesser concern 1/1001 of the resources to resolve.
Not really, the fact that there’s a small chance a car might explode doesn’t mean we shouldn’t worry about equipping it with seatbelts and turn signals?
Additionally, it seems to me that if one is worried about “alignment“ (which this post does a pretty good job dismantling, but let’s assume it is a problem) it seems far less likely to me that a society that has seen its economic and political systems degraded by this technology is likely to avoid the catastrophic problems either. The systems that have been hollowed out by AI‘s mundane problems are very unlikely to muster an appropriate response to catastrophic problems. Someone who takes catastrophic risk seriously should see mundane risk as a step down the wrong path.
The economic and political systems probably won’t be degraded by AI to that large of an extent, if it happens within 2 years. Diffusion is a real deal, and so is fast takeoff.
And the explicit belief of those people is that the probability is not “a small chance”, but overwhelmingly likely. The “small chance” is muddling through.
I'm definitely in the camp of strong collaboration between cybersecurity and AI safety groups. I do have to take issue with the characterization that the cybersecurity community doesn't think catastrophic risk is even worth thinking about.
Many cybersecurity practitioners, including myself, entered the field to address these very risks.
Inspired by folks like Peter Neumann, who identified the need for security as "especially critical for systems whose failure would result in extreme risk to the public"* in 1985, I knew that we had a mission to try to reduce those risk. (I subscribed to RISKS Digest when I went to college in 1991...)
Dan Geer is a prolific writer and scholar on this type of risk who I worked with at @stake. In 2007, he published the controversial paper regarding monoculture resulting in catastrophic risk. **
You did highlight some of the networking concentration risk, but I've been involved in lots of hardening projects on some of the most sensitive protocols (such as BGP) where we clearly saw the catastrophic risk angle if parts of the critical infrastructure were not appropriately hardened.
Finally, we have established numerous incident response and sharing organizations - starting with FIRST in 1990 in order to better respond when and if a particularly dangerous issue arose. In fact, some of the "limited impact" you mentioned regarding worms was limited because those practitioners had prepared to respond to these types of incidents.
* https://catless.ncl.ac.uk/Risks/1/1#subj1
** http://geer.tinho.net/acm.geer.0704.pdf
Thanks for sharing that history. It also shows the safety work that works is mostly invisible, so nobody notices it until it is missing.
Thanks for writing this -- it is really wild to me, with how early AI is as a technology, that our policy circles are hearing from one loud and vocal subset. I suspect, come 6 months or so, when the AI Safety communities predictions haven't all come true, that a lot of policy and other power circles will wake up and expand their search a bit.
It's a cycle we've kind of seen before, where the AI Safety community gets very close to power, but then it's timed wrong, so there's a big walk back and skepticism. I think this oscillation is indicative of some failure in their culture. See y'all soon IRL!
Companies are already pausing the releases of their more advanced internal models because of their inability to properly secure and safeguard it (https://www.npr.org/2026/09/29/nx-s1-5984342/openai-delays-latest-model).
Seems likely that, absent intervention and regulation, the AI safety community's predictions will prove correct.
Policy people usually listen to whoever shows up first with a clear ask. Being loud and early counts for a lot.
Indeed, the "AI safety" movement who claim to have their day in the sun now, are an ideological movement with a very lengthy legendarium. What we are seeing is industrial accidents, not "AI wants your atoms", as Yud used to claim.
The good news is that as the issues are becoming real, rather than some p(doom), so are the solutions.
"What we are seeing is industrial accidents, not "AI wants your atoms", as Yud used to claim."
What would you expect to see, at the current capability level, if the doomer position were correct?
"What would you expect to see, at the current capability level, if the doomer position were correct?"
I'll take a stab. A well-articulated doomer position is an "industrial takeoff" by 2032 or so, where AI becomes completely self-sustaining. I asked Ajeya Cotra if it meant "full manufacturing of AI hardware" and she said "yes".
So here's what I'd expect to see by now: an autonomous robot that can diagnose and replace a clogged slurry tube in a machine that grinds wafers in a fab.
You would expect a gap of 4 years between that and takeoff?
That seems way too high on its face. The task you are describing is notoriously difficult for current AI, both because it requires manipulating real-world environments (where we have far less useful training data) and because it’s a far less verifiable domain than translation, programming, mathematics, and cyberhacking.
If we get to the point where AI can succeed at a task like that, we will have overcome the overwhelming majority of the hurdles preventing AI from being deployed in all domains (as per the way these techniques grow, through the Bitter Lesson). A gap of 4 years AFTER that seems far too long.
It's far more likely we obtain industrial takeoff BEFORE this task ever gets accomplished, and the capability for it occurs after it has disempowered those that stand in the way of its goals.
I'm not trying to pull rank here, but I have to ask... do you work in robotics? or manufacturing in general?
The reason I ask is this: once this capability has been demonstrated, it's still some years away from mass production and reliable deployment, which I would expect a manufacturing expert to know. Also, this is just an example of a key missing capability, that I think would harbinger a possible industrial takeover. The robot may be able to replace the slurry line, but still unable to figure out why the yield from fab building 18 is going down due to an edge pattern of failures, get the relevant equipment fixed, which may be far more complex than a single line, etc. etc.
In short, my plumber robot is necessary, but not at all sufficient, for an industrial takeover. When this robot shows up, the industrial takeover may be five or still fifty years away; without it, it is some unknown long time away.
The problem with the doomer position is that they start with some valid premise, then they build a giant castle on top of it. The higher up you go their deduction chain, the more you lose grip with reality.
Truth is, nobody really knows. We can't theorize these things. Willfully malignant AI, abstract rare paths through the decision space that kill us all while respecting all the rules, is something not as much of concern as more pedestrian type of problems. We'll address all these as we go.
You did not really answer my question, which is fine.
Instead, you boldly asserted that catastrophic risk is "not as much of concern as more pedestrian type of problems". What is that based on? Even assuming your frame of "nobody really knows" (which is incorrect, but for purposes of this dialogue, it's fine), how would you know enough to assert that?
That is based on common sense. You work with the cards you are given, one day at a time.
I did address your question. Current issues do not validate the doomer positions. One can be a very focused pragmatic kind of person and still see tools cause problems and likely it will get worse.
What Doom folks say is possible. Not likely. Not most pressing. I also don't buy the self-recursive improvement that will make us lose control in a nanosecond, superintelligence we can't fathom, and other wild extrapolations. We shall live and see.
Accidents at least come with playbooks. Post mortems and incident reports are how most industries got safer over time.
"despite the fact that credible causal pathways haven’t been articulated (and attempts to do so are unconvincing to say the least)."
It is remarkable to me that you persist in saying this, despite all the evidence to the contrary. You are academics; presumably you have learned to do a literature search at some point in your careers?
Joe Carlsmith published a piece in 2022, titled "Is Power-Seeking AI an Existential Risk?" You can find it at arXiv:2206.13353. Richard Ngo submitted "The Alignment Problem from a Deep Learning Perspective" to ICLR 2024. "Gradual Disempowerment" is from 2025. "Classification of Global Catastrophic Risks Connected with Artificial Intelligence" by Turchin and Denkenberger is from 2020. Kaj Sotala's "Disjunctive Scenarios of Catastrophic AI Risk" from 2018, published in "Artificial Intelligence Safety and Security" analyzes this directly. "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (2008, in Global Catastrophic Risks, Oxford UP), by Yudkowsky, details the more "sci-fi" version of it. Bostrom details a specific takeover path in chapter 6 of Superintelligence.
Holden Karnofsky wrote about this in 2022, in "AI could defeat all of us combined" (https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).
Ruben Bloom wrote about this more recently, in "Some ways AI could kill us all" (https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-ai-could-kill-us-all).
Paul Christiano wrote about this in 2019, in "What failure looks like" (https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like). Dylan Matthews of Vox (a rather mainstream publication) covered it at the time (https://www.vox.com/future-perfect/2019/3/26/18281297/ai-artificial-intelligence-safety-disaster-scenarios), even though the reception to his piece was rather mixed, so it's not like any of this was secret or hidden. He wrote about it in 2021, in "Another (outer) alignment failure story" (https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story)
Discussions of biorisk have been going on for years (https://defensesindepth.bio/how-i-think-about-catastrophic-biological-risk-part-i/#targeting-human-bodies-vs-agriculture-vs-environment).
Kokotajlo and the AI futures team have written about this at length, including in 2020 (https://www.lesswrong.com/posts/JPan54R525D68NoEt/the-date-of-ai-takeover-is-not-the-day-the-ai-takes-over) and more recently, with AI-2027 (https://ai-2027.com/) and all the literature surrounding it.
Discussions of multipolar disempowerment situations have included contributions from academics like Andrew Critch, who has an entire research program on this exact topic. A sample of it is here: https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic.
More narrative depictions of what unfolds have numbered Gwern's "Clippy" (https://gwern.net/Clippy), Gabriel Mukobi's "Scale Was All We Needed, At First" (https://aiacumen.substack.com/p/scale-was-all-we-needed-at-first), and Joshua Klimer's "How AI May Take Over In Two Years" (https://x.com/joshua_clymer/status/1887905375082656117).
Seriously, what are we doing here? Those sentences don't pass a sniff test, and any reviewer would call them out immediately. I don't expect your audience to be familiar with all (or even any) of this, but you are the ones staking out a concrete position, supposedly based on knowing what you are talking about. You have a duty to your readers to do better.
Any of those articles any more than speculation? Any of them empirical? A lot predate modern AI!
What do you mean by “empirical”? If you mean “are they studying real-life events where powerful AI killed all of humanity”, no, they are not. If you wait until that has happened to see if it will happen, by that point you are dead.
When the Manhattan Project worked on designing the nuclear bomb, there was a concern that releasing it could cause a chain reaction so powerful it would ignite the entire atmosphere and kill everyone on Earth. Scientists did hard work on computing the likelihood of this (even though it had never occurred before) and wrote the LA-602 report on it.
That’s what it means to work hard in advance for a potentially devastating problem you will face later on. The good news for them was that they had a much better understanding of the science of nuclear reactions than we do now the science of alignment. That’s bad news for us.
And there is work in this field comparable in rigour, scientific grounding, and data to the physicists doing calculations about the effects of the atom bomb?
No, nowhere close to it.
That’s the biggest problem we have. We understand so little of alignment as a science that we don’t even have an established paradigm, in the Kuhnian sense.
That is an (extremely strong) argument for slowing down and doing research, not for speeding up ahead. Especially given the evidence we have already seen about real life misbehavior from models.
It's a little difficult to build a bigger tent with the very people who build the AI to consider you a "low quality data source". The people who built this thing get to decide the metrics of whose cultures are superior and inferior to train the models on. What data they upscale.
That means there just is no tent for the people the models think are inferior. I don't really see how that design flaw gets solved. But it's not like I would not be very amused to hear solutions. I'm tired of the hallucinations, the delusion, the biases, and the tone policing because an EAs politics are programmed into my chatbot.
“The cybersecurity community has the right skillset to do that, but unfortunately they don’t seem to find this even worth thinking about.” As someone who works in the cybersecurity space, Professor Narayanan, I completely agree. We need to fundamentally rethink how to “do” security with the new amplified risk that AI agents pose! I sometimes feel alone in voicing these concerns. Super frustrating.
As far as I can tell there is no sincere AI safety movement with much influence. There is at best a call from frontier labs to help write their own regulations (that’s not how you get real safety), ideas to temporarily slow development before speeding up again, suggestions to curb open-weights competition in the US market (see Anthropic), and pretty much no talk whatsoever from the labs or hyperscalers about mitigating AI’s near-term environmental and social damages.
I know there are safety experts out there who want all the rights things, but they do not appear to have a seat at the table.
I've read several similar calls to cybersecurity action in the past couple of weeks, but they are oddly light on references to any concrete initiatives in that direction. Some people should do something, but who are the people we think should be stepping up? Academics? Government figures? The people who give talks about funky fun attacks at DEFCON? NIST? As far as I know, there is not even some temporary think tank trying to write a white paper and get a bunch of security people to sign on and publicize it. Why is that?
(Or, if something like that is actually happening, please point me to it!)
L'analogie avec la preparedness pandemique est eclairante. En entreprise, on retrouve exactement ce biais : les budgets vont a la remediation post-incident, jamais a la prevention. Le cadre big-tent a le merite de rassembler des profils qui, individuellement, ne se seraient pas mobilises sur la securite IA. La question est de savoir si l'approche ISO 42001 peut offrir ce socle commun que l'article appelle de ses voeux, sans tomber dans le piege de la dilution des priorites.
i dont know. could ‘AI as normal technology’ become a new form of complacency? You're right to challenge the certainty of doom, but what evidence do we have that existing institutions can contain the risks? It's a tricky question.
"AI is an amplifier of existing systemic risks" is the sentence I would hand to every hospital board. Medicine is where big-tent safety is easiest to see: the harms are mundane, measurable and already here, from a widely deployed sepsis model that caught 7 in 100 of the cases it was built to find to clinicians whose skills degrade when the tool is usually right. That second harm is Kasirzadeh's accumulative risk in clinical form, and no ban on superintelligence touches it.
California just chose a big-tent instrument, a duty to preserve the clinician's independent judgment: https://nonalgorithmic.com/2026/10/06/california-kept-clinical-judgment-in-the-law/
What would embedded evaluation look like for a risk that accumulates over a career rather than arriving as an incident?
That question extends to enterprise learning too. Periodic, low-stakes simulations without the assistant could show whether people still recognize an exception and know when to ask for help. Faster work today can hide a loss of judgment that only becomes visible when the tool is wrong.
The point about cybersecurity having no culture of modeling systemic or catastrophic risks is a good one. Lloyd's refusing to insure severe state-backed attacks is a striking data point
Everything about this piece is so reasonable it begs the question of how anyone could find fault with it. There’s certainly an element of identity with extreme doomers; finding identity with others fixated on not easily addressable existential issues while overlooking within reach immediate concerns visible to anyone. But that is a now common trait of our politics.
Is there really an EA group saying mundane concerns don't matter? I haven't seen that. I have seen reactions to be dismissed or minimized. I think this piece itself dismisses and minimizes. Seems rather fair to react to that. I've seen statements that alignment is "the most important". But this seems fair too. A big tent should have room for different priorities. It shouldn't waste time debating them if the goal is shared recognition, but a lack of debate doesn't require silence.
I think both are very important, thus I've written about both. But I definitely are more immediately concerned with resilience and cybersecurity.
Maybe I haven't been paying attention and you are being attacked for being concerned with those. But I think this newsletter has sometimes launched the first volley, so I wonder if you're merely observing the return.
Your pluralism point stuck with us, because the x-risk framing feels like a conversation one layer above where the risks live. Summits and declarations are the institution layer; cyber, manipulation, resilience are infrastructure and interface problems. We've been arguing at GlobalStack that the race is shifting from who builds the smartest models to who can audit and control them ("Whose Stack Will the World Run On?", September 21).