I think your approach is eminently practical. I have a couple of comments:
1. I disagree with Derek Thompson's premise when he says that AI's (LLMs) emergent capabilities justify extraordinary government responses, because it's not at all clear that the properties of LLMs are emergent. I agree that some of the capabilities may remain unknown since their inner workings are still not well understood, but this is not necessarily due to their potential emergence. Melanie Mitchell talks about several reasons why emergence due to scaling is actually unlikely in her 2024 "Large Language Models" article (https://oecs.mit.edu/pub/zp5n8ivs/release/1?readingCollection=9dd2a47d). The most persuasive study mentioned from my point of view states that "the apparent abrupt emergence of such capabilities is an artifact of the evaluation metrics used, not an intrinsic property of scaling (Schaeffer et al., 2024)."
2. I believe that the biggest Achiles heel of large-scale political governance (Federal and big states) is that it takes too long to pass and enact new laws and even longer to see measurable impacts. Most importantly, the only mechanisms to modify a law based on what has been learned through its application is either legislative, which again is not only slow but the political winds might have changed removing necessary support, or judicial, which is just as likely to break it as to improve it. This is very similar to the problem that large companies have releasing successful new products, which is why instead they often chose to acquire smaller companies to absorb or neutralize theirs. Clearly governments can't do this, but they can potentially include iteration and refinement into the implementation mechanisms of the laws, making them outcome oriented rather than simply checking some boxes that can be used for ammunition for future elections. Jennifer Pahlka has picked up on this and compares government culture to the worst type of waterfall projects (the original "waterfall" paper by Winston Royce had plenty of feedback loops in its methodology, but it was misinterpreted by many organizations until today as a linear process where the feedback loop happens at the end, when it's to late to make changes).
The comparison to issues on the regulation of encryption is interesting, but I think it's imperfect. Encryption is a technology that makes it more difficult to centralize power and support an autocratic regime by giving individual users and groups the opportunity to circumvent malevolent forces. Many AI technologies allow a small number of very large companies to do the opposite, giving autocratic regimes incredible power to collect data and manufacture incriminating material to quash dissent. You need look no further than the cozy relationships between ethno-nationalist Elon Musk and Hindu-nationalist Narendra Modi, or between Trump, Goldman Sachs, and MBS in Saudi Arabia to see the potential risks. Let's not forget that Meta inadvertently (at least at first) allowed for and accelerated the commission of crimes of atrocity in Myanmar. The technology is only as moral as its owners, and most of the owners of these technologies already collaborate with war criminals and dictators without batting an eye.
I appreciate the focus on cyber and bio weaponization, but like Dan Emery’s comment above, my biggest practical concern is the more commercially viable use of AI for autocracy and the infinite extraction of our attention, consumer desire, and ideological allegiances, like social media with geometric growth curves. AI works both as a service and as a cause of economic disruption. In this domain, resilience involves building friction around the release of private data, and the micro aggregation of cohorts which can share data internally for mutual benefit projects and internal governance experiments without ceding this informational resource to the central authority structure (corporate or federal). Without such friction, our choice of resilience is Luddite isolation (Ted Kazinsky) or merging with the machinery of production.
What happens when humans progressively stop thinking without AI?
I like your idea of cognitive surrender.
The deeper risk may be the erosion of judgment, contradiction and courage inside institutions.
An organization that delegates too much to AI may become highly efficient at producing answers, while losing part of its ability to ask difficult questions, challenge assumptions, and hold uncomfortable positions under uncertainty.
Resilience depends on these fragile acts: "the model may be wrong”, “the signal may be misleading" or simply: "I disagree.”
AI could make these acts rarer, more costly, and harder to sustain inside organizations because the machine already sounds confident, immediate and scalable.
A robust institution preserves the space where people can still doubt, resist, and take responsibility under pressure.
The focus on resilience over extraordinary inteveny seems to be better overall in the long term.
Creating social expectations that AI companies should be responsible for harms caused by their products can also be a strong financial incentive for AI companies to act quickly to address harms.
An underrated way to reduce harms is by addressing the root cause of many of our problems: economic insecurity. If we can work towards ensuring the productivity gains from AI are widely shared and the basic needs of every human are met (seems plausible by 2050?), this may end up being the most cost effective and robust solution.
Hey Sayash and Arvind — the nuclear analogy breakdown is the part of this conversation that needed saying out loud. There's no uranium enrichment bottleneck for AI. The gap between frontier and public is months, not decades, and shrinking. Trying to build a nonproliferation regime around something that diffuses like software is like trying to embargo a recipe. The resilience framing is way more honest about what's actually possible — and the cybersecurity parallel lands because we watched that exact shift happen in real time. Bug bounties worked better than trying to lock down every vulnerability.
I build AI agents for small businesses and the trust question comes up in every single conversation — clients don't want restrictions, they want guardrails they control. That's resilience at the individual level. The part about federal capacity quietly undermining the whole approach is the uncomfortable truth nobody in policy wants to sit with. Following this publication.
I might add: we could need interim nonproliferation moves quite soon (or even yesterday; cf Glasswing). And some of these adaptation windows for resilience could be quite challenging indeed. Authoritarian creep is a real concern.
Meanwhile, ‘normal policymaking’, and broader societal sensemaking needs an upgrade! Cf AI for Human Reasoning for You
I think your approach is eminently practical. I have a couple of comments:
1. I disagree with Derek Thompson's premise when he says that AI's (LLMs) emergent capabilities justify extraordinary government responses, because it's not at all clear that the properties of LLMs are emergent. I agree that some of the capabilities may remain unknown since their inner workings are still not well understood, but this is not necessarily due to their potential emergence. Melanie Mitchell talks about several reasons why emergence due to scaling is actually unlikely in her 2024 "Large Language Models" article (https://oecs.mit.edu/pub/zp5n8ivs/release/1?readingCollection=9dd2a47d). The most persuasive study mentioned from my point of view states that "the apparent abrupt emergence of such capabilities is an artifact of the evaluation metrics used, not an intrinsic property of scaling (Schaeffer et al., 2024)."
2. I believe that the biggest Achiles heel of large-scale political governance (Federal and big states) is that it takes too long to pass and enact new laws and even longer to see measurable impacts. Most importantly, the only mechanisms to modify a law based on what has been learned through its application is either legislative, which again is not only slow but the political winds might have changed removing necessary support, or judicial, which is just as likely to break it as to improve it. This is very similar to the problem that large companies have releasing successful new products, which is why instead they often chose to acquire smaller companies to absorb or neutralize theirs. Clearly governments can't do this, but they can potentially include iteration and refinement into the implementation mechanisms of the laws, making them outcome oriented rather than simply checking some boxes that can be used for ammunition for future elections. Jennifer Pahlka has picked up on this and compares government culture to the worst type of waterfall projects (the original "waterfall" paper by Winston Royce had plenty of feedback loops in its methodology, but it was misinterpreted by many organizations until today as a linear process where the feedback loop happens at the end, when it's to late to make changes).
The comparison to issues on the regulation of encryption is interesting, but I think it's imperfect. Encryption is a technology that makes it more difficult to centralize power and support an autocratic regime by giving individual users and groups the opportunity to circumvent malevolent forces. Many AI technologies allow a small number of very large companies to do the opposite, giving autocratic regimes incredible power to collect data and manufacture incriminating material to quash dissent. You need look no further than the cozy relationships between ethno-nationalist Elon Musk and Hindu-nationalist Narendra Modi, or between Trump, Goldman Sachs, and MBS in Saudi Arabia to see the potential risks. Let's not forget that Meta inadvertently (at least at first) allowed for and accelerated the commission of crimes of atrocity in Myanmar. The technology is only as moral as its owners, and most of the owners of these technologies already collaborate with war criminals and dictators without batting an eye.
I appreciate the focus on cyber and bio weaponization, but like Dan Emery’s comment above, my biggest practical concern is the more commercially viable use of AI for autocracy and the infinite extraction of our attention, consumer desire, and ideological allegiances, like social media with geometric growth curves. AI works both as a service and as a cause of economic disruption. In this domain, resilience involves building friction around the release of private data, and the micro aggregation of cohorts which can share data internally for mutual benefit projects and internal governance experiments without ceding this informational resource to the central authority structure (corporate or federal). Without such friction, our choice of resilience is Luddite isolation (Ted Kazinsky) or merging with the machinery of production.
What happens when humans progressively stop thinking without AI?
I like your idea of cognitive surrender.
The deeper risk may be the erosion of judgment, contradiction and courage inside institutions.
An organization that delegates too much to AI may become highly efficient at producing answers, while losing part of its ability to ask difficult questions, challenge assumptions, and hold uncomfortable positions under uncertainty.
Resilience depends on these fragile acts: "the model may be wrong”, “the signal may be misleading" or simply: "I disagree.”
AI could make these acts rarer, more costly, and harder to sustain inside organizations because the machine already sounds confident, immediate and scalable.
A robust institution preserves the space where people can still doubt, resist, and take responsibility under pressure.
The focus on resilience over extraordinary inteveny seems to be better overall in the long term.
Creating social expectations that AI companies should be responsible for harms caused by their products can also be a strong financial incentive for AI companies to act quickly to address harms.
An underrated way to reduce harms is by addressing the root cause of many of our problems: economic insecurity. If we can work towards ensuring the productivity gains from AI are widely shared and the basic needs of every human are met (seems plausible by 2050?), this may end up being the most cost effective and robust solution.
Hey Sayash and Arvind — the nuclear analogy breakdown is the part of this conversation that needed saying out loud. There's no uranium enrichment bottleneck for AI. The gap between frontier and public is months, not decades, and shrinking. Trying to build a nonproliferation regime around something that diffuses like software is like trying to embargo a recipe. The resilience framing is way more honest about what's actually possible — and the cybersecurity parallel lands because we watched that exact shift happen in real time. Bug bounties worked better than trying to lock down every vulnerability.
I build AI agents for small businesses and the trust question comes up in every single conversation — clients don't want restrictions, they want guardrails they control. That's resilience at the individual level. The part about federal capacity quietly undermining the whole approach is the uncomfortable truth nobody in policy wants to sit with. Following this publication.
I think I largely agree with this perspective.
I might add: we could need interim nonproliferation moves quite soon (or even yesterday; cf Glasswing). And some of these adaptation windows for resilience could be quite challenging indeed. Authoritarian creep is a real concern.
Meanwhile, ‘normal policymaking’, and broader societal sensemaking needs an upgrade! Cf AI for Human Reasoning for You
https://www.oliversourbut.net/p/ai-for-human-reasoning-for-you?r=lwkfb&utm_campaign=post&utm_medium=web
How much cash is he offering to give people individually if they agree to not use AI?