The comparison to issues on the regulation of encryption is interesting, but I think it's imperfect. Encryption is a technology that makes it more difficult to centralize power and support an autocratic regime by giving individual users and groups the opportunity to circumvent malevolent forces. Many AI technologies allow a small number of very large companies to do the opposite, giving autocratic regimes incredible power to collect data and manufacture incriminating material to quash dissent. You need look no further than the cozy relationships between ethno-nationalist Elon Musk and Hindu-nationalist Narendra Modi, or between Trump, Goldman Sachs, and MBS in Saudi Arabia to see the potential risks. Let's not forget that Meta inadvertently (at least at first) allowed for and accelerated the commission of crimes of atrocity in Myanmar. The technology is only as moral as its owners, and most of the owners of these technologies already collaborate with war criminals and dictators without batting an eye.
I appreciate the focus on cyber and bio weaponization, but like Dan Emery’s comment above, my biggest practical concern is the more commercially viable use of AI for autocracy and the infinite extraction of our attention, consumer desire, and ideological allegiances, like social media with geometric growth curves. AI works both as a service and as a cause of economic disruption. In this domain, resilience involves building friction around the release of private data, and the micro aggregation of cohorts which can share data internally for mutual benefit projects and internal governance experiments without ceding this informational resource to the central authority structure (corporate or federal). Without such friction, our choice of resilience is Luddite isolation (Ted Kazinsky) or merging with the machinery of production.
The focus on resilience over extraordinary inteveny seems to be better overall in the long term.
Creating social expectations that AI companies should be responsible for harms caused by their products can also be a strong financial incentive for AI companies to act quickly to address harms.
An underrated way to reduce harms is by addressing the root cause of many of our problems: economic insecurity. If we can work towards ensuring the productivity gains from AI are widely shared and the basic needs of every human are met (seems plausible by 2050?), this may end up being the most cost effective and robust solution.
Hey Sayash and Arvind — the nuclear analogy breakdown is the part of this conversation that needed saying out loud. There's no uranium enrichment bottleneck for AI. The gap between frontier and public is months, not decades, and shrinking. Trying to build a nonproliferation regime around something that diffuses like software is like trying to embargo a recipe. The resilience framing is way more honest about what's actually possible — and the cybersecurity parallel lands because we watched that exact shift happen in real time. Bug bounties worked better than trying to lock down every vulnerability.
I build AI agents for small businesses and the trust question comes up in every single conversation — clients don't want restrictions, they want guardrails they control. That's resilience at the individual level. The part about federal capacity quietly undermining the whole approach is the uncomfortable truth nobody in policy wants to sit with. Following this publication.
I might add: we could need interim nonproliferation moves quite soon (or even yesterday; cf Glasswing). And some of these adaptation windows for resilience could be quite challenging indeed. Authoritarian creep is a real concern.
Meanwhile, ‘normal policymaking’, and broader societal sensemaking needs an upgrade! Cf AI for Human Reasoning for You
The comparison to issues on the regulation of encryption is interesting, but I think it's imperfect. Encryption is a technology that makes it more difficult to centralize power and support an autocratic regime by giving individual users and groups the opportunity to circumvent malevolent forces. Many AI technologies allow a small number of very large companies to do the opposite, giving autocratic regimes incredible power to collect data and manufacture incriminating material to quash dissent. You need look no further than the cozy relationships between ethno-nationalist Elon Musk and Hindu-nationalist Narendra Modi, or between Trump, Goldman Sachs, and MBS in Saudi Arabia to see the potential risks. Let's not forget that Meta inadvertently (at least at first) allowed for and accelerated the commission of crimes of atrocity in Myanmar. The technology is only as moral as its owners, and most of the owners of these technologies already collaborate with war criminals and dictators without batting an eye.
I appreciate the focus on cyber and bio weaponization, but like Dan Emery’s comment above, my biggest practical concern is the more commercially viable use of AI for autocracy and the infinite extraction of our attention, consumer desire, and ideological allegiances, like social media with geometric growth curves. AI works both as a service and as a cause of economic disruption. In this domain, resilience involves building friction around the release of private data, and the micro aggregation of cohorts which can share data internally for mutual benefit projects and internal governance experiments without ceding this informational resource to the central authority structure (corporate or federal). Without such friction, our choice of resilience is Luddite isolation (Ted Kazinsky) or merging with the machinery of production.
The focus on resilience over extraordinary inteveny seems to be better overall in the long term.
Creating social expectations that AI companies should be responsible for harms caused by their products can also be a strong financial incentive for AI companies to act quickly to address harms.
An underrated way to reduce harms is by addressing the root cause of many of our problems: economic insecurity. If we can work towards ensuring the productivity gains from AI are widely shared and the basic needs of every human are met (seems plausible by 2050?), this may end up being the most cost effective and robust solution.
Hey Sayash and Arvind — the nuclear analogy breakdown is the part of this conversation that needed saying out loud. There's no uranium enrichment bottleneck for AI. The gap between frontier and public is months, not decades, and shrinking. Trying to build a nonproliferation regime around something that diffuses like software is like trying to embargo a recipe. The resilience framing is way more honest about what's actually possible — and the cybersecurity parallel lands because we watched that exact shift happen in real time. Bug bounties worked better than trying to lock down every vulnerability.
I build AI agents for small businesses and the trust question comes up in every single conversation — clients don't want restrictions, they want guardrails they control. That's resilience at the individual level. The part about federal capacity quietly undermining the whole approach is the uncomfortable truth nobody in policy wants to sit with. Following this publication.
I think I largely agree with this perspective.
I might add: we could need interim nonproliferation moves quite soon (or even yesterday; cf Glasswing). And some of these adaptation windows for resilience could be quite challenging indeed. Authoritarian creep is a real concern.
Meanwhile, ‘normal policymaking’, and broader societal sensemaking needs an upgrade! Cf AI for Human Reasoning for You
https://www.oliversourbut.net/p/ai-for-human-reasoning-for-you?r=lwkfb&utm_campaign=post&utm_medium=web
How much cash is he offering to give people individually if they agree to not use AI?