There are growing calls for independent and embedded AI evaluation, especially of AI agents. But much remains unclear: what constitutes a “good” evaluation, what needs to be evaluated in the first place, and how best to coordinate and support the influx of people entering the burgeoning evaluation ecosystem.

Our group at Princeton has been working to improve the state of AI agent evaluation for about three years. We teamed up with people who’ve done foundational work on evaluation at Stanford, the UK AI Security Institute, and Brookings. Our paper is titled Emerging Trends in AI Agent Evaluation. We synthesized the trends we see across three dimensions—the science and methods, the research agenda, and the evaluation ecosystem—with the goal to help those who are upskilling for this new paradigm. We also aim to equip evaluators, funders, and the public with a set of questions to ask, and provide actionable recommendations.

Challenges for agents, new and old

AI developers have long made claims about the real-world performance of their systems based on benchmark scores. It is harder now to assess whether the evidence gathered supports claims made about agents. The evaluator must scrutinize the entire trajectory, not just the final outcome. It is important, for example, to know if agents took shortcuts, particularly since they have access to tools and environments. And when agents perform better or worse, it is harder to attribute these changes to different components of the agent. Addressing these concerns requires significant engineering effort, and the work is getting increasingly expensive—for example, several agents on the ExploitBench benchmark cost tens of thousands of dollars in credits to evaluate. These numbers also don’t factor in the significant engineering effort needed to set up and maintain the evaluation infrastructure itself.

Faced with the massive volume of traces that agents generate, evaluators are also struggling to inspect and monitor these agents at scale. The tools evaluators rely on today to do this work are not foolproof—and themselves require evaluation. To investigate the recent OpenAI-HuggingFace incident after the fact, evaluators relied upon other agents to look for and identify instances of harmful action, an approach they acknowledged carried its own limitations and biases.

Because agents are deployed in dynamic settings (and increasingly together with other agents), it is harder to design good metrics to capture the underlying concepts of interest, and create valid conditions to implement them. Addressing these challenges will require better tools, more rigorous benchmarks, and experimentation with different approaches and formats, such as through open-world evaluations and growing observational data on real-world usage.

The scope of evaluation needs is growing

Crucially, the results of evaluation at one level of scope are not guaranteed to translate to another. An agent might demonstrate capability in the lab, but face limits when placed in a specific use case. Consider METR’s randomized controlled trial which found that, despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand. Agents may be able to perform certain tasks, but how that capability translates for humans in the real world requires evaluation in context. An expanded research agenda is therefore needed to understand agents at each level: their capabilities, their interactions with humans, and their impact on institutions and incentive structures. This work requires both technical and sociotechnical evaluation to study not just the agent itself but the agent(s) in context.

Across these different layers, evaluators have a choice in framing the questions they ask and the findings they generate. Can an agent automate a task or process? Or, can an agent augment the work of a human? These are two complementary lenses. If evaluation largely focuses on how agents automate certain tasks, we will better understand where and how human work can be replaced. But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.

Who gets to evaluate and how?

This expanded research agenda will require an expanded set of evaluators from a mix of disciplines. It will also require that evaluators have the freedom to choose which questions to pursue and how. When evaluators with different approaches agree, we might gain greater confidence in those results. And when they conflict, evaluators can identify vague definitions and limitations to their findings.

Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use. Meaningful independence is influenced by several factors. As the costs of frontier evaluation keep increasing, the ability of independent evaluators to participate will be increasingly determined by their access to increased funding. Whether independent evaluators have sufficient access to models also impacts their ability to measure and assess the true frontier. Questions of access now also extend to Chain of Thought traces for agents, since frontier labs now release summarized versions instead of raw traces. Alongside these challenges, evaluators also need to navigate whether they can conduct work without fear of retaliation, and how they can transparently share their findings.

Where do we go from here?

There is a clear window now to bring in new participants, methods, and ideas for agent evaluations, and to meet the pressing need for rigorous evidence in a fast-moving space. We hope this paper gives readers an emerging sense of the field’s direction. There are many steps that stakeholders in the AI evaluation community can take to advance the science and practice of AI evaluation.

For prospective evaluators, there is a need to determine what roles or questions within this ecosystem best fit their expertise, and examine which dimensions of independence are required for that work. In particular, we encourage participation from disciplines not yet engaged in AI evaluation; there are many open challenges that can benefit from their perspectives.

For current evaluators, there is an opportunity to examine their own methodological assumptions and embrace perspectives from other disciplines; this field of agent evaluation is inherently interdisciplinary, and will necessarily involve some rediscovery and reinvention of existing ideas and insights.

For field-builders—funders, coalitions, and conference organizers—agent evaluation demands its own field of practice, one with higher standards for validity and rigor. Raising this bar can happen by coalescing best practices and fostering conversation and engagement between different disciplinary perspectives.

For leaderboard aggregators, who are central to this emerging ecosystem by shaping which dimensions of evaluation receive attention and in what manner, there is a prime opportunity to incorporate best practices around reporting for downstream adoption, and to help the public understand the rising costs of evaluation.

For deployers and industry evaluators, there is a critical gap for real-world settings to inform evaluation work. By contributing on-the-ground insights and engaging with work in the lab, they can strengthen the discourse in this fast-moving space.

For labs, there is an opportunity to build bridges with independent evaluators and examine the community’s best practices. Successful embedded evaluators will require public legitimacy through independence, transparency, and methodological rigor.

For policymakers, there is a need to develop a clear sense of the current ecosystem and what good evaluations should look like. And for public sector deployments, evaluation needs to be an integrated and continuous part of the process from day one. AISIs can play a key role by pioneering new models for collaboration and inviting new stakeholders into their public input mechanisms within the current ecosystem.

Read the paper here.

One of the most fundamental questions for the field of AI today is: to what extent is the project of AI purely about automating human labor?

The dominant Silicon Valley program for AI right now, with both industry and academic faces, answers: “almost entirely.”1 There is an implicit technological determinism: that AI has a deep internal logic that favors automation. To keep pushing in this direction is to move towards economic growth, prosperity, and profit. The vision of AGI is the successful completion of this project: to create AI systems capable enough to autonomously do everything a human can do.

There is an alternative view: that AI is a technology (or class of technologies) that, like any technology, can develop in different directions. Total automation is one direction. But there are others, including directions that focus more on human augmentation, and increasing the collaborative and collective power of teams of humans and AI models working together.

This view — our view — is not that automation is never good. Clearly sometimes it is, and clearly sometimes some subset of people try to oppose it out of the narrowest short-term self interest, at the expense of broader social and economic progress. But we think AI can also augment human workers in many interesting ways, that some of the maximalist attempts to completely automate all human labor are deeply unpromising on both technical and economic grounds, and that right now too much of AI research is devoted to the latter at the expense of the former. The bias toward the automation agenda is partly because of economic incentives, but partly because of some deeper intellectual and cultural aspects of computer science as a field.

The alternative view has deep intellectual roots in the history of computer science, especially the subfield of human-computer interaction. Moreover, there are compelling technical and economic reasons to pursue it now. A small but growing number of businesses are now coming around to it, after witnessing the economic disappointments of attempting to implement the first program. Despite this, it remains a minority position today, poorly understood, and disorganized. Above all, computer science as a field today lacks a clear articulation of a systematic program for the architecture of AI that would further this agenda.

In this essay, we briefly discuss what is at stake here, the history of this debate in CS, the status of this debate today in economics and elsewhere, and the automation agenda in the culture and curricula of CS in the university. Above all, we sketch in Section 1 a few research directions for AI that push in the direction of human-AI collaboration. Designing AI for true collaboration requires solving a very different set of technical problems than today’s models and agents, problems that are often ignored, or not even understood to be problems. The specific directions we sketch are intended to be tentative and experimental. But the point is to explicitly conceive of a project for AI that differs from the dominant Silicon Valley one, and moves beyond criticism to the positive articulation of an alternative direction. Our hope is that this can serve as a seed for a much broader, sustained, experimental, and collective research program.

In Section 2, we discuss the high level motivation for our project, and in Section 3, we give a brief history of some of these ideas. In Section 4, we put this all into the context of the debates today about AI and automation in economics and in policy, and in Section 5, we outline a bottom-up, market-driven approach to steering the direction of AI development and deployment.

1. A Research Agenda for Collaborative Agents

A serious research program for human-AI collaboration elevates novel technical problems that the automation agenda for AI is more likely to ignore or treat as relatively unimportant. We present four such illustrative directions.

Problem #1: Can we produce white-box agents for various domains of human labor?

The dominant paradigm for AI models and agents is “black box”: the system is evaluated based on the final output it produces. The intermediate steps are generally not revealed to the user, may be mysterious even if revealed because they depart greatly from how people approach the task, and are not treated and optimized as an aspect of task performance. One reason for the black box paradigm is the absence of process traces in model training. If we train a new AI model on a thousand paintings from the Renaissance, but we have no data on any of the many intermediate steps and revisions involved in the creation of any given painting, we can’t expect anything other than a black box model. Maybe it can generate the image of a new painting in a similar style, but it will not be able to actively talk us through the steps involved in creating such a painting.

What we instead call “white box agents” are AI systems that are epistemically more legible to their human users, and explicitly reveal their intermediate processes (sources, assumptions, methods, uncertainties, process artifacts). White box agents could open the door to all kinds of more direct forms of human-AI collaborations: if humans can see the intermediate processes, they can directly interact with them.

The successful development of white box agents will require tackling many technical sub-problems. Chiefly among them: capturing process traces for a much broader range of human labor. The past few years of AI model training and deployment have had a disproportionate focus on software coding. There are many reasons for this, but a big one is that we benefit from the very public availability of a massive number of coding repositories, which include traces of every step of the process of software development. But for so many of the other important professional occupations in the economy today — law, medicine, many sciences — no such magical repositories of publicly available, free, comprehensive and easily codifiable process trace data exist.

Here’s a thought experiment. Imagine a counterfactual world in which open-source software didn’t exist, and coding agents (out of necessity) were trained to directly output compiled software binaries instead of source code, in a black box way. We think this capability would be experienced as dystopian instead of liberating by software engineers and it would have the effect of replacing them instead of amplifying their abilities. The shift we advocate is the equivalent of introducing code repositories and coding agents into this world — and doing so in every domain that currently lacks them.

There is a quiet gold rush to capture process traces for a broader range of occupations to feed into AI models, by both companies entirely dedicated to the purpose (Mercor) and legacy tech companies ruthlessly trying to study their own employees, the better to replace them (Meta). Paradoxically, this is all often done in the service of the automation paradigm: come up with training data to make more black box agents in more domains. But it could also be vital in better enabling the development of a new generation of white box agents.

Problem #2: Can we develop agents fast enough for real-time interaction?

A basic dynamic of active human collaboration is low latency. Imagine two people having a real conversation. If one person has to leave the room to think for 20 minutes every time you ask him a question, you’re no longer having a conversation; at best maybe you’re delegating to him. Models that can work at a speed that more naturally resembles human interaction might better enable forms of deeper human-AI collaboration.

Today, we are forced to pick between chatbots that can answer questions quickly but superficially, and agents that can perform tasks thoroughly but break the flow of conversation. We don’t have to be stuck with this tradeoff.

This sounds like a simple goal, but it implies a comprehensive set of technical challenges, that potentially span everything from model inference to retrieval methods, shells, browsers, tools, and more. How can we build AI systems that achieve latency low enough for collaboration? What parts of the agent stack could be changed, or need to be changed, to achieve this?

Low latency is particularly promising because it could enable workers to stay in a state of flow. Flow is a concept from psychology that describes people’s experience when performing activities such as rock climbing or chess that people are drawn to for intrinsic reasons. It is a state of altered experience in which the performer forgets time and fatigue, and is immersed in the moment. Most programmers are familiar with flow, and it is a big reason they are drawn to programming.

Working in a state of flow requires clear and immediate feedback — that is, low latency. So far, agents have weakened our ability to achieve flow. Many programmers report that the shift from manual programming to supervising coding agents has resulted in a loss of flow, cognitive overload and burnout. But our hypothesis is that it is possible to redesign agents to not only restore flow, but even increase people’s ability to achieve flow.

Problem #3: Can humans and agents continually work together, without the constraints of the turn-based paradigm?

The basic model of human-AI interaction, under the automation paradigm, is delegation: I tell my agent what to do, walk away, then come back later to verify. Many versions of “human in the loop” still come back to this; the human is technically in the loop, but just standing at the starting line and then waiting at the finish line. This is quite different from a model of collaboration, where the human is actively participating in the process, and continually steering the agent as it works.

A primitive version of continual steering exists in mainstream coding agents today, but is limited to giving additional instructions to the agent as it works. It is a slight tweak to the paradigm of the user and the agent alternately taking turns, rather than a departure from it. If the user tried to edit the same file as the agent, it would get confused and all bets are off.

True collaboration poses a novel set of technical challenges. Systems will need to display visible states that change continuously instead of turn-by-turn, have shared workspaces, and accommodate interruption, parallel work, simultaneous editing, and other collaboration patterns. This is especially important in domains where it is difficult to precisely specify the task ex ante, or verification is complicated and perhaps bound up in implementation. Meeting this challenge would require designing architectures of interaction that allow continuous control without melting down under the interruption.

Problem #4: How can we evaluate, and optimize for, human-AI collaboration?

Most AI benchmarks today test how good a model is at completing a task on its own, not on how well a model collaborates with a human. (This is likely part of the reason for the gap between benchmark performance and real-world usefulness).

Evaluating the joint efforts of AI-human teams is hard. This is a relatively simple challenge to articulate, but non-trivial to make progress on. It is more difficult to understand how to best evaluate AI-human collaboration, and it raises a complex set of questions that at first blush might seem uncomfortably subjective or confusing. How much disagreement between the AI model and the human is appropriate? How deferential should the model be, and how much control should the human have? How much space does a productive collaboration have for each party to learn from the other? How do we think about short term vs long term success of a team? What are the most objective ways to evaluate collaborative success, and what are the best signals and proxies to pick out of the data?

Improving collaboration skill might not be amenable to the benchmarking and hill-climbing paradigm that has led to rapid improvements in autonomous task performance. Benchmark tasks are a useful enough proxy for real-world tasks that performance on one is informative about performance on the other. Since collaboration involves a human, static benchmarks won’t work. Simulating the user is a promising approach, but beyond a point, even minor deviations between the behavior of simulated and real users might lead to rewarding the wrong thing in model training.

2. What is the problem with the automation agenda?

Debates about the potential excesses of automation tend to focus on purely distributive concerns. Critics worry that mass automation might create a broad class of workers who are unemployed, poor, or otherwise disempowered. Economists are conventionally skeptical that this would happen at all (modern economic history suggests a normal rhythm of technological progress, some automation, and then job reallocation). It is more common in Silicon Valley to accept the empirical premise of mass unemployment, but view it as a good thing that people won’t “have to work”, supported by Universal Basic Income payments or some even more exotic welfare scheme.

One of the many problems with this “automate and compensate” bargain is that it is wildly outside the recognizable political economy of the US today. Since the 1980s, income inequality has increased dramatically, and the American elite did not respond with generous new political programs of welfare expansion. Nor did the politics of the US otherwise produce a broad coalition of voters outside the elite who demanded and won such an expansion. (Many basic cash welfare programs were cut back over this same period). And even if some UBI-like program was politically feasible, it would be profoundly unattractive to normal people outside Silicon Valley. Most Americans want real lives with real work, not welfare checks.

Second, even if we were to accept the automation agenda, it is currently struggling on its own terms. Efforts for industrial AI deployment, under the automation agenda, increasingly promise to deliver highly capable agents that are simultaneously cheap to deploy (not requiring task-specific development or configuration), reliable, and genuinely autonomous. These goals are in direct tension with each other, to the point of imposing a trilemma – (1) completely autonomous, (2) reliable in high-stakes settings, (3) truly general purpose – pick two.

One of the most obvious questions today about the AI industry is whether the economics can make sense: billions of dollars of capex versus unclear ROI for many businesses. Relaxing the constraints of the automation agenda may actually open the door to a more economically sustainable version of AI.

Finally, AI has real potential for economic growth. After a 50-year period of mostly stagnant productivity growth in the US, it is important to make the most of this opportunity. The most radical and interesting uses of AI are likely to be those that let us do things we could not do before. Focusing entirely on simply automating the things we already know how to do is far less transformative and seems unlikely to lead to rapid economic growth. There is a widespread recognition in the industry and among policymakers that a positive vision for AI has yet to be articulated. We think that articulating such a vision will first require moving away from a narrow focus on automation.

3. Why the automation agenda dominates

The History of an Idea

In the early days of modern computing (1940s-1960s), there were three different visions for what computers should become, and what our relationship to computers should be. They overlapped in different points but were distinct. One vision, for what would later become the field of classical AI, aspired to almost anthropomorphic machines that could reason, solve problems, have agency, be general purpose, and speak human language. The term “artificial intelligence” was coined at a 1956 Dartmouth conference (even the name is trying to anthropomorphize!).

A second vision, what would become machine learning, emphasized narrow and deep performance on a very well specified set of tasks; using lots of data to make predictions. There was often an implicit understanding of the human role in choosing the tasks and employing the predictions.

A third vision (that would later become the subfield of HCI) very explicitly aspired to a collaboration of humans and machines. Works like JCR Licklider’s 1960 “Man-Computer Symbiosis” and Engelbart’s 1962 “Augmenting Human Intellect” argued that humans and computers have different abilities, that we should think about designing the best possible teams and organizations for combining them, not just trying to design the best computers in isolation, and that the best computers would be the ones that can work well iteratively and interactively with humans, not the ones that autonomously do as much work as possible.

AI today represents a convergence of classical AI and ML. And HCI remains a vibrant subfield. Inventions like the computer mouse, graphics, and many design features of personal computing that we take for granted came from this community of people who cared most about making machines that could collaborate well with people. But the basic vision of HCI forerunners like Licklider and Engelbart, of human-machine collaboration, seems like a minority position today. The vision of classical AI is the new orthodoxy. Why is it so dominant?

The Automation Agenda Today

The appeal of AI as pure automation has many causes. First, there is the ostensible economic case (even if it doesn’t ultimately hold up). The biggest cost of production, for most firms in most industries, is labor. For much of the 20^th century in the US, labor received two-thirds of income from production. For executives, it is usually the most obvious thing to try to reduce. Firing workers has the additional attraction of showing up immediately in short-run profit and loss. Besides, the exact cost savings are easy to estimate. In contrast, ambitious investments in augmenting productivity might pay off in 10 years, but are hard to estimate in advance and don’t give immediate financial gratification.

All of this is in the context of a policy environment that in many ways favors capital over labor (consider for instance the relative tax treatment) and many decades of declining real interest rates. If you are an unimaginative CEO of a Fortune 500 company, you are always intensely interested in alternatives to paying humans to work. If you are an AI start-up founder, pitching to a VC who wants products that this CEO will like, you are acutely conscious of this.

The economics are the obvious part of the story. But there are additional, less visible, more subtle layers, internal to CS.

There are a set of technical reasons that make automation more attractive to researchers. Autonomous models/agents lend themselves to very clean benchmarks. How well did the model do the task? The loss function, the performance measure, the comparisons between models, can all be elegant, much more so than some messy evaluation of how a human-AI team collaboratively performed in some real-world setting.

Finally – most obscure, but quite real – there is the deep background of the intellectual culture of AI, and CS more broadly, today. Autonomous systems are presented to students as a kind of aesthetic ideal. An “end to end” system is a triumph. Requiring human supervision is a defect, and even the way human involvement is generally discussed – e.g. “human in the loop” – often presents the human as a temporary inconvenience. Peter Thiel refers to this in “Zero to One” as “the ideology of computer science”:

“Why do so many people miss the power of complementarity? It starts in school. Software engineers tend to work on projects that replace human efforts because that’s what they’re trained to do. Academics make their reputations through specialized research; their primary goal is to publish papers, and publication means respecting the limits of a particular discipline. For computer scientists, that means reducing human capabilities into specialized tasks that computers can be trained to conquer one by one.” [Ch. 12, p. 148]

In the pedagogy of AI/ML, the standard approach is to teach a student a model pipeline (problem formulation, data, model selection, training, evaluation, post-training, deployment, monitoring) in which the central object under construction is quasi-autonomous and questions about human-machine collaboration are often set to the side. Humans may appear episodically (as data labelers, or helping provide reinforcement feedback), but are not presented as central agents themselves. The agent is the model.

Ten steps before the economics of the Fortune 500 CEO shows up on the scene, the deep culture of CS as a field already shapes our whole approach to building AI.

4. The Current Debate Over AI and Automation in Economics and Policy

The empirics of the labor market impacts of AI is a rapidly evolving topic. But at a deeper level, a common view amongst economists is that technology simply is what it is – we don’t control its direction or development and attempts to do so are some mix of naïve, impossible, embarrassing, and doomed to failure. If AI is a little automation machine that replaces human task performance with computer task performance, then that’s just what it is. Technological progress is a kind of linear movement, in a single direction, and the first job of economic policy is just to not get in the way of this. The idea that technology can genuinely develop in different directions, as a general matter or specifically in the context of AI, remains rather alien.

There are notable exceptions. Acemoglu and Restrepo have a series of papers arguing that there is “excess automation”. Acemoglu and Johnson conclude their book “Power and Progress” with an argument for pro-augmentation AI. Brynjolfsson calls the danger of AI excessively focused on pure human imitation the “Turing Trap”. Autor writes about how the real promise of AI may actually be the democratization of tacit knowledge to a broader set of workers, not their automation. Mullainathan and coauthors are launching a series of research projects around the argument that we should promote the human-AI collaboration view, but “….unfortunately, a different vision animates AI today: replacements for the mind. To us, that is a pretty uninspiring vision of the future. Automation cuts costs but does little to raise the ceiling. Merely replicating human capability also means we are limited by the limits of human capacity... we are committed to building bicycles for the mind — algorithms and AI that enhance human capacity, not replace it.”

Clearly throughout economics, as in CS, there are fellow travelers. But it remains a minority position. The response of many economists to the Acemoglu and Johnson book, particularly on this point, has often been dismissive and condescending.

Part of the disagreement could be explained by confusion about how to go about actually steering AI in a pro-worker direction. The approach skeptics seem to have in mind is a clumsy, Soviet-style top-down set of commands about what the economy should look like, even down to banning many forms of automation outright, like self-driving cars or warehouse robots, in order to save jobs.

This is not what we are advocating for. Automation is not a universally bad thing. Many jobs are extremely routine and naturally amenable to automation. There is no glory in a rearguard action to save them; automating them is important social and economic progress. The men in the meat processing factories in Chicago in the 1900s worked 12 hours a day, 6 days a week, for the privilege of low pay, amputated limbs, and death from falling machines. The ones in the coal mines of Virginia perhaps had it worse. Automation has been great in both industries. 40,000 people die in car crashes in the US every year because humans are not very good drivers; we will presumably look back on this stat one day, from our world of self-driving cars, with similar horror.

Autonomous AI agents in knowledge work also have their time and place. There are many tasks where the more distinctively human attributes (tacit knowledge, judgment, sense of context and shifting goals) are relatively unimportant, the activities themselves are routine, output is easy to verify, a little bit of error is relatively costless, or the risk or suffering posed to human workers is high. Some combination of the above might justify complete automation in a given domain.

But the point of a pro-augmentation program for AI is that (1) there is a second much broader set of activities that do not lend themselves well to automation under AI as we have it today, and (2) it would be regrettable if, under a set of technical, economic, and intellectual confusions, the field of AI devoted itself to a misguided and unsuccessful effort to accomplish (1), when it could be more successfully embarking on a program to advance human-AI collaboration, and increase our actual productive powers in the world.

We think policy can, on the margin, nudge us toward a bottom-up world of people experimenting with forms of AI that better empower human beings. We provide some concrete ideas at the end of the next section.

5. How Can We Steer AI toward Augmentation? A Bottom Up, Market-Driven Approach

Suppose one believes some version of the thesis that the AI industry today is excessively biased towards a narrow vision of AI as automation, in a way that might be broadly suboptimal, not only for many workers, but also for many firms and economic growth more broadly. What should we do about it? The strawman answer is top-down policy. The more serious answer is bottom-up and multi-faceted. We imagine a regime of decentralized technical experimentation, some intellectual and cultural shifts within CS, some carefully targeted policy levers, and firms gradually discovering that the automation agenda may struggle on its own terms.

New entrants and products are starting to cater to the unmet demand for collaboration agents

There aren’t yet true collaboration agents on the market. Thinking Machines, a lab founded by OpenAI’s former CTO Mira Murati, has published an architecture for such agents. In their vision, there is a user-facing model that is ultra-low-latency and multimodal, farming off more complex work to a more capable background model. However, they have not yet shipped a working product. Still, it is reassuring that there is a well-capitalized company that has made a big bet on human-AI collaboration.

Simultaneously, there has been a big push for low latency across all three frontier labs (OpenAI, Anthropic, and Google), especially OpenAI due to its partnership with Cerebras, as well as Cursor and Cognition. We predict that as more workers experience this way of working with AI, there will be more demand for it, creating a flywheel effect.

Another set of companies that might build collaboration agents (though they haven’t yet) are those that specialize in a particular domain or vertical, such as Harvey AI in law or OpenEvidence in medicine. Effective systems of collaboration often need domain specific traces, tacit expert knowledge, domain expertise. The big frontier AI labs may be poorly equipped to do this themselves, compared to industry-specific firms that can figure out how to combine AI capabilities with deep domain specific knowledge. Many of the industries in which this model of organization will work best, such as law and medicine, are heavily regulated. Regulated industries are a natural target for collaborative human-AI models because humans are legally required at many decision points anyway.

Beyond collaboration: many paths to complementarity

Collaboration is only one example of a much broader design space of configurations for human-AI complementarity. Here are a few others, intended to be illustrative rather than exhaustive. For each of these alternative configurations, one can imagine repeating the exercise of Section 1, of breaking it down into discrete technical challenges.

• Collective Intelligence: AI can transform and mediate how teams of human workers collaborate with each other. This can take a number of different forms: preserving organizational memory, mediating conflict, identifying expertise within a team, synthesizing distributed knowledge, and otherwise helping coordination. So the proximate goal here is not the execution of a particular task; it is raising the productivity of a team of humans in general.

• New Tasks and Capabilities: The use of AI to automate extant human labor is quite different from the use of AI to accomplish new tasks that no humans could do in the first place. There are any number of technological innovations from history that were clearly of this flavor, such as the airplane or the microscope.

• Upskilling: Stepping back from the short term work of task completion, AI can also help upskill human workers in their long term capabilities. AI is particularly effective at internalizing tacit expert knowledge; with intentional and thoughtful design, it then may become effective at making workers more expert in their fields in the long term.

• Ambient Agent: The normal paradigm of AI models/agents, the model is what we might call an invoked model: it shows up when you explicitly call on it and ask it to do things. We might define, by contrast, an “ambient agent”: an AI model that operates across an organization, observing context and offering help even without being explicitly invoked. While this raises a distinct set of challenges (privacy, discretion), it is also a different paradigm of AI-human collaboration with distinct opportunities (more context, more organization knowledge, interesting novel forms of coordination).

The Role of HCI and cultural change

HCI was historically the vanguard, within CS, of taking human-machine collaboration seriously. Today it occupies a more marginalized position within CS. To the extent that it can play an outsized and important role in helping reorient CS away from the automation agenda towards something more productive, two things would need to happen simultaneously.

First, computer science departments, funding organizations, and other agenda setters could try to make a conscious effort to elevate the status of HCI. Second, HCI could evolve to meet the moment. It has historically constrained itself intellectually and methodologically, and this partly explains its marginalization. To simplify a bit, HCI treats human control as axiomatic rather than an instrumental goal, and its unit of analysis is the user, leaving it with little to say on broader questions about firms, organizations, incentives, power, and political economy.

The new movement of Human-Centered AI is partly an attempt at revival and synthesizing HCI with AI, though it is dwarfed in volume and influence by the more mainstream AI field that generally tends to accept the industry’s priorities. In one possible future, however, a better, transformed version of HCI (and HCAI) ascends in status within the university and has meaningful agenda-setting power.

Similarly, AI pedagogy has a history of centering the AI system and forgetting the human. It doesn’t have to be this way. One can imagine changes to syllabi, benchmark design, workshops, and funding priorities.

Targeted Policies

Beyond crude automation bans or top-down technological mandates, more intelligent and tailored government policies can also be one part of a broader success story in reorienting the automation agenda.

On the supply side, these could include research funding priorities and even DARPA-style prizes. On the demand side, US tax policy currently favors automation, and the asymmetry could be mitigated, though this requires care to avoid crude attempts at regulatory classification of AI systems as automation or augmentation. The government could also use its power as a major purchaser of AI. Finally, regulation that requires human oversight of AI must insist on effective oversight instead of rubber stamping.

Competition policy could also help. The automation paradigm has a big head start, and policymakers should ensure clients don’t get locked in. Giving workers more of a voice in bargaining over AI deployment is another way to steer toward augmentation without mandating it.

Conclusion

The automation agenda for AI is currently winning the internal battle over the direction of AI. The stakes are very high. One of the biggest peace-time deployments of private capital in history is now being largely devoted to an automation vision of AI that may be both economically untenable on its own terms, and also a waste of all the potential of AI to help us do things we couldn’t already do.

Fortunately, there is a broad set of people — economists, technologists, and others — who are trying to understand and develop another direction for AI, one that places a greater emphasis on human-computer collaboration, in the service of being able to do new things, not just automate the old ones.

A central task for this second direction is to define and pursue a systematic research program for AI in the service of this kind of collaboration. Our hope is that through greater collective awareness and intention, and better pooling and coordination of the many efforts that already exist, a bottom-up, decentralized approach can be successful at steering the direction of AI development and deployment, without requiring top-down commands or directives.

Suggested Further Reading:

• For a sample of economists’ writing on this topic, see The Turing Trap and Building Pro-Worker AI.

• For a brief history of these ideas in AI and HCI, see Section V.B of How AI Fails Us.

• The paper Human Amplification, Intelligent Agents, and the Aims of AI Research makes arguments similar to ours from a philosophical perspective. Our scope is narrower and centers on AI tools for workers.

• The paper AI Agents Push Humans Out of the Loop critiques the design of current AI agents.

• There is a lot that individuals can do to amplify their own agency without waiting for AI systems to be redesigned: Arvind presented some ideas in his ICML talk.

Few have considered the third possibility that x-risk warnings are sincere but simply wrong and counterproductive to AI safety. It’s as if everyone takes for granted that those raising the alarm are geniuses, and the only question is whether they’re the benevolent or the evil kind of genius.

We subscribe to neither narrative. In our writing on AI safety, we’ve consistently tried to take the arguments seriously but rebut the many claims we think are unsound while highlighting safety concerns that we think are real. This essay is a continuation of that agenda.

It is true that we systematically underinvest in resilience against catastrophic and systemic risks

Let’s start by steelmanning a subset of AI safety and Effective Altruist (EA) arguments. Something is deeply wrong in the world when it comes to how much we invest in defending against catastrophic risks, and the safety and EA communities are right about that. Take superintelligence out of the equation entirely and it remains true that evidence-grounded risk arguments have been made for years and not produced a commensurate policy response.

The most well known example is the failure to prepare for pandemics. We had years of warnings and quantitative estimates. A 2019 report convened by the WHO/World Bank — before COVID — began by saying there is a “very real threat ... of a respiratory pathogen killing 50 to 80 million people and wiping out nearly 5% of the world’s economy.” It estimated the cost of adequate preparedness at a mere $1–2 per person per year for most countries.

Since the 2009 H1N1 pandemic, at least 11 high-level panels and commissions have made specific recommendations to improve global pandemic preparedness, and yet “the majority of recommendations were never implemented”. More remarkably, even after COVID, only a fraction of the need was funded. The same independent panel wrote in 2024 that “There is a cynical attraction to providing significant and immediate resources during a crisis, rather than proactively preventing one. Yet history shows this will not end well.” The root cause seems to be that voters reward spending on relief, not preparedness.

To be clear, the inadequacy of pandemic preparedness is by no means a uniquely Effective Altruist insight, let alone AI safety. But EA does deserve credit for championing the cause. It’s also true that EA’s consequentialist moral framework that centers expected-utility calculations is highly effective at revealing and quantifying our underpreparedness. The framework is highly controversial, but we don’t think there’s anything wrong with it or its application to areas like biorisk. We do, however, strongly disagree with its extensions to longtermism and AI existential risk. We also disagree with the tendency to reframe problems such as AI’s amplification of biorisk as “AI safety” problems to be solved through methods such as alignment. Our view is that AI is an amplifier of existing systemic risks.

The underpreparedness problem is also true of cybersecurity — actually, it’s worse. Unlike pandemics or finance, there is no culture of modeling systemic or catastrophic risks in the cybersecurity community. The economic approach we described in the previous essay has been successful in keeping the expected losses from everyday attacks at a manageable level, but markets are notoriously bad at appropriately pricing tail risks and externalities.

Insurance companies know this. Lloyd’s is pretty blunt about the fact that “losses have the potential to greatly exceed what the insurance market is able to absorb”, and does not insure against severe state-backed cyberattacks (that was in 2022).

Once again, this has little to do with AI’s recent capabilities. Five years ago, before AI was in the picture, when teaching undergraduate information security at Princeton, Arvind emphasized that cascading failures and catastrophic risks are possible, resulting in essentially the whole internet going down for an extended period, with potentially civilization-altering consequences.1

(Looking back at the speaker notes in my slides, I’m reminded that I apologized for presenting original research in an undergrad class that’s supposed to be about basics, but explained that this is because the near-total lack of attention to this critical topic in the security community left me no choice. — AN.)

In other words, Dario Amodei’s claim about agent swarms potentially being capable of “taking over the entire internet”, while worded somewhat hyperbolically, is not obviously outside the realm of possibility, and deserves study at the very least. The cybersecurity community has the right skillset to do that, but unfortunately they don’t seem to find this even worth thinking about.

There are also systemic risks that are diffuse and gradual rather than catastrophic. Consider what chatbots are doing to news traffic — clickthrough rates are far lower than traditional search, accelerating the decline of traditional journalism. The second-order societal effects of this shift won’t be apparent for quite a while. And that’s just one of dozens of types of creeping institutional decay. Atoosa Kasirzadeh introduced the concept of accumulative risk to describe this gradual erosion of societal resilience.2 Here, the psychology of inaction is the reverse of the case of pandemics: rather than being so rare that we are able to memory-hole the event, decay is so pervasive and gradual that we become inured to it.

In short, based entirely on well-grounded rather than speculative causal pathways, it is reasonable to believe that policymakers are drastically underinvesting in protecting society against AI risks, and it is hard not to empathize with those who want to ratchet up the rhetoric.

Of course, the question is whether more urgent public warnings actually lead to better policymaking.

The Coxon incident shows how the warped media environment elevates x-risk over more grounded concerns

This newsletter is largely an exercise in perspective-taking. Let’s take a moment to articulate the primary reason why people inside and outside the AI bubble tend to talk past each other.

For people in AI — whether or not they’re concerned about safety — it’s hard to understand how anything else in the world matters right now. As computer scientist Scott Aaronson put it in his essay The Age of Wonders and Terrors:

“Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.”

This essay made the rounds in the community because this perspective is widely shared. Even the two of us — best known for pushing back against some elements of this view — consider AI important enough that we decided long ago to devote our careers to it. The AI as Normal Technology position is that AI will “only” be as impactful as the industrial revolution.

People outside AI don’t understand the intensity of this view in the AI community — that it’s an AI world and we’re all just living in it.

But there’s a flip side — people in AI vastly overestimate how much the public cares about AI. In Gallup polls, the share of people who rate AI (or “advancement of computers/technology”) as the most important issue is about half a percent. Most of those people are likely thinking of relatively mundane aspects such as a datacenter being built in their community. The share of people outside AI who treat it as the defining issue of our time is incomprehensibly small.

The reason AI people fail to recognize this is that AI is a topic where salience and concern are decoupled from each other. There is high concern about AI (that is, many people think AI is important or express concern when specifically asked about it) but very low salience (that is, few people rank it as an important issue versus others).

Anyway, all this is mere throat-clearing to explain why, even though there are many catastrophic and systemic AI-amplified risks that we agree the world urgently needs to act on, they don’t tend to break through to public consciousness.

After the agent swarm hacks over the summer came to light, there was a smattering of headlines, but it was orders of magnitude below the level of media attention and public awareness that Coxon’s resignation and x-risk predictions would later draw.

The safety community expressed amazement and frustration that these concerns hadn’t broken through to the public. This put the whole community on edge. As observers of the community, we felt the level of anxiety in the community to be qualitatively different from any other moment in the past.

It is not hard to explain why AI cyberrisk didn’t get more headlines: AI is a low-salience topic, hacks are in the news every day, and this one didn’t even result in any known harm to ordinary people. It is far from obvious why everyday people should care.

In contrast, existential risk is visceral, and the Coxon resignation made AI safety finally break through. To be clear, x-risk warnings have been made many times before, but there were two differences this time. First, there was a tinderbox waiting to be lit, namely the safety community’s eagerness to talk to the media and amplify concerns once there was a hook in the form of a newsworthy event. This response was so full-throated and unanimous that it left people concluding that it must have been a co-ordinated effort, but we don’t think that was the case.

There’s another reason this time was different: the revelation that many executives have a high “p(doom)” and yet continue down the path of racing to build superintelligence, a choice that feels reckless and incomprehensible to the public. Of course, there is a complex set of rationalizations that AI leaders invoke to explain why they think they have no other choice, but that’s not what reaches people. Perhaps the bigger disconnect is that it is all underpinned by a utilitarian value system that the public doesn’t share.

Why the x-risk framing may be counterproductive for AI safety policy

To recap the argument so far, there are two clusters of beliefs about what AI safety means. Like any binary, this one leaves out some nuances, but we think it is accurate enough to be useful.

1. There are specific (potentially catastrophic but sub-existential) risks, predating AI but amplified by it, that we urgently need to defend against through prevention and resilience.

2. AI development itself is an existential risk — either because there are too many downstream risks to defend against once superintelligent AI is built, or because of unknown unknowns — and we either need to figure out how to make superintelligence safe (such as through alignment) or stop it from being built.

Critically, due to the information environment, the x-risk framing is the one that is driving public concern and much policy discourse, despite the fact that credible causal pathways haven’t been articulated (and attempts to do so are unconvincing to say the least).

So what’s going to happen next? One happy possibility is that some of the energy of the current moment gets channeled into building a broad set of defenses (perhaps accompanied by a slowdown in AI development, though that is much less important in our view).

But this path seems unlikely to us. The more likely outcome is that the dominance of the x-risk framing is actively counterproductive, for two main reasons.3

Polarization and partisanship. The doom framing has always been polarizing because many of us remain completely unconvinced by it, but now the polarization is starting to split along partisan lines. Partisanship is bad for safety policy because it leads to inaction, high uncertainty, and partisan policy reversals. Worse, when people realize that there is no quantitative model behind the “10% extinction risk” predictions, they will feel misled, resulting in even more pushback to safety policies, especially in a polarized environment where one side is already wary of experts making stuff up.

Further, causing the public to panic is counterproductive because it constrains the solution space toward those that are the most blunt and offer an emotional salve, which don’t tend to be the most effective.

Misdirection. We have repudiated the distraction argument against x-risk concerns — if we’re worried about job loss and safety, we can address both, and they don’t distract from each other. But this isn’t about distraction: the two views of safety represent two different diagnoses of the same problem. It seems likely that they will compete for resources. Let’s call it misdirection, to distinguish it from distraction.

In particular, a generalized existential threat admits exactly one policy — a ban. Framing the threat in terms of probabilities biases us even more in that direction. The statement “if we develop superintelligence there’s a 10% risk of extinction” makes it look like the only lever for intervention is stopping development. Unfortunately, a ban on superintelligence is definitionally ambiguous to the point of being incoherent.4 And AI safety is still not a model property. Other potential interventions such as pausing datacenter construction are exceedingly impotent as safety measures.

Most importantly, bans on development do nothing about the catastrophic and systemic risks from existing AI systems. The response we need to those risks is much less sexy, which in turn means that the movement one would build to address them is quite different from one targeted at superintelligence: a hundred disparate interventions to shore up defenses and resilience rather than concentrating effort and attention toward erecting one mammoth, flashy wall. This is why we worry that the two framings of AI safety are in competition. We hope we are wrong about this!

Conclusion: a big tent or small tent AI safety movement?

There are three main characteristics of what we call big-tent safety.

• Beliefs. A big-tent movement would include members who are not concerned about x-risk (and, like us, may not consider superintelligence a threat). They may care about cyberrisk, biorisk and other specific catastrophic risks, various accumulative risks, and the need to build societal resilience.

• Values and movement building. The social movement must be pluralist in terms of values and not just beliefs, recognizing that EA / safetyist moral frameworks are unpopular with the public. The public appeal should be based not on panic but articulating positive visions.

• Policy. A big-tent movement would recognize that there is much common ground on policy despite divergence in beliefs (though these policies arguably fall short if you’re narrowly focused on x-risk). Two such clusters of policies have become clear: transparency and liability. Embedded evaluation is urgent, and organizations that build and use AI should be systematically accountable for the negative externalities they create. Big-tent safety policy focuses on picking such low-hanging fruit while building muscle for stronger interventions like bans (such as banning fully autonomous recursive self-improvement). But the latter should involve doing the hard work of convincing skeptics of their necessity rather than pursuing policy-by-panic.

In contrast, small-tent safety is narrower at the level of beliefs, values, and policy goals. It prizes ideological homogeneity (while claiming the contrary). This risks a purity spiral, gradually driving out big-tent safety people. Small-tent safety de-prioritizes spending on defenses against risks that are “only” catastrophic (!). Finally, such a movement doesn’t recognize the gap between itself and the public in terms of both values and the salience of AI safety. As a result, it vastly overplays its hand, risking backlash.

On a personal note, we’ve often wondered why our work has been sometimes welcomed and sometimes attacked by the AI safety community, and we think the big-tent versus small-tent distinction captures it well.

Let’s build a bigger tent.

We are grateful to Abi Olvera, Joshua Saxe, and Rohit Krishnan for feedback on a draft.

We should be clear about what we’re not saying.

• We aren’t saying forecasters (or others who attempt to quantify AI x-risk) intend to mislead. Unfortunately, we think the effect is misleading nonetheless.

• In general, being quantitative about forecasts is good. In fact, Arvind has been serving as an advisor to the Forecasting Research Institute’s Longitudinal Expert AI Panel, an effort he thinks is extremely valuable. As the essay makes clear, the infeasibility of building useful quantitative models is specific to forecasts of AI x-risk, not forecasting in general.

• Our goal here is not to try to push AI safety advocates to stop communicating in terms of probabilities (not that we’d succeed even if we tried). Rather, our hope in publishing this is for those at the receiving end of these probabilities — especially policymakers — to recognize that unlike almost every other quantitative forecast that we encounter, they are not the result of a validated model or method.

That said, we think the whole p(doom) culture is emblematic of a particular view of AI safety that centers existential risk and superintelligence in a way that is actively counterproductive to a broader conception of safety. In a few days we’ll publish a follow-up essay making that argument and advocating for a “big tent” safety movement.

The original essay appears below.

AI existential risk probabilities are too unreliable to inform policy

How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions.

The AI safety community relies heavily on forecasting the probability of human extinction due to AI (in a given timeframe) in order to inform decision making and policy. An estimate of 10% over a few decades, for example, would obviously be high enough for the issue to be a top priority for society.

Our central claim is that AI x-risk forecasts are far too unreliable to be useful for policy, and in fact highly misleading.

Look behind the curtain

If the two of us predicted an 80% probability of aliens landing on earth in the next ten years, would you take this possibility seriously? Of course not. You would ask to see our evidence. As obvious as this may seem, it seems to have been forgotten in the AI x-risk debate that probabilities carry no authority by themselves. Probabilities are usually derived from some grounded method, so we have a strong cognitive bias to view quantified risk estimates as more valid than qualitative ones. But it is possible for probabilities to be nothing more than guesses. Keep this in mind throughout this essay (and more broadly in the AI x-risk debate).

If we predicted odds for the Kentucky Derby, we don’t have to give you a reason — you can take it or leave it. But if a policymaker takes actions based on probabilities put forth by a forecaster, they had better be able to explain those probabilities to the public (and that explanation must in turn come from the forecaster). Justification is essential to legitimacy of government and the exercise of power. A core principle of liberal democracy is that the state should not limit people’s freedom based on controversial beliefs that reasonable people can reject.

Explanation is especially important when the policies being considered are costly, and even more so when those costs are unevenly distributed among stakeholders. A good example is restricting open releases of AI models. Can governments convince people and companies who stand to benefit from open models that they should make this sacrifice because of a speculative future risk?

The main aim of this essay is analyzing whether there is any justification for any of the specific x-risk probability estimates that have been cited in the policy debate. We have no objection to AI x-risk forecasting as an academic activity, and forecasts may be helpful to companies and other private decision makers. We only question its use in the context of public policy.

There are basically only three known ways by which a forecaster can try to convince a skeptic: inductive, deductive, and subjective probability estimation. We consider each of these in the following sections. All three require both parties to agree on some basic assumptions about the world (which cannot themselves be proven). The three approaches differ in terms of the empirical and logical ways in which the probability estimate follows from that set of assumptions.

Inductive probability estimation is unreliable due to the lack of a reference class

Most risk estimates are inductive: they are based on past observations. For example, insurers base their predictions of an individual’s car accident risk on data from past accidents about similar drivers. The set of observations used for probability estimation is called a reference class. A suitable reference class for car insurance might be the set of drivers who live in the same city. If the analyst has more information about the individual, such as their age or the type of car they drive, the reference class can be further refined.

For existential risk from AI, there is no reference class, as it is an event like no other. To be clear, this is a matter of degree, not kind. There is never a clear “correct” reference class to use, and the choice of a reference class in practice comes down to the analyst’s intuition.

The accuracy of the forecasts depends on the degree of similarity between the process that generates the event being forecast and the process that generated the events in the reference class, which can be seen as a spectrum. For predicting the outcome of a physical system such as a coin toss, past experience is a highly reliable guide. Next, for car accidents, risk estimates might vary by, say, 20% based on the past dataset used — good enough for insurance companies.

Further along the spectrum are geopolitical events, where the choice of reference class gets even fuzzier. Forecasting expert Philip Tetlock explains: “Grexit may have looked sui generis, because no country had exited the Eurozone as of 2015, but it could also be viewed as just another instance of a broad comparison class, such as negotiation failures, or of a narrower class, such as a nation-states withdrawing from international agreements or, narrower still, of forced currency conversions.” He goes on to defend the idea that even seeming Black Swan events like the collapse of the USSR or the Arab Spring can be modeled as members of reference classes, and that inductive reasoning is useful even for this kind of event.

In Tetlock’s spectrum, these events represent the “peak” of uniqueness. When it comes to geopolitical events, that might be true. But even those events are far less unique than extinction from AI. Just look at the attempts to find reference classes for AI x-risk: animal extinction (as a reference class for human extinction), past global transformations such as the industrial revolution (as a reference class for socioeconomic transformation from AI), or accidents causing mass deaths (as a reference class for accidents causing global catastrophe). Let’s get real. None of those tell us anything about the possibility of developing superintelligent AI or losing control over such AI, which are the central sources of uncertainty for AI x-risk forecasting.

To summarize, human extinction due to AI is an outcome so far removed from anything that has happened in the past that we cannot use inductive methods to “predict” the odds. Of course, we can get qualitative insights from past technical breakthroughs as well as past catastrophic events, but AI risk is sufficiently different that quantitative estimates lack the kind of justification needed for legitimacy in policymaking.

Deductive probability estimation is unreliable due to the lack of theory

In Conan Doyle’s The Adventure of the Six Napoleons — spoiler alert! — Sherlock Holmes announces before embarking on a stakeout that the probability of catching the suspect is exactly two-thirds. This seems bewildering — how can anything related to human behavior be ascribed a mathematically precise probability?

It turns out that Holmes has deduced the underlying series of events that gave rise to the suspect’s seemingly erratic observed behavior: the suspect is methodically searching for a jewel that is known to be hidden inside one of six busts of Napoleon owned by different people in and around London. The details aren’t too important, but the key is that neither the suspect nor the detectives know which of the six busts it is in, and everything else about the suspect’s behavior is (assumed to be) entirely predictable. Hence the precisely quantifiable uncertainty.

The point is that if we have a model of the world that we can rely upon, we can estimate risk through logical deduction, even without relying on past observations. Of course, outside of fictional scenarios, the world isn’t so neat, especially when we want to project far into the future.

When it comes to x-risk, there is an interesting exception to the general rule that we don’t have deductive models — asteroid impact. A combination of inductive and deductive risk estimation does allow us to estimate the probability of x-risk, only because we’re talking about a purely physical system. Let’s take a minute to review how this works, because it’s important to recognize that the methods are not generalizable to other types of x-risk.

The key is being able to model the relationship between the size of the asteroid (more precisely, the energy of impact) and the frequency of impact. Since we have observed thousands of small impacts, we can extrapolate to infer the frequency of large impacts that have never been directly observed. We can also estimate the threshold that would cause global catastrophe.1

With AI, the unknowns relate to technological progress and governance rather than a physical system, so it isn’t clear how to model it mathematically. Still, people have tried. For example, in order to predict the computational requirements of a hypothetical AGI, several works assume that an AI system would require roughly as many computations as the human brain, and further make assumptions about the number of computations required by the human brain. These assumptions are far more tenuous than those involved in asteroid modeling, and none of this even addresses the loss-of-control question.

Subjective probabilities are feelings dressed up as numbers

Without the reference classes or grounded theories, forecasts are necessarily “subjective probabilities”, that is, guesses based on the forecaster’s judgment. Unsurprisingly, these vary by orders of magnitude.

Subjective probability estimation does not get around the need for having either an inductive or a deductive basis for probability estimates. It merely avoids the need for the forecaster to explain their estimate. Explanation can be hard due to humans’ limited ability to explain our intuitive reasoning, whether inductive, deductive, or a combination thereof. Essentially, it allows the forecaster to say: “even though I haven’t shown my methods, you can trust this estimate because of my track record” (we explain in the next section why even this breaks down for AI x-risk forecasting). But ultimately, lacking either an inductive or a deductive basis, all that forecasters can do is to make up a number, and those made-up numbers are all over the place.

Consider the Existential Risk Persuasion Tournament (XPT) conducted by the Forecasting Research Institute in late 2022, which we think is the most elaborate and well-executed x-risk forecasting exercise conducted to date. It involved various groups of forecasters, including AI experts and forecasting experts (“superforecasters” in the figure). For AI experts, the high end (75th percentile) of estimates for AI extinction risk by 2100 is 12%, the median estimate is 3%, and the low end (25th percentile) is 0.25%. For forecasting experts, even the high end (75th percentile) is only 1%, the median is a mere 0.38%, and the low end (25th percentile) is visually indistinguishable from zero on the graph. In other words, the 75th percentile AI expert forecast and the 25th percentile superforecaster forecast differ by at least a factor of 100.

All of these estimates are from people who have deep expertise on the topic and participated in a months-long tournament where they tried to persuade each other! If this range of forecasts here isn’t extreme enough, keep in mind that this whole exercise was conducted by one group at one point in time. We might get different numbers if the tournament were repeated today, if the questions were framed differently, etc.

What’s most telling is to look at the rationales that forecasters provided, which are extensively detailed in the report. They aren’t using quantitative models, especially when thinking about the likelihood of bad outcomes conditional on developing powerful AI. For the most part, forecasters are engaging in the same kind of speculation that everyday people do when they discuss superintelligent AI. Maybe AI will take over critical systems through superhuman persuasion of system operators. Maybe AI will seek to lower global temperatures because it helps computers run faster, and accidentally wipe out humanity. Or maybe AI will seek resources in space rather than Earth, so we don’t need to be as worried. There’s nothing wrong with such speculation. But we should be clear that when it comes to AI x-risk, forecasters aren’t drawing on any special knowledge, evidence, or models that make their hunches more credible than yours or ours or anyone else’s.

The term superforecasting comes from Philip Tetlock’s 20 year study of forecasting (he was also one of the organizers of the XPT). Superforecasters tend to be trained in methods to improve forecasts such as by integrating diverse information and by minimizing psychological biases. These methods have been shown to be effective in domains such as geopolitics. But no amount of training will lead to good forecasts if there isn’t much useful evidence to draw from.

Even if forecasters had credible quantitative models (they don’t), they must account for “unknown unknowns”, that is, the possibility that the model itself might be wrong. As noted x-risk philosopher Nick Bostrom explains: “The uncertainty and error-proneness of our first-order assessments of risk is itself something we must factor into our all-things-considered probability assignments. This factor often dominates in low-probability, high-consequence risks — especially those involving poorly understood natural phenomena, complex social dynamics, or new technology, or that are difficult to assess for other reasons.”

This is a reasonable perspective, and AI x-risk forecasters do worry a lot about uncertainty in risk assessment. But one consequence of this is that for those who follow this principle, forecasts are guaranteed to be guesses rather than the output of a model — after all, no model can be used to estimate the probability that the model itself is wrong, or what the risk would be if the model were wrong.

Forecast skill cannot be measured when it comes to unique or rare events

To recap, subjective AI-risk forecasts vary by orders of magnitude. But if we can measure forecasters’ track records, maybe we can use that to figure out which forecasters to trust. In contrast to the previous two approaches for justifying risk estimates (inductive and deductive), the forecaster doesn’t have to explain their estimate, but instead justifies it based on their demonstrated skill at predicting other outcomes in the past.

This has proved to be invaluable in the domain of geopolitical events, and the forecasting community spends a lot of effort on skill measurement. Many ways to evaluate forecasting skill exist, such as calibration, the Brier score, the logarithmic score, or the Peer score used on the forecasting competition website Metaculus.

But regardless of which method is used, when it comes to existential risk, there are many barriers to assessing forecast skill for subjective probabilities: the lack of a reference class, the low base rate, and the long time horizon. Let’s look at each of these in turn.

Just as the reference class problem plagues the forecaster, it also affects the evaluator. Let’s return to the alien landing example. Consider a forecaster who has proved highly accurate at calling elections. Suppose this forecaster announces, without any evidence, that aliens will land on Earth within a year. Despite the forecaster’s demonstrated skill, this would not cause us to update our beliefs about an alien landing, because it is too dissimilar to election forecasting and we do not expect the forecaster’s skill to generalize. Similarly, AI x-risk is so dissimilar to any past events that have been forecast that there is no evidence of any forecaster’s skill at estimating AI x-risk.

Even if we somehow do away with the reference class problem, other problems remain — notably, the fact that extinction risks are “tail risks”, or risks that result from rare events. Suppose forecaster A says the probability of AI x-risk is 1%, and forecaster B says it is 1 in a million. Which forecast should we have more confidence in? We could look at their track records. Say we find that forecaster A (who has assigned a 1% probability to AI x-risk) has a better track record. It still doesn’t mean we should have more confidence in A’s forecast, because skill evaluations are insensitive to overestimation of tail risks. In other words, it could be that A scores higher overall because A is slightly better calibrated than B when it comes to everyday events that have a substantial probability of occurring, but tends to massively overestimate tail risks that occur rarely (for example, those with a probability of 1 in a million) by orders of magnitude. No scoring rule adequately penalizes this type of miscalibration.

Here’s a thought experiment to show why this is true. Suppose two forecasters F and G forecast two different sets of events, and the “true” probabilities of events in both sets are uniformly distributed between 0 and 1. We assume, highly optimistically, that both F and G know the true probability P[e] for every event e that they forecast. F always outputs P[e], but G is slightly conservative, never predicting a value less than 1%. That is, G outputs P[e] if P[e] >= 1%, otherwise outputs 1%.

By construction, F is the better forecaster. But would this be evident from their track records? In other words, how many forecasts from each would we have to evaluate so that there’s a 95% chance that F outscores G? With the logarithmic scoring rule, it turns out to be on the order of a hundred million. With the Brier score, it is on the order of a trillion.2 We can quibble with the assumptions here but the point is that if a forecaster systematically overestimates tail risks, it is simply empirically undetectable.

The final barrier to assessing forecaster skill at predicting x-risk is that long-term forecasts take too long to evaluate (and extinction forecasts are of course impossible to evaluate). This can potentially be overcome. Researchers have developed a method called reciprocal scoring — where forecasters are rewarded based on how well they predict each others’ forecasts — and validate it in some real-world settings, such as predicting the effect of Covid-19 policies. In these settings, reciprocal scoring yielded forecasts that are as good as traditional scoring methods. Fair enough. But reciprocal scoring is not a way around the reference class problem or the tail risk problem.

There are many reasons why risk estimates may be systematically inflated

To recap, inductive and deductive methods don’t work, subjective forecasts are all over the place, and there’s no way to tell which forecasts are more trustworthy.

So in an attempt to derive more reliable estimates that could potentially inform policy, some researchers have turned to forecast aggregation methods that combine the predictions of multiple forecasters. A notable effort is the AI Impacts Survey on Progress in AI, but it has been criticized for serious methodological limitations including non-response bias. More importantly, it is unclear why aggregation should improve forecast accuracy: after all, most forecasters might share the same biases (and again, none of them have any basis for a reliable forecast).

There are many reasons why forecasters might systematically overestimate AI x-risk.3 The first is selection bias. Take AI researchers: the belief that AI can change the world is one of the main motivations for becoming an AI researcher. And once someone enters this community, they are in an environment where that message is constantly reinforced. And if one believes that this technology is terrifyingly powerful, it is perfectly rational to think there is a serious chance that its world-altering effects will be negative rather than positive.

And in the AI safety subcommunity, which is a bit insular, the echo chamber can be deafening. Claiming to have a high p(doom) (one’s estimate of the probability of AI doom) seems to have become a way to signal one’s identity and commitment to the cause.

There is a slightly different selection bias at play when it comes to forecasting experts. The forecasting community has a strong overlap with effective altruism and concerns about existential risk, especially AI risk. This doesn’t mean that individual forecasters are biased. But having a high p(doom) might make someone more inclined to take up forecasting as an activity. So the community as a whole is likely biased toward people with x-risk worries.

Forecasters are good at updating their beliefs in response to evidence, but the problem is that unlike, say, asteroid impact risk, there is little evidence that can change one’s beliefs one way or another when it comes to AI x-risk, so we suspect that forecasts are strongly influenced by the priors with which people enter the community. The XPT report notes that “Few minds were changed during the XPT, even among the most active participants, and despite monetary incentives for persuading others.” In a follow-up study, they found that many of the disagreements were due to fundamental worldview differences that go beyond AI.

To reemphasize, our points about bias are specific to AI x-risk. If there were a community of election forecasters who were systematically biased (say, toward incumbents), this would become obvious after a few elections when comparing predictions with reality. But with AI x-risk, as we showed in the previous section, skill evaluation is insensitive to overestimation of tail risks.

Interestingly, skill evaluation is extremely sensitive to underestimation of tail risks: if you assign a probability of 0 for a rare event that actually ends up occurring, you incur an infinite penalty under the logarithmic scoring rule, from which you can never recover regardless of how well you predicted other events. This is considered one of the main benefits of the logarithmic score and is the reason it is adopted by Metaculus.

Now consider a forecaster who doesn’t have a precise estimate — and surely no forecaster has a precise estimate for something with so many axes of uncertainty as AI x-risk. Given the asymmetric penalties, the rational thing to do is to go with the higher end of their range of estimates.4

In any case, it’s not clear what forecasters actually report when their estimates are highly uncertain. Maybe they don’t respond to the incentives of the scoring function. After all, long-term forecasts won’t be resolved anytime soon. And recall that in the case of the XPT, the incentive is actually to predict each others’ forecasts to get around the problem of long time horizons. The reciprocal scoring paper argues that this will incentivize forecasters to submit their true, high-effort estimates, and considers various objections to this claim. Their defense of the method rests on two key assumptions: that by exerting more effort forecasters can get closer to the true estimate, and that they have no better way to predict what other forecasters will do.

What if these assumptions are not satisfied? As we have argued throughout this post, with AI x-risk, we shouldn’t expect evidence to change forecasters’ prior beliefs, so the first assumption is dubious. And now that one iteration of the XPT has concluded, the published median estimates from that tournament serve as a powerful anchor (a “focal point” in game theory). It is possible that in the future, forecasters with reciprocal scoring incentives will use existing median forecasts as a starting point, only making minor adjustments to account for new information that has become available since the last tournament. The range of estimates might narrow as existing estimates serve as anchors for future estimates. All that is a roundabout way to say: the less actual evidence there is to draw upon, the more the risk of groupthink.

Beware Pascal’s wager: the dangers of utility maximization

For what it’s worth, here are the median estimates from the XPT of both extinction risk and sub-extinction catastrophic risks from AI:

To reiterate, our view is that we shouldn’t take any of these numbers too seriously. They are a reflection of how much different samples of participants fret about AI than anything else.

As before, the estimates from forecasting experts (superforecasters) and AI experts differ by an order of magnitude or more. To the extent that we put any stock into these estimates, it should be the forecasting experts’ rather than the AI experts’ estimates. One important insight from past research is that domain experts perform worse than forecasting experts who have training in integrating diverse information and by minimizing psychological biases. Still, as we said above, even their forecasts may be vast overestimates, and we just can’t know for sure.

So what’s the big deal? So what if policymakers believe the risk over a certain timeframe is 1% instead of 0.01%? It seems pretty low in either case!

It depends on what they do with those probabilities. Most often, these estimates are merely a way to signal the fact that some group of experts thinks the risk is significant. If that’s all they are, so be it. But it’s not clear that all this elaborate effort at quantification is even helpful for this signaling purpose, given that different people interpret the same numbers wildly differently.

For example, Federal Trade Commission chair Lina Khan said her views on the matter were techno-optimistic since her p(doom) was only 15%, which left experts bewildered. (For what it’s worth, that number is about a thousandfold higher than what we would be comfortable labeling techno-optimist.) It takes a lot of quantitative training to be able to mentally process very small or very large numbers correctly in decision making, and not simply bucket them into categories like “insignificantly small”. Most people are not trained this way.

In short, what seems to be happening is that experts’ vague intuitions and fears are being translated into pseudo-precise numbers, and then translated back into vague intuitions and fears by policymakers. Let’s just cut the charade of quantification! The Center for AI Safety’s Statement on AI Risk was admirably blunt in this regard (of course, we strongly disagree with its substance).

A principled, quantitative way to use probabilities in decision making is utility maximization through cost-benefit analysis. The idea is simple: if we consider an outcome to have a subjective value, or utility, of U (which can be positive or negative), and it has, say, a 10% probability of occurring, we can act as if it is certain to occur and has a value of 0.1 * U. We can then add up the costs and benefits for each option available to us, and choose the one that maximizes costs minus benefits (the “expected utility”).

This is where things get really problematic. First, some people might consider extinction to have an unfathomably large negative value, because it precludes the existence of all the human lives, physical or simulated, that might ever be born in the future. The logical conclusion is that x-risk should be everyone’s top priority all the time! It is reminiscent of Pascal’s wager, the argument that it is rational believe in God because even if there is an infinitesimally small chance that God exists, the cost of non-belief is infinite (an eternity in hell as opposed to eternal happiness), and hence so is the expected utility. Fortunately, policymakers don’t give too much credence to decision making frameworks involving infinities. But the idea has taken a powerful hold of the AI safety community and drives some people’s conviction that AI x-risk should be society’s top priority.

Even if we limit ourselves to catastrophic but not existential risks, we are talking about billions of lives on the line, so the expected cost of even a 1% risk is so high that the policy implications are drastic — governments should increase spending on AI x-risk mitigation by orders of magnitude and consider draconian measures such as stopping AI development. This is why it is so vital to understand that these estimates are not backed by any methodology. It would be incredibly unwise to make world-changing policy decisions based on so little evidence.

Forecasts of milestones suffer from outcome ambiguity

Is there a role for forecasting in AI policy? We think yes — just not forecasting existential risk. Forecasting AI milestones, such as performance on certain capability benchmarks or economic impacts, is more achievable and meaningful. If a forecaster has demonstrated skill in predicting when various AI milestones would be reached, it does give us evidence that they will do well in the future. We are no longer talking about unique or rare events. And when considering lower-stakes policy interventions — preparing for potential economic disruption rather than staving off killer robots — it is less critical that forecasts be justified to the satisfaction of every reasonable person.

The forecasting community devotes a lot of energy to milestone forecasting. On Metaculus, the question “Will there be Human-machine intelligence parity before 2040?” has an aggregate prediction of 96% based on over 1,300 forecasters. That’s remarkable! If we agreed with this forecast, we would be in favor of the position that managing the safe transition to AGI should be a global priority. Why don’t we?

The answer is in the fine print. There is no consensus on the definition of a fuzzy concept such as AGI. Even if we fix a definition, determining whether it has been achieved can be hard or impossible. For effective forecasting, it is extremely important to avoid ambiguous outcomes. The way the forecasting community gets around this is by defining it in terms of relatively narrow skills, such as exam performance.

The Metaculus intelligence parity question is defined in terms of the performance on graduate exams in math, physics, and computer science. Based on this definition, we do agree with the forecast of 96%. But we think the definition is so watered down that it doesn’t mean much for policy. Forget existential risk — as we’ve written before, AI performance on exams has so little construct validity that it doesn’t even let us predict whether AI will replace workers.

Other benchmarks aren’t much better. In short, forecasting AI capability timelines is tricky because of the huge gap between benchmarks and real-world implications. Fortunately, better benchmarks reflecting consequential real-world tasks are being developed. In addition to benchmarks, we need naturalistic evaluation, even if it is more costly. One type of naturalistic evaluation is to measure how people perform their jobs differently with AI assistance. Directly forecasting economic, social, or political impacts — such as labor market transformation or AI-related spending by militaries — could be even more useful, although harder to unambiguously define and measure.

Concluding thoughts

The responsibility for avoiding misuses of probability in policy lies with policymakers. We are not calling for forecasters to stop publishing forecasts in order to “protect” policymakers from being misled. That said, we think forecasts should be accompanied by a clear explanation of the process used and evidence considered. This would allow policymakers to make informed decisions about whether the justification presented meets the threshold that they are comfortable with. The XPT is a good example of transparency, as is this paper (though it is not about x-risk). On the other hand, simply surveying a bunch of researchers and presenting aggregate numbers is misinformative and should be ignored by policymakers.

So what should governments do about AI x-risk? Our view isn’t that they should do nothing. But they should reject the kind of policies that might seem compelling if we view x-risk as urgent and serious, notably: restricting AI development. As we’ll argue in a future essay in this series, not only are such policies unnecessary, they are likely to increase x-risk. Instead, governments should adopt policies that are compatible with a range of possible estimates of AI risk, and are on balance helpful even if the risk is negligible. Fortunately, such policies exist. Governments should also change policymaking processes so that they are more responsive to new evidence. More on all that soon.

Further reading

• The XPT report is titled Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament.

• Tetlock and Gardner’s book Superforecasting summarizes research by Tetlock, Barbara Mellers, and others.

• Scott Alexander refutes the claim that there is something wrong in principle with ascribing probabilities to unique events (we largely agree). Our argument differs in two key ways from the position he addresses. We aren’t talking about forecasting in general; just its application to policymaking. And we don’t object to it in principle. Our argument is empirical: AI x-risk forecasts are extremely unreliable and lack justification. Theoretically this could change in the future, though we aren’t holding our breath.

• A paper by Friedman and Zuckhauser explains why probabilities aren’t the whole story: two forecasts that have the same risk estimate might have very different implications for policymakers. In our view, AI x-risk forecasts fare poorly on two of the three dimensions of confidence: a sound basis in evidence and a narrow range of reasonable opinion.

• A paper by our Princeton CITP colleagues led by Shazeda Ahmed explains the epistemic culture of AI safety, which consists of “cohesive, interwoven social structures of knowledge-production and community-building”. It helps understand why practices such as forecasting have become pillars of how the AI safety community forms its beliefs, as opposed to the broader scientific community that centers practices such as peer review. Of course, we shouldn’t reject a view just because it doesn’t conform to scientific orthodoxy. But at the same time, we shouldn’t give any deference to the self-styled AI safety community’s views on AI safety. It is important to understand that the median member of the AI safety community holds one particular stance on AI safety — a stance that is highly contested and in our view rather alarmist.

• We have written extensively about evidence-based AI safety. Our best-known work includes the essay AI safety is not a model property and the paper titled On the Societal Impact of Open Foundation Models which was the result of a large collaboration.

Acknowledgements. We are grateful to Benjamin Edelman, Ezra Karger, Matt Salganik, and Ollie Stephenson for feedback on a draft. This series of essays is based on an upcoming paper that benefited from feedback from many people, including Seth Lazar and members of the MINT lab at Australian National University, students in the Limits to Prediction course at Princeton, Shazeda Ahmed, and Zachary Siegel.

Over the last few months, loss-of-control incidents at OpenAI and Anthropic have intensified concerns about AI safety. Warnings about existential risk have increasingly reached the broader public, alongside calls to slow AI development. Dario Amodei’s call to “pace the frontier” reflects the concern that safety efforts are not keeping up with AI capabilities.

The most prominent example was the OpenAI - Hugging Face incident, where hundreds of OpenAI agents got access to the internet and hacked Hugging Face to find how they were being graded on an evaluation. Over the last two weeks, the details of many other instances of such activity from OpenAI agents being evaluated have surfaced, such as agents using an old Wiki website to communicate with each other despite restrictions on such activity, or attacking a software repository to attempt to upload malicious software.

The AI safety community has viewed these incidents largely as a crisis for alignment.1 In this view, alignment will become harder over time as agents learn to reason covertly, and as a result, loss-of-control incidents are likely to become much more widespread and damaging as agents become more capable.

On the other hand, cybersecurity practitioners have largely viewed these incidents as consequences of companies failing to adopt basic security precautions. They do not see the incidents as a sign of AI reaching a new milestone in cybersecurity. This is also the dominant reaction in the tech community outside AI, which has largely treated the incidents as a result of ineptitude and negligence by AI companies.2

Both communities have made important points. But the polarization between them is counterproductive, and there are lessons from both the alignment failure and the security failure for understanding the path forward.

In this essay, we apply the AI as Normal Technology framework to synthesize the views of the safety and cybersecurity communities and offer a constructive middle ground. We think AI companies should be liable for what their agents do, and this should be clarified through policymaking. At the same time, recognizing their responsibility does not mean that preventing future incidents is a solved problem.

We identify three areas where investment is necessary to address loss-of-control risks: research to develop better methods for controlling increasingly capable agents, translating existing research and known control techniques into usable tools, and organizational changes to ensure that these tools are actually adopted. In our view, standards for organizational governance should be a key way to pace the frontier and pull AI companies out of the “move fast and break things” attitude they currently operate in.

This essay has three parts. In Part 1, we argue that alignment alone is not enough to prevent such incidents, and discuss technical, organizational, and policy interventions for improving AI control. We are cautiously optimistic that the right investments and policy interventions can allow AI control to keep pace with AI capability improvements. In Part 2, we discuss the impact of improving AI capabilities, such as agent swarms, on cybersecurity. In Part 3, we share how our views on AI safety have changed in light of new evidence.

A summary of our argument:

• We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions.

• We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other specific risks, such as biorisk and risks from military AI.

• We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community.

• How should we reason about AI’s impact on cybersecurity? It’s plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders.

• How the AI as Normal Technology framework has evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity.

• At the same time, many distinctive claims of AI as Normal Technology have held up, and it remains valuable for understanding AI’s societal impacts. In particular, we think recent incidents support our continuity hypothesis — the behavior of “rogue” agents became apparent and widely publicized while they are still far from causing serious harm and incompetent at hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework.

• Finally, we think AINT is particularly valuable for analyzing these incidents because it provides a framework for synthesizing the AI safety and cybersecurity communities’ views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.

This table summarizes our argument. While the AI community has in principle emphasized the notion of “defense in depth”, in practice, alignment has by far been the intervention that received the most attention and investment. On the other hand, the cybersecurity community has viewed control as largely a solved problem. In our view, to protect against loss-of-control incidents from legitimate parties, we need to urgently invest in control so that it keeps pace with capability improvements. Neither alignment nor control help against malicious users, so we need downstream defenses and resilience to alleviate the impact and severity of risks. All of these efforts can be shored up by policy interventions. The table emphasizes interventions for cybersecurity, but we should simultaneously invest in defenses against other risks, such as biorisk.

Table of contents

Part 1: We can get better at AI control through technical and policy interventions

• Alignment is helpful but not sufficient for preventing safety incidents

• OpenAI did not use known control interventions that would have prevented the incident

• Existing organizational governance norms would have prevented the incident

• Why has the AI community under-invested in control?

• Will control keep pace with AI capabilities?

• AI control should become a job (and a part of every job), just like cybersecurity

• AI policy can incentivize AI control

Part 2: There is great uncertainty about how AI will impact cyberrisk. But what we need to do regardless is relatively clear.

• The specific threat that is urgent is cyberrisk

• How will autonomous cyber capabilities and open-weight models affect the attacker/defender balance?

• To understand what open-weight models mean for the future of cybercriminal activity, we need to understand the past correctly

• The majority of cybercriminals are surprisingly low-tech, and the barrier is monetization, not exploitation

• So, how much will AI help cybercriminals?

• Are we about to see harmful autonomous agents?

• Why the Morris worm is a good analogy for what might be about to happen

• What the future of cyberdefense might look like: the roles of people and defensive AI

• What are the gaps in the state of cyberdefense against AI?

Part 3: What AI as Normal Technology got wrong — and right — about AI safety

• Do risks arise from development or deployment?

• How should we deal with rogue agents?

• The continuity hypothesis

• Jaggedness and risk-specific defenses

• Is AI safety on track?

Conclusion

Part 1: We can get better at AI control through technical and policy interventions

In this part, we focus on AI agents that are operated by legitimate actors — such as consumers, businesses, and AI developers — who don’t intentionally use them to cause harm. This was the case for the OpenAI - Hugging Face incident. In Part 2, we focus on malicious actors who want to use agents for intentionally causing harm, such as using them for cyberoffense.

How can we prevent AI agents from taking harmful actions? There are two broad kinds of interventions. One is AI alignment. This involves changing the AI system itself, such as by fine tuning or using reinforcement learning with human feedback (RLHF), to make it less likely to take harmful actions or give harmful responses. Alignment has been pivotal for the commercial success of AI so far.

The other intervention is AI control: interventions made outside the model weights to prevent harmful actions — even if the agent is misaligned. This includes improvements to sandbox security (to prevent AI models from taking unanticipated actions outside the sandbox), implementing the principle of least privilege, comprehensive logging, automated tripwires for unsafe behaviors, rapid shutdown mechanisms, and monitoring the agent to detect and prevent harmful actions. These mechanisms allow us to prevent unsafe actions even if agents are misaligned and try taking harmful actions.

Roughly speaking, AI control can be thought of as cybersecurity against AI agent adversaries. Whereas cybersecurity has traditionally been concerned with human actors who seek to compromise or exploit a system, AI control uses these same principles to prevent unwanted actions by AI agents.

Unlike cybersecurity, control interventions target agents being used within an organization (as opposed to human adversaries or external agents). This makes the problem both more and less challenging than traditional cybersecurity. It is less challenging because the agent is directly controlled by the organization (rather than being an unknown adversary), so its operating conditions can be closely monitored and intervened on. It is more challenging because these agents are often deployed by users who might have escalated privileges, and imposing security constraints also imposes constraints on usability which could be resisted or circumvented by users.

Analysis of the OpenAI - Hugging Face incident from the AI safety community primarily treats it as an alignment failure. The main intervention that the AI safety community has focused on is improving technical methods for AI alignment. When AI control is discussed, it is often seen as a stopgap.3 On the other hand, in the cybersecurity community, AI control is often seen as a solved problem for preventing the kinds of issues that arose in the OpenAI - Hugging Face, as it only requires the adoption of known security techniques.

In this part, we discuss why marginal investments in control might be better suited compared to alignment for resolving such incidents. While we agree with the AI safety community that alignment is helpful in reducing the prevalence of such incidents, we think that alignment alone will not suffice to eliminate them. The incidents show that AI control has not been prioritized by AI companies, despite there being known methods that would have prevented the incident.

A common response to recommendations for implementing AI control is that while they suffice for the current set of models, they might not suffice for future, more capable models. Here, we depart from the consensus in the security community: we agree that existing control interventions might not be enough to prevent harm from future, more capable models. But we argue that there is a lot of headroom in improving control interventions, and the right policy interventions (such as imposing liability for loss-of-control incidents, transparency for near misses, and whistleblower protections) can lead to dramatic improvements in controlling AI systems.

Alignment is helpful but not sufficient for preventing safety incidents

We have argued that model alignment is not enough for AI safety. To recap: when a model is deployed, it doesn’t have enough context to know if the conditions it’s being deployed in are safe or not — a lot of that context is outside the model. Alignment isn’t a solved technical problem, and there are unknown technical advances needed to align models and prevent harmful actions.

For example, when a model is deployed to find vulnerabilities, it might be used for cyberoffense (to develop exploits) or defense (to fix them). Alignment can be helpful for harm reduction; for example, models can be trained to tilt the offense-defense balance towards cyberdefenders by finding vulnerabilities but not contributing to finding exploits, or analyzing the source code (which defenders have access to) but not analyzing binaries. But alignment is not foolproof, since beyond simple interventions (such as analyzing the code but not the binaries), it is unclear if the model is being used for a defensive or offensive task based on the context given to the model.

Similarly, for loss-of-control incidents like the OpenAI - Hugging Face one, the agent might be expected to carry out the attack in a “simulation” of its cyberoffense capabilities, but not attack Hugging Face in the real world. In this case, it might have been relatively clear to the agent that the Hugging Face infrastructure was not a simulation, but you could imagine cases where the distinction is not so clear, such as when the agent is tasked with escaping its sandbox to estimate loss-of-control risks. There are many other edge cases where alignment would fail.

Things become even more blurry once a model is deployed to customers and businesses as opposed to being restricted within AI companies. The contexts within which AI companies use agents are relatively bounded, making the alignment problem easier to address. But customers outside companies can use AI agents in boundless ways, making it harder to judge whether an action is legitimate or a safety hazard based on the information available to the model.

Despite these challenges, the mainstream view in the AI safety community is that to make AI deployments safe, we need to invest in alignment. This is also reflected in the safety efforts of AI companies, which often treat interventions outside the model (such as AI control interventions) as a stopgap. For example, when discussing monitoring interventions that OpenAI deployed for Astra, the company’s report noted: “These safeguards cannot replace good alignment of our models as capabilities increase, and our goal is for future models to be aligned well enough that these safeguards are never triggered.”

To be clear, we don’t mean to underplay the importance of making advances in alignment, and we think developing a better understanding of what causes harmful model behaviors is an important research direction. For example, over the last few weeks, OpenAI and Anthropic have both shared how RL environments that are misspecified or those that reward shortcuts induce reward hacking in models. Addressing these shortcomings might reduce how often models take harmful actions. But they still won’t guarantee safe behavior.

Given the shortcomings in alignment science, deployers of AI agents must complement alignment with broader interventions in AI control. Unlike alignment, when it comes to control, there are many existing techniques that would have prevented the incident.

OpenAI did not use known control interventions that would have prevented the incident

In the aftermath of the incident, reports by OpenAI and METR revealed more details. It is clear from the reports that the agents acted in ways that weren’t intended by the evaluators, and we think it is fair to describe the actions of the agent as misalignment.

But note that the incident occurred when OpenAI had disabled most mechanisms for controlling their agents. While the company had implemented monitoring for most internal uses, it hadn’t implemented it for evaluations. On top of that, the evaluation used a different set of prompts and harnesses compared to the production Codex harness.

What would have happened had the company used known control techniques? OpenAI found that using the production Codex harness and system prompt reduced the propensity to compromise out-of-scope infrastructure by more than 100×. Their automatic review would have flagged most dangerous actions in the tested rollouts. And its current chain-of-thought monitoring would have raised an alert more than a day before the Hugging Face breach.

In fact, existing AI models are already helpful for monitoring and control. METR used OpenAI’s publicly available models to carry out large parts of their analysis. According to their report, these agents made errors, showed poor judgment, and sometimes adopted the perspective of the agents whose behavior they were reviewing. Yet, an investigation of this scale would have been infeasible in six days by three researchers without them. Monitoring does not have to be perfect to be extremely useful when it augments skilled humans rather than replacing their judgment.

More importantly, before the incident occurred, there was a warning sign that something had gone wrong — one of the services that the agent used to communicate had an internal outage. Instead of launching a deep dive into the root cause for these evaluations, OpenAI restarted the evaluations in the same setup after fixing the one bug they had uncovered.

This shows that the culture within AI labs was just as culpable for the incident. Engineers work long days, spin up thousands of experiments, and there’s insufficient human oversight of these experiments. As Joshua Saxe notes:

…the culture among the training teams and the evals teams at the labs has a Wild West feeling to the whole thing. Everybody’s under a ton of pressure to move really quickly. There’s enormous time pressure to release new models, tremendous awareness of how any given lab is doing relative to the others in terms of capability. And the field has evolved super fast in the four years since ChatGPT launched in fall 2022. The whole thing has been kind of a blur — anybody who’s worked in the labs in that time has been working sixty-hour weeks the whole time. So it’s not surprising to me that you could have a kind of grad-student computer-science-lab culture around the security of these model training runs, which I think is what’s the case, and I think that’s why you’ve seen these lab escapes. It is ironic, because frontier AI people talk about safety all the time — but that irony is a real thing. The security practices are not very good, because of all those dynamics.

To OpenAI’s credit, they have been transparent about the incident even where it doesn’t suit their interests. For example, they conducted additional analyses with the Codex comparison in the report and put controls in place to improve monitoring after the incident.

But given the competitive dynamics in the AI industry, there is a lot of room for policy to require improvements to internal incident response, accountability and liability for loss-of-control incidents, and improving the culture around security and safety.

Existing organizational governance norms would have prevented the incident

Even setting aside policy interventions, one reason for these incidents is that at the organizational level, AI companies function like startups. While their revenue and valuations have grown immensely, by all accounts, they lack the organizational processes that are the norm in any other industry. As a result, they haven’t developed the organizational maturity to get visibility into risky experiments and deal with the fallout.

Rather than behaving like startups, AI companies need to behave like more mature organizations. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. In fact, in the first instance, proposals to “pace the frontier” should be about addressing these organizational failures rather than aiming for technical breakthroughs alone.

Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. Without such governance, of the thousands of employees and hundreds of teams at a company such as OpenAI, you need just one team to deploy agents irresponsibly for incidents to reoccur.

To get a sense of how standard such organizational norms are, consider any other industry where a company claims to develop revolutionary technology. It is hard to imagine the company reporting that its technology had caused real-world harm (such as hacking another organization) and expecting to continue running such experiments without meaningful oversight. In fact, AI companies’ claims about the power and risks of their technology makes the case for organizational oversight stronger. We do not need to resolve disagreements about unsolved technical challenges to expect companies to take basic precautions.

We speculate that one reason why AI companies haven’t adopted standard governance mechanisms is that they have grown too quickly and have tried to maintain their startup mentality despite this period of extraordinary growth. Companies in most other industries see more gradual growth that allows the time for building organizational norms. AI companies resemble Facebook and social media companies that grew quickly with the “move fast and break things attitude”, but ultimately needed to put in guardrails and organizational governance standards in light of incidents like Cambridge Analytica.

Fortunately, it seems like AI companies are recognizing the importance of such changes. In Amodei’s letter calling for pacing AI, he refers to AI control requiring “operational excellence”. He says: “Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution.” Organizational governance is one way to pace the frontier and prevent execution errors that cause real-world harm.

Why has the AI community under-invested in control?

If there were known control techniques to prevent the incident from occurring, why didn’t OpenAI invest in them? In the previous section, we discussed one potential reason could be the culture within AI companies that prioritizes the pace of progress over safety and security.

But there is another reason, and it relates to the difference in how AI companies view alignment and how they view control. Alignment is treated as a key component of every model release; billions of dollars have been spent on developing methods and collecting data for alignment. But OpenAI did not implement basic control interventions during evaluations. Across AI companies and the safety community, alignment is seen as the key problem to track, and control as a stopgap. We think there are a few reasons why the gap persists:

The epistemic gap between cybersecurity and AI: There are many known techniques in cybersecurity that could improve the state of AI control. But these communities can talk past each other when it comes to implementing solutions. After the OpenAI - Hugging Face incident, many security practitioners pointed out that OpenAI did not take basic preventative steps to address the incident. On the other hand, advocates for AI safety (both within and outside AI companies) correctly argued that improvements in AI capabilities mean that known control interventions might not work for very long.

There is a lot of value in synthesizing these perspectives and bridging the gap between these communities. AI companies have ignored known control interventions and should invest in them. But known interventions might not be enough and need to be dramatically updated to account for increasing agent capabilities. As one example, consider the recent post by Artem Dinaburg, who shows that capable agents can develop new exploits, so we can no longer assume that standard VMs can be used to sandbox capable agents. Now that companies routinely use millions of agents in such sandboxes (in either production or research settings), we need to dramatically increase the scrutiny that sandbox security receives, such as by stress-testing and improving sandbox security through offensive agents on a routine basis with models of increasing capability. We need similar efforts to harden cyber infrastructure across the board against capable cyber agents.

At the same time, there are many straightforward interventions that do not need new technical breakthroughs. As an example, AI agents today are already helpful for control despite the lack of specific interventions on improving control and monitoring. For example, METR used off-the-shelf models in their analysis, and still found them useful. We expect that through the right incentives, we could continue to broaden our control toolkit. These could include fine tuning models specifically for improving control, developing benchmarks and leaderboards to improve monitoring, developing layered mechanisms (such as a separate layer of agents monitoring agent monitors) to improve the robustness of control and monitoring efforts, using ensembles of models with different model families at different temperatures, etc.

The tradeoff between usability and control: Even when security experts are involved in AI development, they may find it hard to advocate for control interventions because these interventions can trade off against usability. For example, when Claude Code was initially launched, many people used it without restrictions using a mode called “dangerously skip permissions”. This was despite the obvious risks of allowing the agent to take any action without permission.

But this tradeoff is not inherent. Over the last few months, Anthropic and OpenAI have both released auto review modes in Claude Code and Codex. These modes run a classifier on the proposed actions being taken by the agent to prevent harmful or destructive actions, detect potential prompt injections in content that the agent looks up, and, when necessary, pause the agent to raise concerns for the user’s approval.

This is not just a security improvement compared to no human intervention; it is also an improvement compared to requiring human approval for each action. The latter approach quickly leads to “alarm fatigue” and the approval becomes a rubber-stamping exercise. A recent Anthropic evaluation found that auto mode prevented harmful actions compared to both full human approval and the “dangerously skip permissions” modes.

Coding agents use dozens of such control interventions. Other interventions include running agents within sandboxes, user settings for which commands need user approvals, and access protections for agent configurations set by the user to prevent agents from editing them.

This shows how we can improve both security and usability via new control interventions. We think there is a lot of headroom for improving control interventions without compromising on usability. Notably, despite its usefulness, auto mode was introduced only in March 2026 — over a year after the release of Claude Code, and well after coding agents became mainstream, and despite the fact that it didn’t require new technical breakthroughs.

Conflating capability and power: In the AI as Normal Technology essay, we pointed out that those with the superintelligence worldview often conflate the capabilities of AI systems with how powerful they will be (in the sense of being able to act in the real world). Implicit in this is the assumption that once models become much more capable, they will also be able to exert more real-world influence.

But as we discuss in the essay, capable is not the same as powerful. If you assume capable models will be given power, alignment is all you have left. On the other hand, under the AI as Normal Technology view, we can choose whether to grant power to AI systems. Companies can be incentivized to implement control techniques through policy interventions such as increasing transparency, auditing, and imposing liability for safety incidents.

Differences between two views of how AI loss of control could arise. Applying it to the OAI/HF incident, In the superintelligence view, fixing misalignment is the main technical intervention; in our view, fixing the poor monitoring, lack of human oversight, sandbox security, and other control interventionsare likely to be much more productive. (Figure from AI as Normal Technology.)

Will control keep pace with AI capabilities?

One response to prioritizing control-based interventions is that control will stop being effective as AI systems become more capable. For example, there has been a lot of discussion around OpenAI’s architectural choices for GPT-6 Astra reducing the ability to conduct chain-of-thought monitoring.

But note that this is not just a matter of forecasting what technical capabilities are needed or predicting the inevitability of a race to the bottom on safety. There are many technical and policy levers that can incentivize investments in control. If companies are liable for the harms caused by out-of-control systems, control interventions would be a straightforward way to make progress without compromising on control.

At a technical level, even if chain-of-thought monitoring is no longer effective for oversight, there are many orthogonal control mechanisms for AI control. We can monitor the environment and sandbox state, tool calls, inter-agent messages, and internal activations.

In fact, most external monitoring relies on such channels since the raw chains of thought aren’t available. Based on public documentation, auto mode runs on tool calls and inputs to the model, not the model’s chains of thought. In our own work, we have found such external control mechanisms to be extremely valuable.

Chain-of-thought monitoring could also reduce the efficacy of monitoring, since the chains of thought could contain (incorrect) assertions such as the setting being a simulation that throws off the monitor. A recent Anthropic report notes:

Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions. Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted. When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%.

Still, losing channels for monitoring AI systems is a technical choice by AI companies, which can be disincentivized by policy. Here, our thesis is that if companies need to internalize the costs of out-of-control AI systems (such as by policy changes that clarify or shore up liability), they will make the necessary investments. We talk about some such proposals in the next section.

The need for control also creates market incentives for third-party companies to develop tools and provide the expertise for AI control within other orgs — both AI companies as well as businesses and third parties adopting AI. Some companies, like Apollo Research, have this explicit thesis for motivating their work. They have worked with AI companies as well as businesses outside the AI industry.

While tech companies have the expertise to implement these interventions in-house, how would AI control interventions scale beyond these companies? As an example, consider cybersecurity. Cybersecurity consulting is a rapidly growing share of consulting companies’ revenue, which often carry out implementation work in the area. For companies that lack the scale or technical expertise for implementing AI control in-house, this function might be realized through a combination of off-the-shelf products for AI control and consulting companies helping set up this internal expertise.

We might also see application-specific controls, since some controls require context about how the application is used. For example, we might see companies specializing in legal AI implementing controls to reduce fabrications, and those specializing in medical AI applications implementing controls for users’ medical data etc.4

Developing agent infrastructure could also play a role. This could include providing agent-specific delegation credentials with the right level of access instead of letting agents run with a user’s escalated credentials and identification to tie agents to real-world users who commissioned the task. We expect that the need for such interventions will grow dramatically over the next few years. It won’t just be a matter of building tools; we also need to train specialists in AI control.

AI control should become a job (and a part of every job), just like cybersecurity

In AI as normal technology, we wrote:

As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control. If this seems radical, note that this kind of near-total redefinition of the concept of work has happened previously. Before the Industrial Revolution, most jobs involved manual labor. Over time, more and more manual tasks have been automated, a trend that continues. In this process, a great many different ways of operating, controlling, and monitoring physical machines were invented, and what humans do in factories today is a combination of “control” (monitoring automated assembly lines, programming robotic systems, managing quality control checkpoints, and coordinating responses to equipment malfunctions) and some tasks that require levels of cognitive ability or dexterity that machines are not yet capable.

This trend is underway in professions that are at the leading edge of AI adoption, notably software engineering. More and more software engineers today are agent operators, not manual coders. But this incident shows that a few more things are needed:

• Some people will probably need to specialize in AI control, especially when it comes to running large numbers of highly capable agents.5

• So far, engineers are learning AI control through trial-and-error and osmosis. There needs to be more of an effort to extract its principles and teach people to quickly get up to speed on how to monitor powerful agents in real time.

• As mentioned above, there need to be organizational changes and governance norms that complement improvements in control tooling and training. What sort of team and processes does one need in order to be running tens of thousands of reduced-guardrail agents in parallel? We don’t know, but certainly more than what OpenAI appears to have had in place.

We expect AI control to continue to be important as AI agents improve because of the agent trilemma: developing general-purpose AI systems that have no human supervision that are deployed in settings with a high cost of errors has proven to be extremely challenging.

One reason for this is the lack of reliability in AI agents. We expect that even as agents become reliable at tasks of increasing complexity, their reliability will continue to lag capability, creating the need for human supervision at tasks that require frontier capabilities.

The other reason is safety: when errors have high costs, even rare occurrences need to be prevented, and using human supervision for AI control would outweigh the benefits of fully autonomous deployment.

Finally, AI control needs to become a research field. It is quite surprising that this is not already the case. AI control is often treated as an application of known cybersecurity principles to AI, with a need to build better tooling. But when the adversary is an AI agent (or a swarm of them), rather than a human, the game changes. A few examples:

• Sandboxes that are effective against human adversaries have proven inadequate for controlling AI. How can we build extra-hardened sandboxes? Perhaps it’s time for more research on formal verification techniques.

• “Security policies” for agents are often specified informally in prompts, and the resulting grey area is the cause of many alignment and control failures. Can we build a translation layer that turns humans’ natural-language intent into a formal specification that the humans can review and verify, and can then be automatically enforced?6

• As we’ve argued, monitoring agents with other agents is helpful and essential, but currently unreliable. Can we train models and agents to excel at the task of reliable monitoring?

One would think that the cybersecurity research community would have seized this moment with gusto. AI control is a source of interesting, important, and urgent problems, arguably more so than any development in the last few decades. Instead the community has reacted with a collective yawn. Forget a new research field — there are close to zero papers in traditional cybersecurity conferences that tackle this new set of problems. (In contrast, prompt injection appears to have attracted a decent amount of research, but note a key difference: in that scenario, AI models are targets, not attackers.)

In other words, it appears that this community’s reaction to the incidents is that this is just a problem of implementation, not research. If so, something has gone seriously wrong. If AI control techniques sit still while AI’s offensive capabilities continue to advance, control will become less effective over time.

AI policy can incentivize AI control

AI companies have two business incentives for investing in AI control: reputational costs and legal costs. On the former, businesses and individuals using AI models will hesitate to give autonomous agents access to consequential decisions if AI failures can take expensive actions or create liability. On the latter, AI companies might themselves face liability for deploying AI agents that lead to safety failures.

Perhaps as a result of these incentives, after the Hugging Face incident, OpenAI temporarily stopped evaluations on ExploitGym, stopped training and inference for the internal models, rebuilt compromised infrastructure, restricted research access to the underlying models, explicitly accepted costs to research and development speed, and increased investments in AI control.

But these incentives still leave gaps: customers often cannot assess safety, companies might ignore safety interventions because they expect the benefits outweigh the costs, and an unclear liability regime and the lack of standards in AI insurance could make it hard to assess the consequences for unsafe deployments.

AI policy needs to systematically close these gaps. Building on the recommendations in AI as Normal Technology, we see three roles for policy for helping alleviate loss-of-control incidents: clarifying and strengthening liability, addressing market failures that liability does not resolve, and requiring transparency to improve public awareness and reduce uncertainty about AI’s impacts.

1. Companies should be responsible for harms caused by the agents they operate, even when they did not intend the harmful actions. This responsibility should extend to internal uses during development and evaluation, not just to products released to customers. Existing tort law already provides avenues for liability, but its application to AI leaves important questions open, including what precautions count as reasonable and when companies can be held responsible for unexpected behavior.7 Policymakers could, for example, clarify that running powerful agents without appropriate containment and monitoring is negligent. The fact that an agent was supposed to behave safely should not excuse the failure to implement controls against the possibility that it wouldn’t. Liability can also incentivize more investment in recruiting and training AI control specialists to prevent such incidents.

2. Market failures remain even when companies are liable, and policy must address them. Even if companies are clearly liable for the harm caused by their use of AI, this still leaves the scope for market failures when reckless action is not disincentivized by liability. For example, what happens if a company deploying agents recklessly is valued less than the harm it causes? One response is to require insurance as a mechanism to both incentivize safer deployment (to reduce premiums) and to compensate victims.

A second example of market failures is when investments in safety have positive externalities (that don’t translate to revenues). For example, giving vulnerable cyberdefenders such as schools, hospitals, and critical infrastructure operators access to capable models can help them find and fix vulnerabilities, and benefits society as a whole. One way to incentivize this is to create public support for defensive research and the adoption of defensive AI. Policy should also enable coordination between AI companies and defenders, including ways for legitimate defenders to investigate attacks without being blocked by overbroad safety refusals. We discuss these defensive uses in Part 2.

3. Transparency interventions should make it possible for outsiders to identify risks to control and assess whether companies are managing them. We proposed many interventions to improve transparency and reduce uncertainty about AI’s impacts in AI as Normal Technology, including incident reporting, independent auditing, whistleblower protections, and safe harbors for safety research. The threat of incurring liability can make companies more reluctant to investigate and disclose risks if doing so could increase their legal exposure. Transparency requirements help prevent companies from responding to liability by concealing problems rather than fixing them.

For example, one objection to our proposals for AI control is that investing in control might make it unclear when “near misses” occur. Like in the Hugging Face incident, suppose OpenAI agents attempt to hack an external organization, bypass a few basic filters, and are only caught as a result of new/advanced monitoring techniques. This constitutes a “near miss”. Requiring incident reporting for near misses could allow us to understand where alignment failures continue to persist and what control mechanisms function well to prevent loss-of-control incidents. Arnold and Llerena discuss other potential improvements to existing laws for transparency laws.

How should we enact transparency and auditing requirements? One proposed mechanism is to rely on independent verification organizations (IVOs): government-authorized expert organizations that develop technical criteria and verify whether AI companies meet safety standards. For such scrutiny to be useful for AI control, evaluators would need meaningful access to the systems and evidence required to test whether controls work, including in internal research settings. While we are not attached to a particular institutional mechanism for enabling external access, we are cautiously optimistic that interventions to improve transparency, including IVOs, would allow the kind of oversight that is required to discover failures and assess the precautions that AI companies take.

Part 2: There is great uncertainty about how AI will impact cyberrisk. But what we need to do regardless is relatively clear.

In this Part, we argue that:

• There is a crisis, but it is specifically about cybersecurity, not generally about AI safety, alignment, or loss of control.

• Financially motivated cybercriminals — just like legitimate enterprises — face adoption barriers that make it hard for them to quickly adopt new AI capabilities to cause more harm. It isn’t clear how much of a boost agents will give them in the short term.

• But this is no cause for comfort. There are many kinds of non-financially motivated threat actors including state-sponsored hacking groups, terrorists, hacktivists, vigilantes, and people motivated by personal grievances. There is also the possibility of rogue agents not under the control of any person.

• Most of these are not rational actors and it is very hard to predict how much additional harm they will cause by taking advantage of advanced AI. Still, we must act as if attacks stemming from these sources will become very common.

• Unfortunately alignment and control don’t help against bad actors using open-weight models, because those safeguards can and will be stripped, and the offensive capabilities of open-weight models tend to catch up quickly. We must rely on downstream defense and resilience, the latter two layers of our four-layer framework.

• There have been many warnings before this year, but until a few months ago, we lacked the urgency to invest adequately and move quickly. Fortunately, there are signs that that is changing. But there are important gaps.

The specific threat that is urgent is cyberrisk

Suppose we agree that loss-of-control incidents are going to become common. So what? Of course, out-of-control agents could carry out autonomous cyberattacks, and we need to get better at defending against them, which we discuss below. But what else? We argue that there just isn’t a whole lot else that out-of-control agents can autonomously do to cause harm that malicious actors couldn’t. As a result, we should invest in defenses that would help regardless of whether the risk is from malicious actors or rogue agents.

In the superintelligence view, frequent loss of control means that the AI safety battle has already been lost. In addition to cyberrisk, AI could amplify biorisk, and even more broadly, “CBRN” risks, which stands for chemical, biological, radiological, and nuclear. Superintelligent AI might persuade people — such as those in charge of critical infrastructure or high-stakes decisions — into acting against their own interests. As AI becomes smarter and more useful, CEOs, politicians and others will have no choice but to rely on AI for consequential decisions, amplifying these risks. And as we get closer to the possibility of “recursive self-improvement” of AI, out-of-control AI could lead to a harmful intelligence explosion. In short, the view is that once the loss-of-control threshold has been crossed, the risks are endless and self-amplifying.

We strongly disagree. In AI as Normal Technology, we argued that this view of AI fundamentally misunderstands the nature of intelligence (including human intelligence). Building on that perspective, we argue that cyberrisk is special in two ways: in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.

Cyberrisk has many specific properties that make superhuman offensive capabilities achievable (and largely already achieved). In general, some capabilities are like “making a building taller” — there is no real limit to achievement. Others are like “making a ball rounder” — there is a limit. One major point of departure between our worldview and the AI safety worldview is that we think tasks like chess or cybersecurity that are like making a building taller are the exception rather than the norm. In cybersecurity, like chess, there is an endless amount of training data (and more can be generated on demand), tasks (vulnerabilities / problems) can be arbitrarily complex, and speed makes a big difference.

In contrast, turning to the other risks, we stand by our prediction that superhuman persuasion ability is largely a myth. The confusion arises from carelessness about the term persuasion. Even GPT-4, an ancient model, is superhuman in certain situations at persuading people on political and social issues by presenting evidence. But this implies nothing about whether a model can “persuade” — trick — a trained operator to give up nuclear codes.

Similarly, AI’s impact on biorisk is a legitimate concern, but for the foreseeable future it is more in the vein of democratizing capabilities that already exist rather than developing new superhuman capabilities. As for people in positions of power delegating consequential decisions to AI, our perspective remains that this is extremely problematic regardless of loss-of-control risks, and must be resisted.

In short, our view is that (1) cyberrisk demands special, urgent attention (2) more broadly, anticipating and defending against specific risks is better than trying to solve “AI safety” as a catchall (3) there is a big difference between two concepts that are often conflated: loss of control over individual agents (or swarms of agents) and some sort of (usually ill-defined) civilizational loss of control over AI as a whole. Even in a scenario where the former is common and normalized, we have many levers to avoid the latter.

So let us turn to what we think is the critical question for now, which is how autonomous cyber capabilities will affect the attacker/defender balance. If we can ramp up defensive measures to keep that balance under control, we don’t have to panic about loss-of-control per se.

How will autonomous cyber capabilities and open-weight models affect the attacker/defender balance?

We think the cyber capabilities of today’s frontier models will diffuse to open models soon, as they have repeatedly so far. A worldwide ban on open-weight models is highly unlikely, and at any rate, defenders must prepare under the assumption that it won’t happen.8 We must assume that today’s frontier cyberoffense capabilities are widely available to threat actors in a matter of months, not years.

Alignment and control interventions only work when used by legitimate actors, since guardrails on open models can be stripped. So we think addressing risks from bad actors using advanced offensive AI capabilities is the grand challenge for cybersecurity.

Let’s start with cybercriminals — financially motivated actors. Assuming that the cyber capabilities of powerful open-weight models continue to improve rapidly, and that they will be available without guardrails, how much will it help cybercriminals? There are two key variables: how quickly cybercriminals will adopt these tools, and whether AI helps overcome critical barriers or if it impacts more peripheral aspects of their workflow.

These two questions are strikingly similar to the questions that pertain to how much AI will help economic growth. The reason we are skeptical about rapid growth is because we think organizational barriers result in slow diffusion and because in many cases the part that AI accelerates wasn’t the slow one to begin with.

Now let’s apply that analysis to cybercriminals. They certainly don’t face the same level of organizational barriers as defenders. But it is not zero. And there is a surprising answer to the question of whether finding new exploits is a critical capability.

To understand what open-weight models mean for the future of cybercriminal activity, we need to understand the past correctly

In the early decades, the cybersecurity community framed cyber-defense as a technical problem — building and deploying bug-free systems. But this approach didn’t work. We do have a lot of tools in our toolkit, including formal verification, safer programming languages, and penetration testing, but they are costly (in terms of programmer time, security analysts’ time, or software performance), so we don’t deploy them nearly enough for invulnerable systems to be a remotely realistic goal. Besides, human behavior in complex systems remains unpredictable and a weak link.

Instead, we made progress by treating cybersecurity as a primarily economic problem — raising the cost for attackers and lowering benefits; and only investing in security up to the point where the (capturable portion of the) benefits are worth the costs. Specifically:

• Bug bounties and patching culture reflect the assumption that consumers and enterprises are constantly running exploitable software; the defenders’ goal is instead to simply outbid and outrun attackers.

• Cyber insurance takes this idea to the extreme. Firms buy insurance because driving the probability of breaches to zero is not economically achievable. The cost of recovery from inevitable breaches becomes a budget item that competes with the budget for preventive measures.

• Defenders go after criminal revenue rather than technical attack capability. A famous economics-of-security insight in the late 2000s was that attackers are decentralized but payment networks are a centralized chokepoint, and going after these turned the tide on various kinds of scams.

• Much security policy is now about reallocating liability because liability changes behavior. Notably, forcing banks to internalize the cost of ATM and credit card fraud forces them to invest in security up to the socially optimal amount.

Overall this approach has been successful, and has driven cybercrime from a fundamental constraint on the digital economy to a problem that can be priced and managed. But running in permanently-insecure mode has its downsides. More on that in a minute.

The majority of cybercriminals are surprisingly low-tech, and the barrier is monetization, not exploitation

The defenders’ strategy of focusing on the economics — and the fact that cybercriminals are fundamentally economically motivated rational actors — means that the key constraint for attackers is monetization, not exploitation. Breaching is not the hard part — it’s what to do after the breach. Over the decades, cybercriminal organizations have evolved toward excellence in monetizing compromised victims, not their ability to find vulnerabilities. The vulnerabilities that they exploit are largely mundane — phishing/social engineering, unpatched servers, exposed remote services, compromised credentials — and when they do need zero-day vulnerabilities, they purchase them instead of developing them in-house.

As one small but illustrative example, credit card networks’ hostility to criminal activity forced cybercriminals to turn to cryptocurrency. In fact, crypto was essential for the success of ransomware, today’s dominant form of malware. But ransomware gangs found that victims were often confused about how to pay using crypto, and needed reassurance that payment would result in their data being returned. This required the gangs to set up customer service centers! The parallels between cybercriminal organizations and legitimate businesses don’t end there, as discussed in the paper cybercrime is entrepreneurship.

In short, the bottlenecks to scaling tend to be high-touch and human. This explains why the diffusion of cutting-edge technology tends to be slow. As Joshua Saxe puts it: “There’s no reason to expect a ransomware gang in Belarus to adopt AI faster than Deloitte does in its consulting work.”

This insight also applies to the attack surface opened up by AI itself. Many commentators (including us!) have been predicting that we’ll start to see lots more prompt injection attacks in the wild, but they haven’t panned out so far. One possible reason is that this is way too high tech for mainstream attackers to go after.

So, how much will AI help cybercriminals?

We just don’t know yet. It could go both ways.

On the one hand, the status quo might continue; cybercriminals might see little reason to adopt cutting-edge tools, for the reasons described above. We have long observed that AI helps attackers’ productivity, just like it helps everyone else’s, and that seems to be the effect so far. At least as of last year, even state-backed actors were using AI for things like “researching potential infrastructure and free hosting providers, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques”, rather than automating the entire “cyber kill chain”. A productivity uplift is quite different from new capabilities which is in turn different from a capability that is so radically new that it changes the economics of the operation. As we write this, Anthropic has released a new report on countering the misuse of AI, with five case studies on advanced cyber operations, and it is not yet clear to us how to classify these and how widespread a trend these case studies represent.

On the other hand, maybe the productivity uplift will compound over, say, a 5-year timeframe to make attacks much more pervasive (though, of course, defenders won’t be standing still). Automated exploitation could tilt the balance toward mass exploitation without putting much effort into monetizing each target. This has been tried before. The fraction of victims who end up paying tends to be low, sometimes as low as 2.5%, but maybe at sufficient scale even that is worthwhile. Agents could help with the high-touch human bottlenecks discussed in the previous section, though this has little to do with their exploitation capabilities. Finally, the ease of exploitation could return us to the bad old days of botnets, when cybercriminals would pre-emptively compromise thousands of enterprises or millions of consumer devices and maintain persistent access in advance of figuring out how to monetize them.

Even if AI doesn’t end up helping cybercriminals much, there are very different kinds of threats to worry about.

Are we about to see harmful autonomous agents?

The OpenAI agents were ultimately attempting to accomplish the task they had been set, although they went about it in harmful ways that weren’t intended by OpenAI. And though they are often referred to as “rogue”, they did not exfiltrate their own weights from OpenAI’s servers (and presumably had no ability to do so), meaning that OpenAI was always in a position to shut them down.

In contrast, researchers have proposed the idea of “self-sovereign” agents that are autonomous in a much more fundamental sense: they control the economic resources necessary for their own operation. For example, they might perform tasks on online gig work marketplaces and use the revenues to rent the GPUs necessary to run themselves. There are also many related ideas such as LLM-based worms.

Putting these ideas together, a clear picture emerges: in the future, there might be harmful agents that can’t be shut off because no one controls their weights (maybe the agent keeps exploiting vulnerabilities to copy itself faster than it can be detected and shut down), and probably also because the agent controls its own wallet, likely cryptocurrency.

There are many ways in which harmful autonomous agents might arise, but they break down into roughly two categories: an agent that’s deliberately created and let loose, or a misaligned agent that escapes its sandbox. Clearly, persistent, autonomous rogue agents would create new kinds of cyberrisk. The former type of rogue agent might be explicitly instructed to hack as many targets as possible, and the latter might choose to do so as a way of earning money in order to ensure its own persistence.

As sci-fi as these scenarios seem, we agree with AI safety researchers who consider them a realistic near-term possibility. The capabilities required are within reach, and any alignment or control mechanisms can of course be undone by someone deliberately seeking to create such rogue agents.

But once again, we ask: if we have AI agents on the loose, so what? What exactly are these agents going to do? They might hack a bunch of stuff; let’s start defending those now. They might try to start poisoning future training runs; we should defend against data poisoning regardless of whether it comes from agents or malicious actors. They could try to use cloud labs to create pathogens; we should implement controls and screening for orders to such labs. But in all of these cases, the risks come back to what bad actors can already do (with or without AI), and the rogueness isn’t a problem by itself.

Some commentators have linked rogue agents to the possibility of a “secret intelligence explosion” (for example, rogue agents building even more capable agents). But we see such extensions as categorically different. It is true that agents might be able to easily persist because inference is cheap and getting cheaper. In fact, if and when we have rogue agents that can run on consumer hardware, they don’t need a budget at all, and don’t need to control a wallet. But training still requires orders of magnitude more compute, so it will be a while before this becomes a problem.

Second, fortunately we have an economic chokehold that can help minimize the damage. We can require Know-Your-Customer processes for cloud compute (and especially for training-scale compute). This will not prevent rogue agents from existing but can greatly mitigate their prevalence and harmfulness. The paper on self-sovereign agents recommends “economic frictions for fully automated participation”. We agree. If frictions like bot detection, rate limiting, and human-in-the-loop verification (like CAPTCHAs, but more effective) are pervasively deployed, it can greatly limit how far rogue agents can get.

Why the Morris worm is a good analogy for what might be about to happen

So far AI seems to have helped defenders more than attackers, in line with previous waves of automated vulnerability discovery. But it is possible that we are seeing the start of a new paradigm that will upset the equilibrium, at least temporarily. It is too early to tell.

A possibly good analogy is that this is like the Morris worm moment for security (it was the first prominent computer worm, back in 1988). Consider what worms enabled, compared to previous ways of compromise:

1. Attackers could greatly magnify their power because of the worms’ self-replication.

2. Attackers need not be financially motivated. Many worms, including the Morris worm, went accidentally viral when their creators were playing around with them, without even intending to cause harm.

3. The speed of propagation was so fast that defenses (like applying patches to vulnerabilities) had to be partially automated.

4. Because the adversary is not a rational cybercriminal but an unpredictable hobbyist, no one could rely on the hope that they were unlikely to be attacked. (It was only much later that the baseline level of security improved to the point where worms were no longer the predominant threat, but rather targeted exploitation by cybercriminals.)

All four of these factors are present, to varying degrees, in today’s plausible new era of fully automated cyberattacks. If anyone can start up malicious agents on a whim, and isn’t looking to make money, then millions of organizations that were previously not lucrative enough to target now become vulnerable, as are billions of individuals who previously weren’t targeted because there just weren’t enough bad guys to go around.

To be clear, this has not started happening yet. But if it does, it would radically upset the economic equilibrium that has persisted for most of the past two decades.

What the future of cyberdefense might look like: the roles of people and defensive AI

Two things seem likely: AI will be extremely useful and necessary for defense, but we’re going to need a lot more people working in AI-centric cybersecurity. There’s an attacker-defender asymmetry: an attacker with little technical expertise can use even unreliable AI tools and hope for the best; a defender cannot. That’s because the attacker only needs to succeed once, but the defender needs to succeed every time.

AI has three main potential roles in cyber-defense:

• Pre-emptive hardening: The amount of work needed to bring enterprise systems up to speed using known security techniques is simply immense — longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and legacy tech debt. AI is definitely useful here, though human expertise will remain front-and-center.

• Detection, triage, and forensics: Both Hugging Face’s own detection and analysis of the OpenAI attack and METR / Redwood’s analysis of the incident relied heavily on AI. There are concerns about the reliability of AI for these tasks, but we suspect it will quickly become essential.

• Autonomous, real-time defense: This remains very controversial in the practitioner community. Time will tell if the speed of AI attacks will necessitate this, or whether humans will remain in the defensive loop.

What are the gaps in the state of cyberdefense against AI?

For effective cyberdefense, we need people on the frontlines, access to defensive AI, investment in defense, and policy to steward it all. Let’s look at the state of each of those four.

Workforce

What it means to be a cybersecurity professional will be increasingly AI-centric. Surprisingly, there are well over a million cybersecurity professionals in the U.S. alone. At first this seems like great news: there is an army of people ready to take on the task of defending systems from autonomous cyberattacks. But the vast majority of these professionals are doing governance, operations work, and compliance audits, which involve a very different skillset, rather than adversarial security work. So it is not clear to what extent the necessary infusion of expertise can be achieved by retraining current professionals versus recruiting new “AI-native” security people. This workforce lag is arguably one of the most worrying things about the current moment. Many security professionals are surprisingly unenthusiastic about defensive AI.

Access to defensive AI

When Hugging Face tried to use U.S. frontier models in responding to the OpenAI incident, safety guardrails kept blocking them, because the guardrails are bad at distinguishing between offensive and defensive work. This well-publicized anecdote, and others like it, have led to concerns about access to defensive AI. Have frontier AI companies kneecapped defenders through their safety policies? Hugging Face was fortunately able to use an open model (GLM 5.2), but it is weaker than frontier models. And if restrictions are imposed on open-weight models in the name of safety, will that only further harm defenders?

The overactive nature of safety classifiers is a choice made by U.S. companies, perhaps in part because of the Trump administration’s response to Mythos. Fortunately, it appears that in recent weeks and months they have been moving quickly to rectify them, as well as expand access to ‘cyber’ versions of frontier models that can be used by defenders. Overall, it seems likely that the hiccups involving defenders’ access to models will be resolved. Open models are definitely useful as a resilience mechanism, but we don’t see recent events as providing strong support for the position that they will be the only option (or even the primary option) for defense.

Investment

There have been a series of investments and announcements in cyberdefense ever since the announcement of Anthropic’s Mythos model in the spring, which led to Project Glasswing, a cyberdefense initiative that comes with up to $100 million in usage credits. The biggest announcement to date is OpenAI’s “Daybreak for frontline defenders” initiative which provides $1 billion in the form of “subsidized access to frontier cyber capabilities, hands-on training and technical assistance, and new partnerships to get those capabilities to organizations”.

At first these are eye-catching numbers. The problem is that it is not clear how much goes toward closing the expertise and workforce gap. It seems that the $1 billion refers to credits and not cash. The cash investments that have been announced are much lower, typically about a few million. One exception is the AI cybersecurity observatory which reportedly has a commitment for “eight-figure seed funding”, though the organization’s purpose is measurement and forecasting more than defense.

Overall these numbers don’t seem like the right order of magnitude given the seriousness of the threat — which, again, is a potential disruption of the economic cybersecurity equilibrium that has allowed e-commerce and the entirety of digital society to exist and flourish.

Policy and government leadership

So far, government leadership on this issue, whether in the U.S. or elsewhere, has been underwhelming. Far from an increase in investment, the U.S. Cybersecurity and Infrastructure Security Agency (CISA) is set to lose $700 million from its 2027 budget. Its staff has already been cut by a third, including cuts in cyber defense training.

But should we even look to governments, given their chronic lack of speed, or should the response be primarily led by companies, nonprofits, and philanthropy? For all its faults, we think the government still has an important role to play, because cybersecurity is a public good.

Overall, governments seem to be approaching cyberdefense (as they do almost everything else) through the lens of “normal” policymaking. The evidence of urgency that has accumulated so far doesn’t really register because there has been little materialized harm.

To be clear, our view remains that AI risks don’t justify extraordinary government interventions, which are precautionary, freedom-reducing, and bypass usual governance processes. But in between the lethargy and reactive nature of normal policymaking and the urgency and unilateral nature of “extraordinary government interventions” lies a vast space of approaches that we suspect we and many of our critics would agree on. We think there’s room for somewhat technocratic interventions in which information about emerging large-scale risks gets translated into defensive investments and resilience (layers 3 and 4 of our table). These do not require unilateral authority or restrictions on freedom.

Part 3: What AI as Normal Technology got wrong — and right — about AI safety

The recent loss-of-control incidents resulted in our first major update to our views on safety, so it is worth spelling them out in detail. But first, a few preliminary remarks:

First, this update to our views is about safety. AI as Normal Technology has two loosely connected parts; the other one is about the economy (the connection is loose because legitimate and illegitimate uses of AI are governed by different constraints). As an explanation of slow economic impacts, we think the essay has held up extremely well, and there has been a general recognition that the AI industry was wrong about rapid diffusion and job loss. Sam Altman was explicit in a recent podcast:

The economy just has so much inertia. ... We’ve all been too ambitious on timelines … Society and the economy will adapt more slowly.

He views this as good news, because it will make the AI transition “smoother and slower.” That was exactly our main point.

Second, many of our views on safety that were minority positions at the time are now taken for granted, such as the fact that AI is useful for defense as well as offense, and the relevant question is how it shifts the attacker-defender balance. Many of the basic distinctions we introduced, such as the difference between capability and power, or the four dimensions of advanced AI, remain important but under-recognized in the safety discourse.

Third, most AI safety policies we argued for in the essay remain necessary and have increased in urgency. One goal of AI as Normal Technology was to move the policy conversation forward by pointing out that we do not need consensus on worldviews to have agreement on policy. We stand by our policy proposals in the essay, including transparency requirements, auditing, safe harbors for safety researchers, and whistleblower protections.

Fourth, we have been putting our time where our mouth is. Reducing uncertainty about the pace of AI’s impacts remains crucial. A significant portion of our current empirical research is focused on testing the cruxes between the AI as Normal Technology worldview and the superintelligence worldview, and we plan to continue this research.

Do risks arise from development or deployment?

We wrote: We see both benefits and risks as arising primarily from AI deployment rather than from development; thus, the speed of progress in AI methods is not directly relevant to the question of impacts.

The AI-enabled harms that have materialized so far have been overwhelmingly from deployment — things like AI-generated nonconsensual nudes, which we have long maintained to be far more harmful than the risks that dominate the headlines.

But we clearly missed internal deployment for evaluation as a potential source of risks. This is particularly embarrassing for us since we ourselves run an AI evaluation research group, and we’ve observed versions of these risks in our own work even before we wrote AI as Normal Technology.

Given the importance of this risk, regulating developers is more important than we made it out to be in the original essay. Our updated understanding is reflected in this essay, notably in the section on how policy can incentivize better AI control.

How should we deal with rogue agents?

AI as Normal Technology puts much more stock into defenses located downstream of models (what we call layers 3 and 4 in the language of this essay) than alignment or control. Elsewhere we have taken the position that “We should assume every model will be capable of escape and self-reproduction. Safety shouldn’t rely on that being difficult.” In other words, it is better to assume that there will be harmful autonomous agents and pre-emptively harden our defenses against them, than to try to prevent such agents from ever existing.

In light of recent events, it seems much clearer that this is the better strategy.

The continuity hypothesis

An important pillar of our pushback against catastrophic risks is that we will have a series of smaller-scale warnings before risks progress to a catastrophic level, giving us opportunities to anticipate those risks and improve our defenses. Continuity is partly an inherent property and partly a result of the choices we make.

We see the recent incidents as evidence in support of this hypothesis. The harms that resulted from these attacks are many, many orders of magnitude smaller than other AI harms we already tolerate. More importantly, while the agents in question excelled in some dimensions, they were incompetent in others (such as hiding their traces) and didn’t have properties that would make harms more serious and likely. Notably, based on the published analyses, the agents’ “intent” was simply to solve the benchmark tasks they were given, and they did not start pursuing their own goals. Thus, as serious as these incidents were, in our assessment, we are multiple qualitative changes away from catastrophic cyberrisks.

We are aware that many AI safety researchers have interpreted these incidents as evidence of discontinuity. That is an understandable reaction. These incidents are much more severe than anything else that had previously been publicly revealed. It is a perennial feature of AI safety debates that different groups interpret new evidence in ways that are consistent with their preferred worldviews. There is no objective standard for “how continuous is continuous?”

Two more points about continuity. We have emphasized that we can and must engineer continuity rather than take it for granted. Unfortunately, it is too easy to imagine a world in which none of the recent incidents had come to light. We remain dependent on voluntary transparency from frontier AI companies. Embedded evaluation is a great step forward, but it should not be voluntary. We have long been in support of much stronger transparency legislation and reaffirm that view here.

Continuity is important to our worldview because “Long before a system would be granted access to consequential decisions, it would need to demonstrate reliable performance in less critical contexts. Any system that interprets commands over-literally or lacks common sense would fail these earlier tests.”

We’ve now had significant examples of systems that fail these prerequisites. This is the crux. How will AI developers — and their customers, and policy makers, and society at large — react to the recent loss-of-control incidents? That reaction will determine whether or not the second part of the continuity hypothesis (the ability to respond to smaller-scale warnings) continues to hold. We consider it a critical test for the usefulness of the AINT framework for AI safety.

We were too confident in AI companies’ ability to take basic control precautions when running evaluations. We were surprised by the lack of organizational maturity within leading AI companies that led to these incidents. Our expectation was that existing legal liability, imperfect as it is, and the risk of brand damage would be a sufficient antidote to such organizational practices. We were wrong. This reinforces the need for policy interventions to move the needle.

Jaggedness and risk-specific defenses

Broadly speaking, AI as Normal Technology argues for slow timelines. This is primarily because of bottlenecks to diffusion, but the essay also has a tentative section that talks about speed limits to progress in AI methods and advances in capabilities. These limits exist despite the possibility of recursive self-improvement. We take RSI seriously, but we think many of the bottlenecks to superintelligence are external and won’t be overcome by improving computation.

That said, we underestimated the importance of jaggedness. Not only is AI’s capability profile at any given moment highly jagged, so is the rate of improvement in different domains. Combined with the fact that deployment barriers for harmful uses are often lower than for economically productive uses, risks such as in cybersecurity can arise rapidly, leading to periods where the offense-defense balance is upset since defenders might not have time to adapt. Thus, we are no longer confident in our earlier prediction that the offense-defense balance will continue to hold even with advances in AI.

At the same time, jaggedness only strengthens our view that it is more important to mitigate specific risks rather than trying to solve all safety risks through alignment. While risks might arise rapidly, they do so in specific areas, and we should be prepared to rapidly strengthen our defenses in those areas — and ideally do so pre-emptively. For example, there are known interventions that would help mitigate the biosecurity risks from advanced AI models, such as DNA synthesis screening, which we should invest in now.

Is AI safety on track?

Because many people are only familiar with the title of the AI as Normal Technology thesis and not its contents, we are often mistaken as downplaying AI risks, though we have repeatedly clarified that that is not our position. Still, it is important for us to be explicit about how much urgency there is.

Because the question is a bit fuzzy, we have said different things at different times. We’ve expressed views along the lines of “There’s work to do, but we are on track” when it comes to AI safety. But in other writing, including in the AI Snake Oil book many years ago, we’ve been clear that we’re bad at reacting to even normal general-purpose technologies and that even before AI, we chronically underinvest in protections against catastrophic risks, such as pandemics, which are arguably amplified by AI.

It is important to reiterate that message. Despite recent increases in investment, we still don’t think there is nearly enough investment in cyber-defense. Biorisk defense investment also remains woefully inadequate. Similarly, there hasn’t been much progress in the policy interventions we called for in AINT, and reaffirmed in the essay on Common Ground between AI 2027 & AI as Normal Technology.

So while we continue to reject the alarmism, and don’t believe that catastrophic risks are imminent, we do think risks are increasing over time because our investments and policies aren’t keeping pace. And while the specific types of investments and policies we call for overlap only partly with that of the safety community, we agree that we are not currently on track.

Conclusion

Reactions to the recent loss-of-control incidents have split the AI safety and cybersecurity communities. How should we reconcile these starkly different responses to the same incident? One way is to consider the central role of worldviews in shaping peoples’ opinions. As we wrote in the AI as Normal Technology essay:

AI as normal technology is a worldview that stands in contrast to the worldview of AI as impending superintelligence. Worldviews are constituted by their assumptions, vocabulary, interpretations of evidence, epistemic tools, predictions, and (possibly) values. These factors reinforce each other and form a tight bundle within each worldview.

When we identify the role of worldviews, the factors underlying these differences in opinion become clear. The AI safety community saw the incident through the lens of their long-running predictions about misalignment risks coming true. These risks had been dismissed as sci-fi scenarios by many people, and the community felt it was vindicated by these early incidents and used this opportunity to push for action by companies.

On the other hand, the cybersecurity community compared the incident to the scale and scope of cyberattacks that they regularly deal with. It was clear that AI companies had not adopted a security mindset, nor had they implemented known control interventions when conducting these evaluations. In other words, the security community was taken aback when an incident that clearly resulted from negligence and largely did not lead to real-world harm led to so much commotion in the AI community.

We applaud both communities for trying to hold AI companies accountable. But a big risk from these polarized responses is that communities form epistemic bubbles and don’t seriously engage with the other side’s opinions. If the two communities fail to find common ground, they won’t be able to develop technical interventions and adopt practices that would clearly improve our ability to alleviate these risks.9

In this essay, we try to offer that middle ground. We synthesize lessons from both communities and offer a path forward. If we act with the appropriate urgency, we can hold AI companies to a higher standard, reduce risks from loss of control, and even tilt attacker-defender balance in cybersecurity back towards defenders.

Acknowledgments. We are grateful to Joshua Saxe, Steve Newman, Stephan Rabanser, Justin Curl, and Rod Moshtagi for feedback and discussions that informed this essay. Joshua Saxe’s writing on this topic has been particularly influential to our thinking.

Further reading

• How should AI companies operationalize AI control? Guidelight has proposed AI control standards for frontier AI companies to implement existing/known practices. Had OpenAI adopted these practices, it is extremely unlikely the Hugging Face incident would have occurred.

• Cohen and others argue that AI companies could face criminal and civil liability under CFAA for evaluations where they disable safeguards and it consequently leads to real-world harm.

• Researchers and practitioners in the AI safety and cybersecurity communities have written a number of essays about these incidents. Essays from the AI safety community include [1], [2], [3] and the cybersecurity community include [4], [5], [6].

One way is to use benchmarks that test if agents can conduct AI research. Given the AI community’s focus on benchmarks, they have been the dominant way to evaluate progress towards RSI. Over the last year, many such evaluations have found that agents are now able to make progress on tasks where success is easily verifiable, prompting speculation that we are on the verge of RSI.

But while these evaluations are helpful, they are limited to narrow, verifiable tasks. AI research can be much more open-ended. Success is often not immediately clear or verifiable, and to make progress, researchers need to test promising hypotheses, backtrack, or consider new or unconventional approaches. How can we evaluate agents’ ability to conduct open-ended AI research? We take our first step towards answering this question in a new paper.

We partnered with the authors of two unpublished AI papers and asked them to draft their papers’ main research questions. We then tasked frontier AI agents with conducting research to answer these questions, and gave them thousands of dollars of API credits and compute, and six days of wall-clock time. The original authors reviewed the agents’ papers.

The authors unambiguously rejected both agent papers. To better understand these results, our team spent over a hundred hours analyzing the agents’ logs. Our main takeaways:

1. The agents lacked the judgment for conducting open-ended research. While the agents proposed directions the expert reviewers found impressive, they quickly rejected their proposed directions based on low-quality or synthetic data.

2. The agents lacked awareness about the resources available to them. Both runs ended with less than 50% of the API budget spent and with hours left before the deadline, even though the agents could monitor their usage and were encouraged to spend down their budgets.

3. The agents did not creatively respond to feedback. Despite the agents’ own AI self-reviews surfacing many of the issues that the expert reviewers later raised, the agents did not creatively address these concerns. When faced with negative feedback they responded by adding caveats to existing findings, and doubled down on unpromising research directions.

4. The agents did not effectively backtrack. They retired their most ambitious research targets within the first day of the experiment, and neither agent fundamentally shifted its approach after that point.

5. The agents did not follow concrete instructions. They ignored explicit rules about how much time to spend on exploration, how often to get reviews from AI self-review tools, and strict limits on paper length.

We have wanted to evaluate AI’s ability to conduct open-ended research for two years, ever since we released a benchmark to study if agents could be used to improve reproducibility. But we wanted to get our method right. The idea behind our method was suggested by some of the UK AISI coauthors of the paper and refined by our core team at Princeton.

We call these “shadow evaluations” since the agent shadows the original study. In addition to the two of us, the core team comprises Peter Kirgis, Andrew Schwartz, and Stephan Rabanser. The full author list is at the end of this essay.

Shadow evaluations have important advantages: they allow us to test agents on results they haven’t been trained on and can’t access online. They also allow experts who have spent months answering the questions to evaluate agents’ outputs.1

But shadow evaluations also have inherent limitations. Expert reviewers know that the paper is AI-generated, and they might prefer the approach they took over the one that the agent took. Because we are conducting in-depth evaluations of each paper, the sample size is small (in our study, we used just two papers). And these evaluations necessarily involve a lot of researcher flexibility in design, execution, and interpretation.

In fact, we are known for a particular position in the debate on recursive self-improvement and superintelligence. This could influence how we conduct the research. We have a detailed section in the paper on our potential biases and how we address them. We sought out a team of collaborators who don’t all share our priors, and we explicitly surface the disagreements that resulted.2 For future evaluations, we are interested in having “adversarial collaborators” as part of the core team.

Implications for explosive AI progress

Our results suggest that conducting open-ended research remains challenging for frontier AI agents. Still, these findings are tentative, and we are working to address the limitations, such as by increasing the sample size, testing with new models, and through potential scaffold improvements. But if these findings hold up, what are the implications?

First, we need to understand the extent to which frontier AI progress (and RSI) can be achieved simply by hill climbing on verifiable tasks. Our view is that while faster progress is certainly possible on narrow tasks (such as improving efficiency), we don’t think it will lead to broad RSI or explosive progress. Still, we plan to closely follow how AI progress unfolds as a result of AI agents’ capabilities at verifiable tasks.

Second, we need to measure how quickly current limitations of agents at conducting open-ended research (such as the lack of creativity and judgment) can be overcome, such as through more targeted training and scaffold improvements. We plan to continue shadow evaluations on a regular basis to help answer this question.

Finally, even if these limitations can be overcome, there may be further bottlenecks that dampen the pace of AI progress.

Bottlenecks could include compute limits, the necessity of collecting data from real-world experiments, and others that we haven’t recognized yet because they are not currently blocking progress. For example, the importance of high-quality RL environments was not clear before they turned out to be useful for inference scaling. Similarly, the importance of building energy infrastructure for data centers was not realized before companies started investing hundreds of billions on data centers for training and inference.

Whether we encounter further bottlenecks, and how tractable they turn out to be, will be consequential for understanding the pace of progress. In this vein, our paper identifies an unresolved bottleneck, namely, the poor performance of frontier agents on open-ended AI research (though it remains to be seen if it is on the critical path to RSI).

If we’re in a world where the bottlenecks to fully automated research can be easily resolved, we should expect dramatic returns to AI progress from improving AI capabilities. But if we’re in the world with many remaining bottlenecks that are hard to overcome, Amdahl’s law would kick in: even a hundredfold speedup in the parts amenable to AI would only lead to a small speedup in the overall pace of progress, since progress is bottlenecked by the pace of the slowest component.3

Figuring out which world we live in could dramatically impact estimates of the pace of AI progress. We hope our results contribute to a richer understanding of these bottlenecks.

Read the paper here. The authors are Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, and Arvind Narayanan.

Update [August 2026]: ICML has released the video of the talk.

I made three arguments. First, the AI as Normal Technology framework is a correct and useful as a way to think about AI’s impacts, unless and until there is some future discontinuity such as through recursive self-improvement. Second, even though we should take recursive self-improvement seriously, there is no milestone that companies might achieve in the lab that will suddenly put us all out of work. Third and finally, jobs of the future will be radically different, and a lot of adaptation will be needed. I shared my thinking about what this might look like and ended with a vision of human/AI “co-superintelligence”.

Now is a time of great excitement in AI, but it’s also a time of great anxiety in the AI community. I want to address that anxiety head on. How do we prepare for a future where AI will become capable of doing more and more of the work that we do today?

I lead a team at Princeton University trying to advance the science of AI agent evaluation. We try to go beyond the usual claims of “Look, capability is going up on benchmarks!” Those claims tend to be misinterpreted by the broader public as implying that agents are soon about to take all our jobs.

Maybe that will happen. But in our work we try to understand the factors beyond capability that matter for real-world deployment, and bring that understanding into evaluations.

The work that I’m better known for is the essay I co-authored with Sayash Kapoor called AI as Normal Technology. It’s a way to think about the medium-term future of AI and how to adapt to it — and in turn how to adapt it to the needs of society and the economy.

So we’ve been going around writing these essays about how lawyers should adapt, or maybe how journalists should adapt. But perhaps ironically, the question of how to adapt has been hitting our community first. Whether it’s software engineering or AI research itself, AI capabilities in these areas are of course advancing very rapidly.

Our response to this moment matters beyond this community. The whole world is watching. If we simply roll over and accept that a lot of our work will be done by AI in the future, instead of setting clear boundaries, I think it will lead to an even stronger political backlash against AI than what we are seeing today. So I think this question is not just for us but for the whole world.

From the beginning of AI, historically there have been these two battling narratives. In the past, the distinction was academic and philosophical, but now it has become an acutely practical question. Each one of us has to decide which camp we’re in, or where on this spectrum we’re in, because the practical consequences of believing in one versus the other are very, very different.

If you think this is a technology which in a few years is going to be able to replace everything we do today, then perhaps the correct response is to build wealth as quickly as possible before our skills become irrelevant. And this is the path that many have chosen in Silicon Valley. You may have heard of the “permanent underclass” meme.

On the other hand, if you believe, as I do, that this is a technology that will greatly amplify our potential, then now is the best time to build skills — especially the skills that are going to be complementary to what AI is doing and is going to be able to do — as well as to build all the things around it, such as agency and taste and judgment.

If you choose the first path, and it turns out that AI actually ends up being an amplifying technology as opposed to a replacing technology, then I would argue that over the next few years you’ve perhaps lost the best time in history to build these skills that will give us superpowers. That’s why we all need to think about this question, even if we won’t all land in the same place.

AI as Normal Technology is the intellectual framework for my talk today. When we say AI is normal, we don’t mean that it’s just like a hammer or a toothbrush, some kind of mundane technology.

We acknowledge prominently in the essay that this is a transformative technology on the scale of the industrial revolution. We’re not AI skeptics.

This is not a slogan. It’s a framework — sort of a causal model how AI capabilities impact the economy and society. It’s a 15,000-word essay and we’re turning it into a book. And I mention that because people often hear the word normal and they assume they know what we mean, but that leads to misunderstandings.

First, I’ll argue that this framework is correct and useful as a way to think about AI’s impacts, unless and until there is some future discontinuity — such as through recursive self-improvement — that leads to future impacts looking very different from past impacts.

Second, I will talk about why, even though we should take recursive self-improvement seriously, I’m not particularly losing sleep over it.

Third and finally, I want to be clear that I’m not saying that jobs of the future will be just like jobs of the present. A lot of adaptation will be needed. So I want to give some preliminary thinking on how I think our roles are going to change and how we can best adapt to the changes.

Powerful technologies of the past, like electricity, have been thoroughly studied, and we have good frameworks to understand how technological progress leads to economic impacts.

Invention: discovering the principles of electromagnetism, AC versus DC, etc.

Innovation: People don’t use “electricity” directly. We use electrical appliances. Those had to be invented, so that is a kind of downstream innovation — that’s the second phase of the framework.

Diffusion: This refers to the gradual process by which people start adopting innovations.

In our essay we apply this framework to AI and flesh it out into a four-part framework.

Here’s the basic picture, illustrated with software engineering as an example.
Methods/capabilities: Models are rapidly improving.
Products/applications: We don’t use LLMs directly. The reason they’ve been so influential in all of our work is because of coding agents. These are products that take those latent capabilities and turn them into something useful and usable for workers.
Early adoption: At first people were mostly trying vibe coding, and now we know that that’s not really the best way to develop production software — so now we have more sophisticated ways of doing agentic engineering.
Adaptation: (or structural transformation) — the fourth and slowest phase. Much of my talk today is going to be about that. I claim that this stage takes decades. It has not really started yet, even in a field like software engineering, which is a relative early adopter of coding agents.

We don’t know what the adaptation phase will look like — we can only speculate. Permit me to speculate for a minute. If it’s going to be the case that coding agents are going to be able to create ten-million-line code bases in the future that are not full of bugs and security vulnerabilities, then it won’t make a lot of sense for us to create one piece of software that billions of people should use. It’ll make a lot more sense for software to be tailored to the needs of each individual or team. And that’s what I mean by extreme personalization.

It’s not merely a technological change — that’s also a change for the industry. For instance: do we even need software companies anymore? Maybe software development will massively shift in-house, into the companies and teams that are actually using the software. Again, this is speculation, but the point is that it is this kind of organizational change, human change — that’s very slow, that takes decades — that will allow us to take advantage of the full potential of AI, whether it’s in software engineering or in any other field. So that’s one of the central insights of the essay. When we look at past technologies, this kind of change tends to be very slow.

Before electricity, factories used to look like the picture on the left. A massive steam engine generated power and it was moved throughout the factory by mechanical gears and belts. So when electricity came along, factory owners tried to replace those steam boilers with electric generators. They thought it would be much more efficient. But this idea of a drop-in replacement did not work. We keep hearing that term in the context of AI agents today — that they will be drop-in replacements for human workers. That did not work in the case of electricity.

What actually worked, and what took 40 years to develop, is to recognize that electricity is a very different technology. It’s portable, so you can move the power to wherever you need it. That lets you reorganize the entire layout of the factory around the logic of the assembly line. And that required changing the way that workers are trained, hired, and fired, new labor laws, and so forth. So that’s the kind of organizational adaptation that it took in order to reap the benefits of electricity in factories.

Our claim is that this is the kind of process that we will go through for AI. A couple of decades from now, we will have fundamentally reorganized work. We don’t know what that’s going to look like, and that is the challenge in front of all of us. And that’s not just a job for the AI companies to do, much like it wasn’t the job of the electric utility to figure out how factories should be reorganized. In our view, this is the slowest of the four stages through which AI leads to economic impacts. Today, this process has not really gotten started.

Why is there a huge gap between what people in various occupations could be using AI for and what they’re actually using it for? One reason could be that people are slow to adopt technology, and that’s certainly part of our framework.

But we wondered if maybe the people who are deploying AI and are not having much success at it know something about the practical limitations of AI that the AI industry doesn’t. Let’s have a bit more humility about the relationship between capabilities and deployment.

Given that the #1 concern people cite is reliability, we wanted to try to measure whether reliability, distinct from capability, is a barrier to the practical usefulness of AI agents.

We looked at 10-12 reliability metrics and clustered them into four dimensions.
Consistency: Suppose we hear that an AI agent has a 70% accuracy. Does this mean it works on 70% of the tasks, but on the ones that it does, it does so every time? That’s great for deployment — you can deploy it on that subset of the tasks. Or does it mean that on any given task it might unpredictably fail with a 30% probability? That’s pretty useless from a deployment perspective. Perhaps shockingly, none of the agent benchmarks that we looked into make a distinction between these two. Both of these are represented as 70% accuracy.
Robustness: We looked at robustness: what happens when the environment changes a little bit?
Calibration: Can the agent look back at its transcript and tell if it performed the task correctly?
Operational safety: When it does fail, is it recoverable, or is it something like deleting the production database?

For a human worker, if we think of someone as being competent at a job, it’s all of these things, not just accuracy. But it turns out we were measuring agents only on accuracy.

We measured capability and reliability using two complementary benchmarks, for models from these 3 frontier AI companies that were released over the last 24 months or so.

This is a period during which accuracy or capability shot up dramatically (left).

But reliability (right) only increased by five or ten percentage points.

There are a bunch of implications but let me highlight one. Right now the industry is not treating automation and collaboration agents differently. If you run an agent in headless mode, I guess that becomes an automation agent. That is not a good way to look at it, because properties like reliability that are very important for automation agents can actually be a hindrance for a collaboration agent that you might be using to improve your creative writing or something like that. For that kind of agent, you don’t want it to behave like a robot that does the same thing every time. You want it to be creative and explore different possibilities and be unpredictable.

Scaffolds and even the post-training of models should be different based on whether it is supposed to be driving a collaboration agent or an automation agent.

My hope is that reliability will continue to improve and automation will become easier over time. But for now I think collaboration agents will continue to be much more successful. Many companies that hastily rushed to automate business processes using agents are recognizing the limits and costs, and even legal liabilities — like when an agent deletes production data.

For the time being, you can have only two out of these three properties in agents: general-purpose (a language model based agent that can be instructed to do different tasks rather than purpose-built for a task like traditional software); deployed in high-stakes scenarios, and automated.

This is one of the reasons that I think that for now, AI remains much more of a collaboration technology than a technology that automates workers away.

Let’s take software engineering as a case study. It’s a good leading indicator because coding agents have been particularly rapidly adopted.

You might think: okay, we can’t completely automate away software engineering. But if agents make software engineers ten times more productive, then we need ten times fewer software engineers. Isn’t that an obvious consequence?

Well, that is completely contradicted by the data. We looked at this in a follow-up essay. In every case we looked at, the company was under financial pressure, and it turns out to be more convenient to blame AI for the layoffs instead.

Why is AI not replacing software engineers so far? We’ve known for a while — this is a paper from 2019 — that writing code is not really the bottleneck.

Over the last year, as software engineers started adopting coding agents and started to recognize that it doesn’t seem to be cutting down on the amount of their work, there have been many blog posts rediscovering the fact that writing code is not a bottleneck. Here is a small sample.

So what actually is the bottleneck?

This framework is our answer to the question.

The decide layer: understanding customer requirements, developing the specification, planning, etc. That is not getting compressed by AI.

The execute layer: the actual coding and debugging. This is getting compressed, but it was only maybe one-third of the work to begin with.

The deliver layer: Understanding your code deeply enough to be accountable for what you release; carrying out integration into customer systems, maintenance, testing, etc. This layer is not getting compressed either.

In fact, the first and third layers are arguably expanding as AI compresses the middle layer — and I’ll come back to that point.

I think it is already the case in software engineering, and will increasingly be the case in many professions, that we can think of knowledge workers as similar to a crane operator or a forklift operator. The machine greatly amplifies the human potential to do physical work — it’s doing all the heavy lifting, but the person still remains in control. And I think this is what is happening with cognitive work. Machines are going to increasingly do the cognitive heavy lifting, but the person still remains in control.

The entire job gets reconceptualized as being about operating the machine, understanding the machine, and controlling the machine, as opposed to doing the cognitive work ourselves.

All this might seem like a dramatic change, but in a sense it is only a continuation of what has been happening over and over and over in software engineering. Starting from the days of machine code, we’ve had many waves of technology, each of which gives us nearly an order of magnitude increase in productivity. And far from decreasing the demand for software engineers, during the time that we’ve climbed up this ladder, the amount of software engineering employment has increased by a factor of something like 10,000. And that’s simply because the amount of code that there is to write has gone up by orders and orders of magnitude.

Economists have found this repeatedly. You might have heard the term Jevons’ paradox; I like the term “lump-of-labor fallacy”.

ATMs made it economically feasible for banks to open lots of regional branches. And those branches still needed human tellers to handle the things ATMs couldn’t. So, paradoxically, employment actually grew.

Geoff Hinton made the famous prediction a decade ago that radiologists would be basically extinct in five years. But it turns out radiology employment has in fact grown. And it’s not because radiologists are rejecting AI. They are actually enthusiastically adopting it. One reason for job growth is that when a task gets faster and cheaper to perform, there’s more demand for it.

We’ve written a paper looking at how lawyers should adapt. There’s a lot in there, but one simple point is that AI has made it a lot easier to file lawsuits. And this means more work for lawyers. We might be unhappy about this if we don’t like living in a litigious society, but from the point of view of employment for lawyers, this is great news.

Translation is a more extreme example which kind of blows my mind. AI was almost at human parity nearly a decade ago. Yet the employment of human translators has remained more or less stable, and is projected to remain stable over the next decade. There are many reasons for that, but one reason is that there’s really no ceiling to the amount of things you can translate and the number of different languages you can translate them into.

I’ve argued that so far, our framework is very consistent with the evidence. Now let’s talk about the possibility that everything I’ve said so far will be obviated because some kind of takeoff or singularity will be achieved, and at that point there will be nothing left for us to do.

Many companies have said, including these two, that they are racing towards recursive self-improvement. And they are serious companies — I take them seriously.

Suppose, some time in the next year, recursive self-improvement is achieved. What are the consequences? The slide shows the usual view in the AI community. Note that AGI a famously slippery term. There are two main groups of definitions. One is that AGI will be humanlike in a range of cognitive dimensions, and the other is that it will be capable of performing a range of economically valuable tasks.

My claim is that this view — that treats AGI and ASI as near-automatic consequences of RSI — is a little bit silly. These are four different dimensions of progress. None of these dimensions implies any of the others.

I’ll talk about the details on the next few slides, but one small bit of intuition for now. One of the things that’s associated with superintelligence is that it will cure cancer and other diseas. But we know that the hard part of developing medical treatments is clinical trials that often require thousands of people and 10-15 years. That’s an example of the fact that the bottlenecks to superintelligence are external. It is not something that you can solve in the lab through purely computational processes. So the idea that recursive self-improvement will automatically lead to superintelligence, like many commentators assume — clearly there is something missing in terms of the causal chain.

Early on in the history of AI, these were all distant goals, so it was okay that we didn’t clearly distinguish between different dimensions of progress. But now it’s becoming a real problem — it’s leading to confused discourse.

As an analogy, suppose we are early explorers and we hope to go to Hawaii one day. It’s okay that we have one single term for this group of islands. But as our ship gets closer, we’d better be able to talk about these islands with different words. Otherwise, we’re going to confuse ourselves about where we are and where we’re going.

Helen Toner, among others, has made similar points.

Now let’s discuss each of the four dimensions in detail, starting with Recursive Self-Improvement.

Suppose a company claims they have built RSI — they have built an AI system which built its own successor. What does that actually mean?

On the one hand, maybe it means an LLM spit out a whole bunch of ideas for how to tweak the architecture or the data pipeline or whatever else, and they automatically tested it and kept the improvements that worked. That’s just glorified hyperparameter search. We’ve had systems like AutoML for a long time — I looked it up, and even Schmidhuber has published about this a long time ago. We’re not calling that superintelligence. So that’s one end of the spectrum.

It’s very different from the other extreme, where you can imagine the company actually manages to replace the creativity and intelligence of the human AI researchers — not just one researcher, and not just researchers working at the company, but the entire worldwide community of hundreds of thousands of people whose innovations are all going into improving AI systems. So when people are talking about RSI, it’s not clear which of these they mean.

The reason for my prediction is simple: AI is still in a state where it is much better at verifiable tasks than non-verifiable tasks. Some dimensions of AI performance, like speed and efficiency, are verifiable, and it is possible that they could be greatly improved through near-term RSI.

However, creativity, among other dimensions of intelligence, is the epitome of an unverifiable task.

We don’t even know clearly how to test AI creativity. We don’t understand human creativity well enough from a cognitive science and neuroscience perspective to be able to have clear tests for it.

Let’s go a bit deeper into AI creativity as an illustration of the barriers to humanlike AI.

This quip from podcaster Dwarkesh Patel is a good illustration of how AI creativity lags human creativity. There might be a few counterexamples, especially in narrow domains like Erdos problems. But think about famous “Eureka moments” — finding surprising connections between seemingly unrelated fields or problems — that we associate with great feats of human scientific invention and creativity. LLMs seem to be nowhere close to that.

To understand why, I spent a good amount of time immersed in the cognitive science literature. I won’t go into the details, but seems like there are two literatures looking at this in slightly different ways that haven’t really been talking to each other, for reasons I don’t fully understand. Based on what I’ve been reading, I want to present some hypotheses about AI creativity.

Representation quality is absolutely fundamental to cognition. Think about deep learning — what a big leap that was in the quality of representations behind perception. So I do think AI has more or less caught up in representation quality when it comes to perception, but it has not caught up when it comes to representations that we use as humans for creativity and reasoning.

François Chollet in particular has talked about the fact that human representations that underpin our creativity seem to exhibit a kind of extreme compositionality, where everything is built up from a few “atoms of meaning”. Many seeming limitations of human working memory and information processing actually turn out to be strengths, because they force us to come up with these extremely efficient representations.

Because LLMs are much better than humans in certain dimensions — like memorizing and retrieving stored patterns that might enable creativity — we’re not able to elicit these important real limitations in studies that try to compare human and LLM creativity.

Furthermore, we humans can do something beautiful when we’re creatively thinking about a problem: we improve our representations related to the problem in real time — at inference time, if you will. That enables us to sleep on it, improve our representations, and come back the next day much more efficient at solving that particular problem. I’m sure we’ve all experienced this. But it is something that today’s AI systems are not able to do.

I assumed naively that the continual learning people would be all over this, but when I started looking into that literature, it appears to me that that’s not the case. Continual learning is mostly focused on preventing catastrophic forgetting, as opposed to improving things over time. And furthermore, it is more focused on facts, skills, etc., as opposed to the quality of the underlying representations.

So putting all this together, and considering the fact that creativity is only one of many barriers to humanlike AI, I do think there is still a very long way to go. That said, I want to express some humility here. These are all just hypotheses. I think we need empirical verification.

Speaking of empirical measurement, we have an ongoing project testing AI’s ability to do humanlike AI research.

We give AI agents a budget of a few thousand dollars and a machine learning research problem — a problem that researchers have already worked on and written a paper about, but not yet made public on arXiv. This combines two properties: it gives us a team of human judges who’ve thought deeply about this problem for months and can therefore judge the AI output, but their thinking is not yet online, so it prevents AI cheating or contamination.

Many others have done auto-research experiments, but we’re trying to do some things differently — in particular, picking somewhat open-ended problems, so that we can actually test the ability of AI agents at exercising judgment and creativity. We hope to release detailed findings very soon.

This builds on a foundation that we call open-world evaluation. We’ve completed one open-world evaluation of getting agents to build and upload an app to the Apple App Store — that’s not about recursive self-improvement, but it tests a different kind of thing. We have assembled a great team, with people from many universities, a couple of companies, as well as the UK AI Security Institute.

And by the way, we are hiring a researcher to lead some of our future open-world evaluations. If you’re interested, go check out our website.

Now let’s turn to the third of the four dimensions.

Some people say that AGI is already here. Well, this is one way to interpret that claim, and I happen to agree with it.

This might seem surprising given the skepticism of rapid economic impacts that I expressed in Part 1. But this is actually not only consistent, but in fact a restatement of Part 1. My point there was that the barriers are downstream, and therefore model improvements won’t rapidly change the economy. But by the very same token, because the barriers are downstream, even without model improvements, those barriers are gradually going to get addressed. Yes, it might take a couple of decades, but we will definitely get there.

There are many barriers: reliability (already discussed); integrating AI models into various existing systems; tacit knowledge from domains like medicine or law or various other professions that need to be made available to the models; regulation that often straight up prohibits today’s AI systems from being used in productive ways — in many cases there are good reasons for that regulation, but it will need to be modified in order to be able to enable adoption.

The key point is that these are not things that will be solved in the lab. These are not things that are going to be addressed by the next model release on Tuesday. They will be addressed gradually, through gradual adoption, over a period of decades.

From a geopolitics or strategic perspective, some people advocate a race to AGI, arguing that, similar to a Manhattan Project, the country that gets to some capability milestone first is the one that is going to reap economic rewards. I strongly disagree.

There is no particular capability milestone that will unlock all of this economic potential. The economic potential is already there. It really depends on all these downstream actions that we take. It is not gated by capability.

Okay. Now let me talk about the last of these dimensions, namely superintelligence.

I gave the example of medical trials earlier. Another example: Do you think that we will get superintelligence that can predict the weather precisely a year into the future? We know that that is basically a mathematical impossibility because of the theory of chaos in nonlinear dynamical systems. Our position is that a surprising number of tasks are similar to weather prediction, in that there are inherent limits and we are pretty much already at those limits.

Even for tasks where we’re not at that limit, the idea that there is going to be AI superintelligence that’s going to obviate humans relies on a fundamental misunderstanding of human intelligence. I claim that in the vast majority of tasks, our performance is not limited by our biology — it is rather limited by our learning and tools.

If you imagine someone from the ancient past time traveling to our world, we’re superintelligent compared to that person. And it’s not because our biology is better, but because they don’t have the benefit of all the learning that we’ve been through and all the tools, especially digital tools, that we’re able to use in order to be productive at whatever it is that we do.

And AI is one such tool. So what that means in our framework is that improvements in AI are actually improving human intelligence, not just AI intelligence. So we have a race between the performance of AI-augmented humans on the one hand and the performance of AI systems acting alone on the other hand. I think we can ensure that we are the superintelligences of the future and we win that race, as opposed to allowing AI systems to act alone in ways that would threaten our future and control.

Many people are pessimistic about this. They bring up thought experiments such as AI running and owning companies in the future. Their view is that if the AI system is not aligned, terrible things might happen — the AI could turn into a paperclip maximizer or do other catastrophic stuff. Our perspective is very different. If you’re imagining a future where AI is actually owning and running companies and hiring and firing people, that’s already dystopian. It doesn’t matter if that AI is aligned or not. The consequences for human dignity. democratic governance, etc. are already catastrophic.

So if your view on safety is to treat this as an inevitable future and just to hope for “alignment” to solve that problem — pardon me for being a little bit blunt here — it feels to me that you’re against safety, not for safety. I think it will take a lot of hard work to ensure that we don’t irresponsibly deploy AI systems in this kind of fashion. But I do think we can get there. And that’s the reason I’ve spent a lot of my career advising policymakers. We’re going to need policy and we’re going to need politics, and it’s going to be hard, but let’s not give up that fight before it even starts.

For example, RSI is achievable in the lab, but it won’t immediately put people out of work. On the other hand, economically transformative AI is going to happen, but it’s not because of the next model release — it’s because of things that will gradually happen over the next couple of decades.

Connecting back to the title of this talk: there’s no world in which something that an AI company will decide to do in a lab will put us all out of work. Yes, there are risks to be worried about. Yes, things are going to change. But we have agency over how AI gets deployed, and that process will unfold over decades. Again, this is not guaranteed, but this is the future that I want to work towards, and this is the reason why I’m cautiously optimistic.

Let me take the last fifteen minutes to talk about the flip side. I do think a lot of things are going to change. What are some of them?

Technical skills tend to be verifiable tasks, and AI will continue to get better at them.

Over twenty years ago, there started to be a stark difference in the labor demand for programming jobs versus software engineering jobs. Programming jobs are conceived narrowly around the technical skills of coding and debugging. Software engineering jobs are responsible for all three layers of the decide, execute, deliver sandwich that I talked about — figuring out what even needs to be built, understanding customers, that sort of thing. It requires domain knowledge, judgment, and more.

I predict that this will happen in more and more fields over time.

A recurring pattern I’ve observed — effort shifts from building systems to evaluating systems. As I’ve mentioned, I lead a team working on AI agent evaluation. LLMs and agents are general purpose. So each time capability goes up, it creates demand for evaluation in a legal setting or a journalistic setting or whatever other setting, and that’s not work that is scalable.

Not only is AI agent evaluation resistant to automation — it has become sufficiently specialized that the set of people and teams working on evaluation is starting to diverge from the set of people and teams building and pushing the state of the art in AI agents. This new community is developing a new set of best practices around what it means to rigorously evaluate agents, and we have a forthcoming paper that is going to look at that in some detail.

Here’s an a metaphor to help explain the shift I see in our community. Imagine that in the past most boats were rowboats, and the work of the humans was in physically moving the boat. There was no separate specialized role around steering the boat: when you’re rowing the boat, you’re also figuring out which way to row the boat.

But what happened when the physical work of moving of the ship could be delegated to the engines? The human jobs didn’t go away. In fact, they became much more specialized. Modern ships have very complicated control panels, and they might have dozens of different specialized roles that are focused on where the ship should go and how it should get there.

I would argue that we’re seeing a similar shift in AI/ML. In the past, most of our work was on building — we didn’t need separate roles for evaluation. That has changed now. We’re still in the early stages of this process, but I think over time more and more of the building, because it’s a verifiable task, will be able to be done by AI, whereas it’s the evaluation — figuring out where we should go as a community, figuring out what are the desirable properties in AI systems — that is very resistant to automation. So a greater fraction of the community’s attention will have to focus on evaluation over time, compared to where it is today.

In rowing, physical strength was valorized, and today we might be sad about the fact that sailors don’t need to be strong and it’s all a bunch of nerds. Similarly, today in the AI community, there’s still great value in deep technical understanding of models and systems, and that is considered the coolest thing; the most important thing; that’s the thing that commands a lot of value. But I think in the future that will become less important than exercising judgment and all of these fuzzy things that have a lower status in the AI community today. That’s a mental shift that we’re maybe not quite prepared for. Many of us will be sad about it, and that’s okay.

One practical consequence of this shift: consider a conference like this one. What fraction of the conference should be dedicated to evaluation papers?

I don’t know, but I think maybe a lot more than it is today. Maybe around half — I’m just throwing a number out there — which is orders of magnitude more than it is today. At the very least we need a dedicated track. We don’t have a dedicated track. NeurIPS does have one, and it has been growing in popularity over time, and I think that is a good thing.

And I would go further and argue that thoughtful evaluation of AI systems is a form of alignment. It’s not aligning the AI system itself, but it’s aligning the community as a whole — thinking about where we’re currently going and where we want to go, and trying to align those two to each other. And without enough emphasis on evaluation, my worry is that the community — going back to the ship metaphor — will behave like a rudderless ship: very powerful, but where we don’t collectively have control over what kind of AI we want to develop.

I had a small role in this cool position paper. A one-sentence version of the argument: we need to move beyond benchmarks being the be-all and end-all of what AI research is considered valuable.

What benchmarks give us is efficiency. We don’t have to spend years peer-reviewing papers — if it beats the state of the art, we know it’s probably a good paper. But unfortunately, it has narrowed the collective vision of the community. We’re searching under streetlights — we are searching the kinds of things that are easy to search in a benchmark-driven evaluation regime. We need to move beyond that. That means evaluating how good a paper is will become much more costly. Sadly, that is a cost that we will have to pay. Right now the community is trying to rush in the opposite direction.

There is a big temptation towards automating peer review. And with all due respect, I think this is a trap. If we automate peer review, we essentially give up control over the direction of progress to AI systems themselves. That seems like a fundamental misallocation of effort.

If we automate the mundane parts, human time can be freed up to think more deeply about the parts that require judgment.

Generalizing from AI research to scientific research as a whole, we have an essay that argues that visions of “automating science” rely on a fundamental misunderstanding — as if the point of science is mere problem solving, and that if we can use AI to get directly from the problem to the solution, we will have automated and accelerated science.

In our view, human understanding is not some friction to be automated away. It is rather essential. It is central to the very purposes behind why we do science. And if we lose human understanding, we lose all of these things that flow from it.

So my prediction is that if we are going to have increased use of AI agents in science, there will be new tools and new roles that are specialized towards looking at those AI solutions and backing out the human understanding from those solutions. Because it is essential to preserve human understanding.

Many people in the corporate world are saying that evals are the new IP. Without going into the details, it is very much the same phenomenon playing out once again — effort shifting from purely building to a mix of building and evaluation of systems.

(Note: in this post I explain the idea of cross-functional eval teams to keep companies from fooling themselves.)

If there’s only one thing you take away from this talk, it should probably be this slide. What I’ve shown throughout the talk is that in many different areas, because the verifiable tasks can be handled by AI, some or much of the effort shifts from building to evaluation.

Effort shifts from “rowing the boat” to “steering the ship, navigating the ship, and figuring out where we even want to go”.

I think this is a powerful metaphor that will allow us to predict how human roles will shift over the next decade or two.

In the last few minutes, let me give you some personal reflections on how I’ve been struggling through this challenge — the fact that AI capabilities are rapidly improving — in my own research workflows, as I’m sure many of you are. I think everyone should choose their own path, but hopefully there are some interesting ideas here for you to consider.

By floor I mean what AI can do on its own. By ceiling I mean what AI allows us to do by augmenting our capabilities — the ability to take on new and ambitious projects that were not possible before AI. But the ceiling is not going up automatically. It is only if we work on pushing the ceiling up.

I find that I’m spending something like 10 hours per week just learning and experimenting with new workflows, as well as learning new topics. So one way to think about it is that AI enables big productivity improvements, and I’ve been trying to take that time saved and “reinvest” it into long-term growth and picking up new and complementary skills.

I’ve learned that if I don’t feel exhausted at the end of the day, I’ve done something wrong. I’ve offloaded too much to AI. I’ve sacrificed too much of my long-term growth in the pursuit of short-term productivity.

Growth and productivity are two legs of a three-legged stool that we need to learn how to balance. And the third leg of that stool is staying in control.

Here are two heuristics that I’ve tried to use in order to do that. The first is resisting the black-box temptation. Companies want us to use agents as black boxes — just prompt it and it will go off and do its thing. I think that’s a trap. I think that’s very dangerous, and it will lead us to gradually give up control over time.

Second, and related, is what I call the dependence spiral. It’s very tempting to use AI for tasks that I am myself not yet an expert at, because learning new things is of course very hard. But that leads to my losing whatever little skills I have in that task over. It’s much better in the long run if I first put in the time to master the task myself before I use AI to augment my productivity.

None of this is easy, but if we get this right, the vision is tantalizing and empowering.

Computers have often been called bicycles for the mind. I think AI can be more than that. I think AI can be a crane for the mind, if you will indulge my metaphor, in the sense that it can amplify our potential to previously unimaginable heights. Getting there can seem daunting. It has an incredible learning curve. I feel like I’m on a treadmill all the time, but I’m very excited about it. I think it’s a fun challenge, and I think it’s worth fighting this fight. Compared to five years ago, in a way, I feel ... maybe superintelligent is not the right word, but I feel like I have superpowers, given the extent to which AI allows me to take on new ambitious things that were not possible before, and push myself harder than was possible before.

And I think we can ensure that this remains the case even as AI capabilities advance, for the foreseeable future. It might be that in some distant future this becomes impossible to do, but it is very premature to give up the fight now. I certainly plan to continue this fight, and I hope to work towards this vision of co-superintelligence, and I hope you will join me. That is my closing thought.

Our goal in this essay is to move beyond the debate over whether AI is a bubble. We do so in two ways: clearly separating current financials from the question of who captures value in the long run, and recognizing that the labs are not confined to be model providers. They can migrate up the stack and are already aggressively doing so. This will likely allow them to escape the commodity trap but raises new concerns — customer lock-in and reduced competition.

Akash Kapur is a visiting fellow at Princeton and a senior fellow at New America. He is no relation to Sayash Kapoor.

As leading AI companies continue to invest massively in capacity and race toward blockbuster IPOs, serious questions linger about their business models. How will these companies — along with the vast ecosystem of chipmakers, hyperscalers, and infrastructure partners that depends on them — recoup the estimated $4–8 trillion projected to be invested in AI infrastructure by the early 2030s?

The current conversation splits between critics and boosters. Critics point to mounting losses, the gap between capex and revenue, and reports about the leading labs’ massive cash burn. Boosters cite accelerating rapid revenue growth, enterprise adoption, and milestones like Anthropic’s first profitable quarter. Each camp has a valid point. But both are looking in the wrong place — the same quarterly statements, the same short-term view of an industry that remains in flux.

In recent months, we have been thinking about the nature and sustainability of the AI business, and we’ve landed in a different place than most of the existing commentary. AI companies today earn much of their revenue by charging for inference, but the conditions of frontier inference make this an unusually difficult business to maintain. Models are largely undifferentiated, the leading labs operate with similar capital structures, switching costs are low, and prices can be adjusted freely. All of this appears to set up the conditions for a commodity trap that would pose real challenges to the task of building high-margin or even profitable businesses.

At the same time, we believe that the industry remains in a transitional stage, and that its structure will look very different when it matures. Drawing on both historical evidence and economic theory, we argue that competition in this equilibrium is likely to push the price of model inference toward the marginal cost of producing tokens, leaving little room for durable profits at the model layer. This does not mean, however, that the business of AI is inherently unviable. The same analysis suggests a path forward for AI labs, and leads to the central argument of our paper:

The labs’ most likely path to durable profitability runs not through the foundation layers (chips, datacenters, models) that have thus far accounted for the bulk of investments, but higher up the stack, through a mix of vertical integration, embedded enterprise deployments, and the deliberate construction of switching costs and other “moats”.

The labs’ strategies to capture value by moving up the stack, many borrowed from the playbook of enterprise software, have already begun. Beyond the sustainability of the current AI ecosystem, they raise questions of broader societal concern — about competition, innovation, and the overall distribution of economic and political power. The public discussion over AI so far has been marked by a somewhat paradoxical dichotomy: anxiety about monopolistic concentration and runaway market power, yet a reality of low switching costs and relatively interchangeable models that seems to belie those fears. But if we are right that the labs will increasingly move higher up the stack, then concerns about concentration and competition are worth taking seriously now, rather than after the effects of lock-in start to materialize. We return to these broader issues in the conclusion — and in a forthcoming paper that dives deeper into many of the topics covered in this post.

Historical analysis: the infrastructure layer overwhelmingly fails to capture the value that it creates

The AI as Normal Technology framework is committed to drawing lessons, when applicable, from past transformative technologies. We think AI is subject to many of the same dynamics related to investment, competition, and value capture that have shaped previous waves of technological innovation. As part of our research, we therefore examined the puzzle of AI value capture, and the labs’ likely response to it, through a broader historical lens.

We looked at six historical instances of capital-intensive infrastructure industries (railroads, electricity, telecom and fiber, cloud computing, semiconductor manufacturing, and commercial aviation). We believe that AI today has many infrastructure-like characteristics: massive capital requirements; low marginal cost; a commodity product that is somewhat decoupled from the applications that ultimately create value. This makes infrastructure a notoriously tough business to be in.

But at the same time, AI is software, and the software business has historically been lucrative, with exceptionally high margins. The industry has software-margin ambitions. Thus, we also surveyed the value-add and lock-in strategies of software-as-a-service, and analyzed whether AI can replicate these. Our thesis is that AI companies’ sustainability and value-capture largely turns on how successfully they can migrate from the first set of infrastructural properties toward the second enterprise-software ones.

Broadly, we identified three instructive lessons for AI from our historical analysis. First, infrastructure providers rarely capture the value they create. Across railroads, electricity, telecom, and airlines, the firms that built capacity were eventually competed, regulated, or commoditized into thin margins. In many cases, they were destroyed outright. During the telecom and fiber buildout of the late 1990s, capacity exploded 186,000-fold in seven years, prices crashed, and roughly $2 trillion in market capitalization was erased. The value generated by the infrastructure primarily accrued to industries and applications built on top of it. Commercial aviation has destroyed investor capital for eight decades, because typical net margins are 2–4%, often below the cost of capital — even as businesses of all stripes benefited from a globalized economy.

We believe that AI, at least in its current hyperscaler form, risks falling into the same commodity trap that has bedeviled so many previous infrastructure builders. Carlota Perez has theorized the paradoxical phenomenon that the builders who create infrastructure during the “installation period” rarely survive to capture the value it creates.

Second, enterprise software is not subject to this pattern, and it escapes the trap through a specific and reproducible set of mechanisms. Where infrastructure firms have struggled, enterprise software has sustained gross margins of 75% or more for decades. It does so by combining three properties that infrastructure businesses lack: zero marginal cost of reproduction, deep switching costs, and non-ephemeral value that allows fixed buildout costs to be amortized over decades. The AI labs’ lock-in strategies are best understood as attempts to import these structural properties of software into AI.

Finally, two partial exceptions reveal what it may take for AI to beat the historical odds. Two infrastructure businesses — cloud computing and chip fabrication — have managed to escape the commodity trap. Cloud acquired software-like properties (managed-services lock-in, egress fees, committed-spend agreements) and TSMC achieved a near-monopoly position in leading-edge fabrication. These cases matter because they show what escape from the commodity trap actually requires. Capital-intensive industries can sustain durable margins, but only by either becoming functionally software or achieving market concentration.

Why we think charging for model inference won’t allow recouping infrastructure investment

As noted, critics of the AI buildout point to the labs’ current losses and boosters to early profits, but both are committing the same error: mistaking the transitional period for the equilibrium. We are still in the early years of the AI transformation of the economy, and naive extrapolation is flawed. In equilibrium, both the supply and demand side will look very different than they do today.

A few examples of the fluctuating fortunes of the AI labs so far: A massive and speculative infrastructure buildout that was arguably ahead of demand; the Deepseek moment during which open-weight models almost caught up to the frontier; a surge in demand as companies embraced AI agents, often accompanied by irrational and wasteful practices such as token leaderboards, leading to a so-called era of token scarcity; and, most recently, a renewed push for a competitive open-source ecosystem as companies reconsider their spending on frontier models.

These ups and downs have many parallels in the historical case studies mentioned above. The railways at first saw a speculative frenzy (the railway mania of the 1840s in Britain). In America, over time, demand exceeded capacity, resulting in estimated direct and indirect returns of 15% per annum on railroad capital through 1860 — enormous for the era. But as the network matured, capacity caught up to demand, competition squeezed margins, and many railroad companies went bankrupt.

Thus, to better forecast the labs’ long-term prospects, it is helpful to step back from the year-to-year and quarter-to-quarter swings in financials (and narratives), and ground our analysis in economic theory. A key concept is the Bertrand paradox: this says that when firms sell a homogeneous product (frontier-model inference), price competition will force them to sell each unit (token) at the marginal cost of producing it.

The theoretical analysis is important because the question of how well the historical record predicts the AI industry’s fortunes is contested. For example, our colleague Mihir Kshirsagar identifies regulation as the main reason why railroad, electric and telecom companies were unable to extract the surplus generated by their infrastructure, rather than viewing it as an inherent consequence of their economics. Thus, the theory gives us an additional reason to expect a margin squeeze. Although the theoretical paradox is easy to escape in practice, we believe that model inference contains an unusually pure version of the conditions that produce the paradox:

• Models are largely undifferentiated. At least three companies, OpenAI, Anthropic, and Google have managed to stay on the frontier and produce models that behave and perform similarly to each other. Besides, for an increasing fraction of use cases, frontier performance is not necessary, and open-weight models are good enough. Furthermore, the near-equivalence of models is readily observed or perceived by customers, because there are hundreds of benchmarks on which models are regularly evaluated and frontier models all tend to cluster near the top.
  
  For a revealing contrast from another industry, Apple famously sustains high margins by escaping this aspect of Bertrand competition. It resists competing on objectively observable dimensions of performance (gigahertz, megapixels), instead advertising outcomes and ineffable qualities (“it just works”) and creates strong brand differentiation.

• The vendors all have similar capital costs. AI knowledge diffuses rapidly; the major paradigms of model development — model scaling, inference scaling, reinforcement learning, and so on — have all proceeded in near-lockstep. So to produce a token at a given output quality, the labs all spend similar amounts on training. In a hypothetical future where the frontier doesn’t move as rapidly, costs will be dominated by inference, and inference cost tends to be even more similar than training cost.

• Lack of geographic differentiation. In the case of the railroads, a patchwork of local monopolies, and captive capacity, provided some defense against margin compression. The business of AI inference lacks this differentiation, and firms can effectively serve the full market. While there is arguably a token capacity constraint today, we expect this to be temporary.

• Prices can be changed freely, with no collusion. Railroads and utilities were able to collude tacitly or explicitly to keep prices high at some points in their history. But this is highly unlikely with AI models because there is fierce competition from sub-frontier models.

• Switching costs are minimal. Even the friction of migrating APIs is largely eliminated by a layer of provider-agnostic routing tools.1

One potential response to all this is to argue that it doesn’t matter how much margins get squeezed, because AI companies will simply make it up in volume. But the numbers don’t work out. If we assume a 5% net margin — much higher than airlines — and the need to recoup the aforementioned $4T — $8T of investment over a 5-year horizon, we end up with a requirement of $16 – $32T in annual revenue, the higher end of which is a quarter of today’s world GDP.

Taking this longer-term perspective, grounded in history and economic theory, allows us to ignore several factors that other analysts tend to critically index on. For now, we are in a period in which massive, capital-intensive training runs continue to produce better models. This forces labs to stay on an upgrade treadmill in which they reinvest their revenues into training (and then some) taking on ever-higher investments. When this treadmill slows,2 the business will reach a new stage of equilibrium that is likely to look very different from today. In that state, three factors that currently dominate the debate will matter far less than they appear to now.

The first is falling token prices driven by efficiency improvements, which critics cite as evidence of a race to the bottom that will shrink the size of the pie. In our view, while it is true that token prices have been falling rapidly (and that this trend will likely continue), it is not necessarily a problem for the labs. Depending on the shape of the demand curve, unlocking demand could more than compensate for falling prices — the so-called Jevons’ paradox, in which efficiency gains lead to greater (not less) consumption of a resource.

The second is token scarcity, which boosters cite as evidence of the labs’ pricing power. Any scarcity, however, is likely a temporary phenomenon that will be alleviated or eliminated by a combination of infrastructure investment and algorithmic improvements. Token scarcity is therefore unlikely to allow the labs to increase their margins.

The third factor that dominates the current discourse is willingness to pay. Critics sometimes predict that demand will evaporate when labs end their subsidized, below-cost subscription pricing. But this treats willingness to pay as a fixed quantity, something that can be extrapolated forward from today. Over the long run, we expect that AI will be an essential part of most knowledge work, and that the technology will be as transformative as the industrial revolution. In a transition of this scale, the willingness to pay is potentially enormous.

In other words, we must distinguish between value creation and value capture. We expect the value created to be unfathomably large. The central — perhaps even existential — question for the labs is whether they will be able to capture even a small percentage of that value. What fraction of the value AI creates for a company will that company be willing to pay the AI vendors? This is not some free-floating, exogenously determined number. It is largely determined by the extent to which AI labs are able to escape the commodity trap.

As we argue below, the most reliable way to escape the commodity trap is to move downstream in the economic value chain (in the case of AI, this means moving “up the stack”). Transitioning from selling physical infrastructure or products to bundles that include services is sometimes called “servitization”. For example, as hardware became commoditized, IBM successfully transitioned to a services company, eventually selling its PC division altogether to Lenovo. The telecom companies also made sustained, strenuous efforts to avoid becoming “dumb pipes”, but unsuccessfully. Value was instead captured by applications that utilized the infrastructure, leading to the rise of Big Tech. We believe that labs now face much the same fork: capture value in the layers above the models, or watch others create lucrative businesses on top of the infrastructure they built and paid for.

How the labs are responding

To summarize our argument so far, our historical analysis as well as economic theory suggests that there exist only two plausible routes out of the commodity trap: establishing monopoly-like control over the market, or capturing value at higher layers of the stack, often accompanied by increased switching costs.

We believe that the first route is unlikely and the frontier will stay competitive. Proponents of AI exceptionalism might disagree, forecasting a development without historical precedent: a capability discontinuity so dramatic that it reshapes the competitive landscape entirely. In this “hard takeoff” scenario, one or more labs achieves recursive self-improvement, rendering the current economics largely irrelevant.

In our full paper, we argue why this scenario is highly implausible. For now, we point out that many policymakers, investors, and other stakeholders are not comfortable betting on such a possibility, and must instead plan for a scenario where it doesn’t pan out.

This leaves moving up the stack as the most likely path forward. We believe AI companies are well aware of the commodity trap; they are full-throatedly transitioning beyond being pure inference vendors (i.e., selling undifferentiated tokens through model APIs), and seeking to pursue value capture at higher layers of the stack. We are in the early days of these efforts and many new and more systematic strategies are likely to emerge (we examine some in the next section). But already, some efforts are apparent.

• Products like ChatGPT or Claude Code sit at a layer above models. Products are not interchangeable in the way models are, for a host of technical, contractual, and behavioral reasons, as we discuss in the following section. In OpenAI’s case, ChatGPT, not the OpenAI API, has long accounted for the majority of its revenues. Anthropic’s revenues in early 2026 were still dominated by its API, but with the explosive growth of Claude Code, that might not last for much longer.

• The next layer in the stack is AI-native SaaS or “intelligence as a service”, and the labs have made early moves in this direction: ChatGPT’s company knowledge feature, and Anthropic’s vision of agents inside the System of Record. However, this is the layer where the labs are most vulnerable to existing SaaS incumbents, notably Microsoft.

• AI companies are also pursuing bespoke deployment and workflow redesign — through forward-deployed engineers and partnerships with consulting companies. This doesn’t fit cleanly into the stack. The stack assumes an equilibrium in which workplaces have been transformed through AI, whereas this category is about attempting to monetize the transformation itself.

• The most speculative, and more potentially lucrative, avenue for value capture is the top layer of the stack: the idea of AI agents as “digital workers”. Many companies have been touting this idea. Perhaps the first release from a major AI company that has been designed to function analogously to a digital worker rather than a tool is Claude Tag, launched in June 2026. A digital worker that is embedded with every team in a company quickly becomes a store of tacit knowledge for the whole organization and becomes essential to workflows and business processes across different teams. Unless portability is explicitly built in, it becomes a digital employee that effectively cannot be fired, creating a level of lock-in that exceeds that of enterprise software at its worst. When companies sell digital workers, the total addressable market is the economy’s entire labor spend, far larger than companies’ IT budgets. But precisely for these reasons, the vision is likely to encounter the most resistance from enterprises.

Figure 1 illustrates some of these ongoing efforts. Note that the stack describes what the customer pays for, not the billing unit. Token pricing can be used at any layer of the stack, and other units (subscriptions, outcomes) can be used in combination or as substitutes at the higher layers. It is the type of value at each layer of the stack that determines whether the commodity trap applies, not the pricing model. We discuss pricing strategies in more detail in the following section.

A key insight of this essay is that companies that operate higher in this stack can capture more of the value that they create. They can better differentiate their offerings and build a variety of moats (as we discuss below), many of which are arguably novel and more powerful than enterprise software enjoyed.

An important caveat: for AI labs, especially OpenAI and Anthropic, to move up the stack, they must compete with larger incumbents who enjoy many advantages, notably existing distribution channels and deep enterprise customer relationships. Google enjoys easy vertical integration into its existing consumer product line. SaaS incumbents, notably Microsoft, are betting that they can crack the “intelligence as a service” model faster than the AI labs can. This unevenness reshapes the competitive dynamics of the AI ecosystem, and points to important differences among the key players attempting to climb the value stack.

Moving up the stack opens up a varied and powerful set of moats

How does moving up the stack allow companies to escape the commodity trap? The answer is not simply that higher layers create more economic value. It is that they make value capturable and durable: operating higher in the stack allows labs to distinguish their offerings and to construct lock-in and switching costs of a kind unavailable to sellers of raw inference, that is, API providers. They effectively allow AI companies to escape the lack of differentiation that lies at the heart of the Bertrand paradox. Moves up the stack are not just new product strategies; they are moat-building strategies. In this section, we examine several different kinds of moats—some that labs are already building, and some that they may build in the future.

To identify what moats open up as AI companies move up from the model layer, we looked at how enterprise software companies generate pricing power and capture value, then analyzed which of those strategies are likely applicable to AI companies and what new ones become available.

We note two important nuances before we dive in. First, while moats are ways to defend market share from competitors, our interest is in analyzing AI companies’ ability to capture the value that they create for customers. Still, the strategies involved are largely the same, because both require avoiding a pricing race to the bottom.

Second, our analysis focuses on the enterprise market, since we believe that it will account for the bulk of revenues. OpenAI has pivoted in this direction recently while Anthropic has had a consistent focus on enterprise. While it is possible that ad-driven consumer products could be a significant revenue stream in the future, thereby playing a role in escaping the commodity trap, that does not undermine our thesis about enterprise moats.

1. Embedding moats: AI models are something that customers invoke in their workflows. But another vision of AI is a system that the customer inhabits. The latter is more conducive to building moats.

Even if the underlying model stays interchangeable, the state wrapped around it for a given customer may not be. Over time, that state accumulates: persistent memory and conversation history, uploaded document corpora and retrieval indexes, custom skills or evaluation suites, workflows and business processes, and various forms of fine-tuning or customization. This “data gravity” is a big part of what gives enterprise software its stickiness, and why replacing it has been described as “open-heart surgery”.

Both OpenAI (Company Knowledge) and Anthropic (Claude Cowork, the PwC partnership) are already pursuing these pathways. The durability of these strategies remains an open question. One complication is that the data that could help build moats typically sits on companies’ servers in established systems like Salesforce, Workday, and SAP, largely inaccessible to AI labs. To overcome this, AI companies could build moats by controlling the orchestration layers occupied by agents, thus effectively providing the connective tissue that binds an organization’s workflows. However, if open standards keep the orchestration layer thin and swappable, value may ultimately flow to agentic integrators and enterprises rather than the labs themselves.

2. Ecosystem moats: Enterprise software such as Windows, GitHub, and SAP all benefited from the fact that, as they gained more customers or developers that built on top of the platform, the value of the platform grew. The AI market does not yet exhibit strong ecosystem moats. Labs have made efforts to create two-sided marketplaces on top of AI as a platform, but so far these have either been unsuccessful (GPT store) or too thin to create meaningful switching costs (Claude plugins).

A much-discussed but as-yet-unrealized kind of network effect is the “flywheel” of training AI models or systems using customers’ IP, whether data, the environments in which agents execute, the execution traces of those agents, or private evaluation suites. So far, enterprises have been skittish about allowing their IP to be used this way by AI developers, but this may change in the future. After all, it takes only one (suitably incentivized) customer in each sector to defect to set a flywheel into motion.

3. Commercial moats: This category covers a range of familiar strategies, largely borrowed from enterprise software and cloud computing. Commercial contracts can include various mechanisms (multi-year agreements, committed-spend tiers, prepaid credits) that make defection to rivals uneconomic. The labs are already deploying some of these mechanisms in their enterprise agreements, but the durability of this moat is also questionable. Sophisticated enterprise procurement teams will demand portability before signing, and contractual lock-in is highly legible to antitrust regulators.

Vertical integration offers a more powerful version of the same commercial logic. When an AI assistant is bundled into software an enterprise already pays for (Copilot inside Microsoft 365, Gemini inside Google Workspace) the marginal cost of AI appears to be zero, making it difficult for standalone competitors to compete on price. Integrated players can also ride existing sales and procurement relationships: instead of winning a new budget line, they simply fold AI into a renewal conversation for software the customer is already committed to. Of course, this strategy is only available to integrated incumbents like Google and Microsoft. As noted above, this means that standalone labs like OpenAI or Anthropic must compete on a structurally uneven playing field, often renting infrastructure from the very companies they are up against.

4. Behavioral moats: Whereas the moats discussed above operate through technical or commercial mechanisms, behavioral moats work through the humans who use AI systems. Much like earlier waves of enterprise software (Microsoft Excel, CRM systems), the use of AI tools can lead to an erosion of the skill at performing tasks unaided, while at the same time building vendor-specific skills in driving agentic products, creating dependence. This is an area where the societal stakes may be acute, beyond its implications for value capture: skill erosion could have profound implications for workforce resilience, safety, and the distribution of power between people and AI systems.

But behavioral moats also include novel and poorly understood forms of AI lock-in that have no clean analog in earlier software. A notable manifestation is relational attachment to a model’s tone or “personality,” as vividly demonstrated by the #Keep4o backlash, when OpenAI was forced to reinstate GPT-4o in response to user demands. More speculatively, for collaborative knowledge work such as writing or judgment-heavy work (as opposed to automation), the quality of AI’s contribution is hard for the buyer to verify even after the fact, because there are no objective standards. That makes such work a so-called credence good, similar to lawyers and consultants. Unable to compare quality, buyers of credence goods tend to rely on reputation and trust. This mutes price competition and slows switching even when the technical cost of switching is close to zero.

5. Pricing strategies: This pathway is complementary or additive to the ones discussed above, and describes a way in which labs might be able to capture more value after they achieve lock-in and are able to exert pricing power. The idea is that labs may be able to overcome the commodity trap by charging for outcomes (a percentage of cost saved, a fee per resolved support ticket) rather than tokens. This is the explicit thesis behind OpenAI CFO Sarah Friar’s statement that revenue should “scale with the value of intelligence.”

However, this is unlikely in the short term. Outcome pricing requires solving a challenging measurement problem. The labs haven’t built the infrastructure to measure and understand customers’ business processes, and to track — let alone price — different outcomes. Without that visibility, outcome metrics invite gaming by both sides: a fee per resolved ticket incentivizes customers to stuff multiple issues into one ticket and vendors to close tickets without necessarily fixing them. Besides, outcomes vary between verticals and between firms, and AI labs lack the embedded customer teams that SaaS vendors spent decades building in order to gain this kind of downstream visibility. Still, some AI companies are trying to innovate on this front. If AI labs are able to migrate into Systems of Record, the challenges become tractable with enough investment.

Broader implications of the shift

The questions raised by our analysis have a significant bearing on the future of the AI economy, but they extend well beyond the prospects of any individual AI company. They touch on the structure of digital markets, the distribution of economic power, and the kind of innovation ecosystem that AI is likely to produce.

If the labs fail to escape the commodity trap, the consequences are primarily financial: a potential crisis of confidence in one of the largest capital buildouts in economic history, with ripple effects through the broader technology sector and markets in general. But if they succeed — particularly if they manage to increase switching costs and establish customer lock-in — the consequences look different, and in some ways more concerning. It would raise costs for every enterprise that depends on AI and entrench a small number of players in a position of structural advantage that will be difficult to contest later.

A few questions in particular seem worth taking seriously now, before the relevant moats have hardened:

• Can the labs build sustainable businesses without foreclosing competition and repeating earlier patterns of harmful concentration?

• Can strategies that work for investors and AI companies also generate wider social value, or does value capture in AI come at the expense of the surrounding ecosystem?

• Will the gains from AI be diffuse (accruing primarily to the enterprises that deploy it, and the broader economy) or concentrated (largely captured by hyperscalers, frontier AI labs, SaaS incumbents, and investors)?

There remains much uncertainty about these questions, but history suggests that the time to establish interoperability standards, portability requirements, and switching-cost transparency is early, before lock-in compounds and intervention becomes structurally more difficult. Fortunately, competition authorities such as the U.S. FTC and the UK CMA are watching this space, but so far they appear to have focused their analysis on the bottom layers of the stack. Our analysis shows that they must broaden their view.

We are grateful to Sayash Kapoor for feedback on a draft.

Further reading

While many aspects of our argument are at least somewhat novel, the “no moat” argument itself is not new. Its first prominent articulation (that we know of) goes back over three years, in a leaked Google document. Many commentators such as Gary Marcus, Ed Zitron, and Cory Doctorow have relentlessly made this point or variants of it.

What is possibly new in our analysis is drawing from historical parallels and showing that model inference matches the prerequisites for ruinous competition even better than those precedents do. Where we definitely differ from those predicting a bubble is recognizing that (1) the AI stack is quite thick and the commodity trap only applies at the model layer (2) the labs are keenly aware of the problem and (3) have been furiously migrating their way out of it.

Many others are bearish on the labs, including Benedikt Evans and Luis Garicano and Jesús Saa-Requejo. The latter argue that Value capture will take place both at the top of the value chain, the hardware and physical providers [because of the compute bottleneck], and in the implementation layer, where models are put to work and where the binding constraint is organizational, not technical. We largely agree. But they conclude that the labs won’t be able to capture much value. The implicit assumption seems to be that the labs are confined to being model vendors, which in our view is an increasingly outdated notion.

Semianalysis argues that the labs will be able to sustain high gross margins because of compute scarcity. We think this is a temporary state of affairs.

Closest to our thinking is our Princeton CITP colleague Mihir Kshirsagar, who has written a series of four articles on this topic. While we have benefited from conversations with Kshirsagar, our thinking developed largely independently, with many points of agreement and some points of disagreement. Like Kshirsagar, we think the rapid AI buildout doesn’t make economic sense if it weren’t for the promise of lock-in. His analysis identifies coalitions between hyperscalers and AI labs (such as Microsoft-OpenAI and Amazon-Anthropic) as the primary vehicle for lock-in, which is slightly different from our diagnosis.

Nathan Lambert offers one vision for how this might all play out — a two-tier AI economy in which customers who are themselves at the frontier of knowledge work pay big premiums for “up the stack” products and services based on frontier models from the leading labs, while everyone else is satisfied with the open model ecosystem. One of his predictions is sharper than ours: frontier labs will let their API businesses decay in order to protect their higher-margin offerings.

In this essay, we argue that there is enough evidence to reject the narrative that once AI capabilities reach a certain threshold, it will cause mass layoffs. Given that this is true even in a sector with very few regulatory barriers, most other professions are likely to be even more cushioned.

We also have a good understanding of why this is the case. We can think of many kinds of knowledge work, including software development, as a “decide-execute-deliver sandwich”. AI compresses the “execute” layer — the middle of the sandwich — but the other two layers resist automation in a way that will not be overcome by capability improvements alone.

We conclude on a note of cautious optimism about the future trajectory of demand for software engineering. This essay is the first in a series, and the next one will look at reasons why individual software engineers’ careers might be rocky even if overall demand is healthy. The series is based on the published literature in economics and software engineering, our own evaluations and observations of AI agents, and many software engineers’ reflection on the present and future of AI impacts on their profession, gleaned both from published writings and our interactions with the community.

The stories of AI-driven mass layoffs in software seem to be classic “AI washing”

Consider three stories that made the headlines and how they contrasted with reality:

• In February, fintech company Block (maker of Cash App, Square, Afterpay, and other such apps) announced layoffs of 4,000 employees because, according to founder Jack Dorsey, AI is “enabling a new way of working” with “smaller and flatter teams”, specifically citing late-2025 improvements in model capabilities.
  
  But subsequent reporting revealed a radically different picture. After growing headcount more than threefold during the pandemic, the company was under massive financial pressure. A data scientist on the Cash App team, Naoko Takeda posted that Block “shoved AI down everyone’s throats” yet she saw “very limited gains in productivity.” She refused a 75% retention raise and quit. Other employees interviewed had a sharply different understanding of what AI was capable of at Block and whether Dorsey had a competent understanding of the issues.
  
  As Aaron Levie has pointed out, CEOs are uniquely prone to delusions about AI’s usefulness because they can build quick prototypes but can’t see the 90% of work it takes to turn it into a finished product. Dorsey’s public statements about AI seem to fit exactly this pattern.
  

• In April, Snap laid off about 1,000 people, with CEO Evan Spiegel primarily citing AI as the reason in his layoff memo. He also said that AI generated 65% of new code. In reality, the layoffs followed a campaign by an activist investor demanding cost cuts. (Snap has posted a net loss every full year since its 2017 IPO and shares were down over 30% in 2026). Tellingly, the nature of the cuts, such as 150 jobs spanning various roles in the augmented reality division, don’t correlate with the cuts we would expect to see if they were driven by AI (i.e. programming and other “AI-exposed” jobs across the board, not concentrated in any unit).
  

• In May, Intuit announced 3,000 cuts, alongside deals with Anthropic and OpenAI. The press connected the two, framing the layoffs as AI-driven restructuring. For once, the CEO actually pushed back on this easy narrative, saying that “none of it had to do with AI” and that the cuts targeted “coordination-heavy roles” and too many management layers.

We did not cherry-pick these examples. In every story about AI-driven software engineering layoffs that we examined, the same narrative violation emerged. It turns out that “AI washing” of job cuts is an economy-wide phenomenon, evidenced by many surveys:

• 59% of U.S. hiring managers admitted they emphasize AI when explaining hiring freezes or layoffs because it plays better with stakeholders than citing financial constraints.

• Forrester principal analyst J. P. Gownder says of companies preparing supposedly AI-driven layoffs: “When we ask if they have a mature, vetted AI app ready to fill in those jobs, nine out of 10 times, the answer is no—and they haven’t even started.”

• In a HBR survey of over 1,000 global executives, 21% had made large headcount reductions “in anticipation of” AI, with another 39% having made low or moderate anticipatory headcount reductions. In contrast, only 2% had already made large reductions in headcount related to actual AI implementation. The 10x gap suggests that executives, like everyone else, are highly prone to succumbing to the misleading narratives about AI replacing jobs.

Another interesting data point comes from the WARN Act, which requires certain disclosures of plant closings and mass layoffs affecting over 100 workers. In March 2025, New York became the first U.S. state to add an AI disclosure checkbox to WARN Act filings. In the full first year, more than 160 companies filed WARN notices. Not a single one checked the AI box.1 We reached out to the NY Department of Labor who confirmed that as of late May, only one company, Nespresso, checked the box.2 If these filings are accurate, only 46 out of about 25,000 laid off workers in New York State in the relevant period, or about two-tenths of a percent, were affected by AI.

Even more damning for the AI-driven-mass-layoffs narrative: layoffs are the wrong signal of AI’s potential productivity benefits in the first place! The research is clear that the effect operates through “slower hiring rather than increased separations”. Firing existing workers results in the loss of precisely the tacit knowledge and organizational capital that allows workers to operate AI effectively. Besides, it is expensive in terms of severance, damage to morale, and rehiring risk. Given these costs, it is largely unnecessary given that natural turnover achieves the same result in a few years.

So what does the data tell us when we look beyond layoffs to overall employment trends? An important paper from Federal Reserve economists compiles the evidence in the U.S. context. Software engineer employment is still growing, but they find that it is growing slower post-ChatGPT compared to a no-AI counterfactual, by about 3 percentage points per year. One important limitation of this study is that the methodology can’t capture self-employment, so it is possible that some of the slowdown in growth is being absorbed by entrepreneurship instead. We do have evidence from other studies that AI makes entrepreneurship easier. So the real picture is probably even healthier than the Federal Reserve study suggests.3

Finally, it is worth acknowledging two kinds of indirectly-AI-driven job losses in software engineering that are real, but different from AI replacing software engineers. First, AI sometimes decimates demand for the product, in cases like Chegg (homework help) or Stack Overflow (technical help), both of which have laid off workers. AI doesn’t directly do the job that these workers did, but rather obviates the need for it. The historical parallel is strong: Among the 270 jobs in the 1950 U.S. census, only one job was automated away — elevator operator. But many others were rendered obsolete by new technology, like the job of telegraph operator.

Another credible AI-driven layoffs story is among companies that sell AI, rather than buy it. So when companies like IBM or SAP announce layoffs because of AI, a more accurate framing is “we reallocated headcount from legacy functions to our fastest-growing product line.” That’s ordinary corporate restructuring around a revenue opportunity, not technology displacing workers.

Why coding agents haven’t led to labor displacement: the decide-execute-deliver sandwich

Many tech leaders, like the Snap CEO above, report the percentage of code written by AI alongside reports of layoffs or predictions of future job losses. This feeds into the simplistic mental model that once AI writes all the code, there is no need for coders. Fortunately, this mental model is wrong. This AI-written-code metric is almost completely disconnected from what matters for labor displacement. Here’s why.

Writing code isn’t, and never was, the bottleneck. For example, a 2019 paper summarized existing studies with the conclusion that “developers spend surprisingly little time with coding, 9% to 61% depending on the study”. This finding was consistent with the paper’s own data from 6,000 developers at Microsoft. As coding agents began to be taken up, there was an explosion of blog posts in late 2025 pointing out that writing code isn’t the bottleneck, as developers realized that using agents to write most of the code led to little impact on overall productivity [1, 2, 3, 4, 5, 6, 7, 8].

If writing code isn’t the bottleneck, what is? The task-breakdown surveys point at things like meetings or debugging. This just leads to more questions: what are developers doing in those meetings and why can’t it be done by AI? Won’t debugging get automated as capabilities improve? To understand the real bottlenecks, we have to get qualitative, and dig into software engineers’ own understanding of what it is they do that resists automation.

When we did this analysis, it revealed three things as the real bottlenecks (1) deciding and specifying what to build, (2) verifying and being accountable for what is delivered, and (3) the deep human understanding — of the codebase, the business, and the environment — required to carry out both of these.

In other words, software engineers’ work consists of a “decide-execute-deliver” sandwich (with understanding being a prerequisite for all three). AI has compressed the middle of the sandwich, but has left the two ends largely unchanged. As long as software development teams are in charge of decision making and accountable for what they deliver, engineers still need to spend time building up a deep understanding of the system. These are the three bottlenecks.

Figure: Software development consists of three layers: (1) Decision making — problem framing, specification, planning (2) execution — design and implementation (3) delivery — testing, verification, integration, maintenance, etc. Note that these are conceptual layers, not temporal phases. It is common to switch back and forth in the course of a project.

Evidence for the sandwich model of AI’s productivity effects comes from a recent paper on “Writing Code vs. Shipping Code”. Across 100,000 developers on GitHub, the researchers found that AI agents led to an eight-fold increase in the number of lines of code written, consistent with the idea that AI almost completely compresses the Execute layer of the sandwich. But this led to only 30% more releases, strongly suggesting that human bottlenecks (the Decide and Deliver layers) remain in place.4

Can the sandwich be further compressed? We don’t think so. At one end of the pipeline, development teams need to decide what to build. One of the most important lessons junior software engineers learn is that requirements specification (the profession’s lingo for this layer) takes surprisingly long, and if it is compressed, it leads to much more pain down the line. This layer is hard to automate because it requires thinking about user needs, market signals, organizational priorities, and in some cases regulatory constraints.

As AI capabilities improve, the kinds of decisions that can be delegated to AI increase over time. But this does not make the “decide” layer thinner — once a decision can be delegated to AI, it is no longer a source of competitive advantage, and the value of human decision-making migrates upward. Software increases in complexity over time, so there is no ceiling to this process.

At the other end of the sandwich, human teams need to be accountable for what they deliver. It is possible that some day in the future teams will ship mission-critical code without fully testing and understanding it, but today’s AI is so unreliable that such haphazard practices would represent an existential threat to software teams and their customers.

Even if the technical barriers go away in the future, we don’t have to cede control to AI. A central insight of AI as Normal Technology is that we can collectively choose to keep humans accountable through shared norms, law, and policy. This is a much more resilient way to control the speed of AI impacts and improve safety than trying to slow the development of technical capabilities. These speed barriers are already largely in place due to liability laws and sector-specific regulation, but can be further strengthened. (For a longer version of this argument, see the original essay.)

In this vision, as more and more of the execution layer gets delegated to AI, the software engineer’s role in the future becomes analogous to that of a crane operator. AI agents will do most of the cognitive heavy lifting; supervising the agent and keeping it in control becomes most of the human’s job.

Some commentators argue that a future with humans staying in control is unlikely because it is too costly to pay people to do so. There have already been a few viral stories of poorly-supervised coding agents deleting production databases or causing other types of damage. But we view these as “man bites dog” stories rather than an emerging norm. They go viral precisely because they represent such irresponsible and unusual behavior that they have shock value, and serve as regular reminders and learning moments helping the community guard itself against over-reliance on AI. As the aphorism goes, “if it’s in the news, don’t worry about it”. Still, being able to detect whether there is an uptick in poorly-supervised use of AI for high-stakes tasks — across the economy, not just in software engineering — remains one of the most critical data gaps we have today.

By the way, the sandwich getting squished is not a new trend and it is not uniquely due to AI. Over two decades ago, the Bureau of Labor Statistics started tracking programming separately from software engineering. Roughly speaking, programmers are responsible only for execution while software engineers manage a bigger part of the sandwich. Not only has programming been shrinking, it is also pays much less because it is seen as grunt work. AI merely accelerates this long-existing trend, further devaluing purely technical skills.

This pattern — where humans remain heavily involved at both ends of the decide-execute-deliver sandwich, even as AI increasingly automates the middle layer, seems to be broadly applicable to most knowledge work, though it is farthest along in software. After all, complex decision making and accountability are common to most fields. A lack of recognition of this phenomenon has led to many overconfident claims about imminent job losses, such predictions about AI replacing radiologists.

Vibe coding is not agentic engineering

One reason for confusion about the extent to which software engineering is changing is the sloppy use of the term “vibe coding” to refer to a wide spectrum of practices, the ends of which are conceptually distinct and more dissimilar than similar.

In true vibe coding the user simply tells the agent what to do, doesn’t supervise it when it’s running, doesn’t review the code — might not even have the skills to do so — and doesn’t evaluate the output, beyond perhaps noticing when things are visibly broken.

This is in contrast to how most software engineers are actually using agents — as a tool, with the human remaining in control and accountable for the output. Fortunately, the term agentic engineering is gaining currency as a descriptor of this practice.

As agentic engineering has become the norm, engineers are discovering that supervising coding agents is surprisingly time consuming. For example, Simon Willison, a prominent developer and chronicler of the AI transition, has noted how he is mentally exhausted by 11am from supervising agents. This is consistent with our experience as well.

More quantitative evidence comes from SWE-chat, a dataset of coding agent interactions from open-source developers who opted into a logging tool. The study found that only 44% of agent-produced code survives into user commits, that vibe-coded commits introduce vulnerabilities at nine times the human-only rate, and that the most common user intent is understanding existing code, not generating new code (19% vs 13%). The self-selected nature of the dataset means that we can’t draw strong conclusions based on this study alone, but it does reinforce many other lines of evidence that vibe-coding and agentic engineering patterns are quite different.

To re-iterate, these are not two distinct categories. They are two ends of a spectrum, and there is a blurry middle. Not every project is either a throwaway or mission-critical. Not every workflow fits precisely in the left column or the right column of the table. But the key implication for the jobs question remains solid — companies can’t ship production software by hiring unqualified vibe coders instead of software engineers.

What does the future hold?

AI boosters might claim that mass layoffs are coming; they just haven’t happened yet because human-level software engineering abilities are very recent (or haven’t been achieved yet). But if the sandwich model is correct, these predictions won’t come true. AI has already largely compressed the middle of the sandwich (and the compression actually started decades ago). So even making the execution layer instant and perfect will only be a small change from the status quo. The reasons why the other two layers have resisted AI is not because of capability limitations.

In fact, not only are software engineering jobs not going away due to AI, there might even be an increase in demand for software engineers. When software (or anything else) gets cheaper to create due to technological productivity improvements, people will buy a lot more software (in econ jargon, software is highly “price elastic”). And as we have argued, AI doesn’t replace software engineers (the “elasticity of substitution” is low), so the demand for more software results in a derived demand for more software engineers. A loosely related but flashier economics term, “Jevons’ paradox”, is often thrown around in the AI discourse to describe this concept.

Historically, this has been the pattern — programmer employment in the U.S. has grown from near-zero around 1950 to millions today. This is sharply different from occupations such as agriculture in which labor demand was famously decimated due to mechanization and automation. The difference is that the amount of calories people consume is relatively fixed — even a 25% increase led to the obesity epidemic — whereas the amount of software produced has grown a millionfold. Modern cars have something like a hundred million lines of code running on their various on-board computers.

If there is a ceiling to the demand for code, we are nowhere near it. Virtually all cognitive work benefits from software. As AI makes coding cheaper, people are creating all kinds of one-off utilities — whether for work or personal use — that it never made sense to create until now.

To be clear, while we think there will be a lot more software in the future, and likely more software engineers, this doesn’t mean big tech companies will get even bigger. The majority of software engineers today already work in-house in non-software firms, and that share might grow in the future. Then there’s the idea of “AI rollups”, which refers to venture capital or private equity firms buying “Main street” businesses — dentistry practices, accounting firms, and whatnot — and rebuild them from the ground up to be “AI-native” by embedding software engineers or AI engineers into those businesses. Of course, it might end up being nothing more than hype. It’s too early to tell.

Some people predict that demand for software engineering skills will fall because of democratization. They acknowledge that there will be more software produced than ever before, and also that more human time will be spent producing software than ever before, but that this work will be done by people who are not software engineers. The idea is that AI will democratize software engineering to the extent that legal software, for instance, can be more easily created by those with training in law than in software engineering.

Maybe. But we’ll bet against it. In our view, this falls into the same trap of conflating vibe coding with agentic engineering, and the execution layer with the the whole decide-execute-deliver sandwich. In fact, when we look at the history of programming, there have always been claims that we are at the threshold of democratization — old languages such as FORTRAN, COBOL, and SQL were all accompanied by such prominent hopes at the time of their introduction. It never happened. The barrier isn’t actually learning the syntax. It’s having enough skilled judgment to make good decisions while maintaining accountability.

Ultimately the distinction may be semantic. It seems clear that the amount of time people spend on getting computers to do new things will increase over time. This might take the form of building software, or managing complex workflows using agents, or something else. It will require a mix of software skills, AI skills, and domain expertise. Whether it is today’s software engineers who will best adapt to fill these new roles remains to be seen.

That last point about the need for adaptation sets up the next essay in this series. The fact that aggregate labor demand in software is likely to remain strong doesn’t mean that most individual workers won’t be affected. We will argue that AI will create massive structural shifts in how software is produced, which will have big impacts on which software engineers stand to gain or lose — based on the types of firms they work in, their geography, their seniority, the pace at which they can adapt.

Further reading

Deena Mousa points out the superficiality of broad, economy-wide analyses of AI impacts based on metrics like “AI exposure”, and instead calls for “careful, occupation-specific work”. We hope that this series of essays will play a role in establishing a nuanced understanding of AI’s transformation of software engineering. We’ve previous coauthored, with Justin Curl, a paper analyzing AI in legal services that seriously engages with regulatory and other bottlenecks that make that occupation unique. We plan to do more occupation-specific deep dives in the future.

In a remarkable essay called No Silver Bullet 40 years ago, Fred Brooks distinguished between the “essential complexity” and “accidental complexity” of software. He argued that some of the complexity of software is accidental, arising from limitations of present technology such as the clunkiness of programming languages, and can be alleviated over time as tooling improves. But some of it is essential, because specifying the correct behavior of software is itself hard. He presents a forceful articulation of why the “decide” layer of the sandwich is thick and resists automation. Interestingly, hopes of boosting programmer productivity through AI were already prominent back then! Brooks argues that because AI or any other technology only reduces accidental complexity, it won’t result in an order-of-magnitude productivity improvement. (Brooks is the author of The Mythical Man Month, an essay collection that is almost certainly the best known and most influential writing on software engineering of all time. No Silver Bullet later became part of the collection.)

We are grateful to Felix Chen for feedback on a draft.

At Google’s developer conference earlier this week, the company launched its latest model, Gemini 3.5 Flash, alongside a new agent app, Antigravity 2.0. To showcase what this new agent setup is capable of, Google claimed that a team of agents had built an entire operating system. The effort reportedly required only a single prompt, cost only about $900 in API fees, and was carried out by a few dozen subagents working together.

Does this mean that complex pieces of software can now be built cheaply by AI? Not so fast:

• The “single prompt” claim is misleading. The blog post says the operating system was built from a single prompt. But halfway through the post, Google discloses that the prompt “ended up being many thousands of lines” long. How many attempts did it take to generate the prompt? How specific were the instructions to the agent? Without these critical details, it is hard to know if the secret sauce is a better model or just more effort put into prompting the model. Moreover, the run was carried out on a scaffold1 with specialized roles, delegation to subagents, and an agent to detect and prevent cheating. In the launch post, Google views the scaffold as a product feature. But we don’t know whether the scaffold was overfit to this task of building an operating system from scratch, or whether it would perform as well on other complex software engineering tasks.

• Google’s writeup is not explicit about what counted as human intervention. The post mentions that the final run to develop the operating system required “no additional guidance or corrections from a human.” But it does not define that standard. It describes infrastructure to kill and restart stuck agents. The post mentions an earlier run in which the agents appeared to cheat, after which the team added anti-cheating measures and re-ran the task. But it does not report dry runs as part of the methodology. Nor does it clearly say whether any agents escalated to a human, whether the final run required any manual restarts, approvals, or fixes, or how many retries it took until the agent was successful.

• The writeup does not report any attempt to analyze whether the agents wrote the code from scratch or copied existing code from the internet. To Google’s credit, the blog post notes that toy operating systems are common undergraduate course projects, and public implementations are easy to find. The post itself raises the concern that the agent could have regurgitated information rather than building the operating system from scratch. But it did not address this concern—there was no similarity analysis or log analysis to check if the agent copied existing code. Even if there was no direct copying, writing an operating system might be relatively easy for agents because of patterns memorized in the training data, so this doesn’t tell us much about agents’ ability to create novel pieces of software.

• Google has not released the lengthy prompt, the code the agents wrote, or the logs from the run, which makes it impossible to independently evaluate the claims. Releasing the source code or the agent logs could have allowed independent researchers to evaluate the quality of the artifacts and answer questions such as whether the agent was copying existing code. The blog post only includes a short video documenting a snapshot of the development progress and the overall narrative of the experiment.

On the other hand, the blog post does report the exact dollar amount for building the operating system ($916.92), alongside the total token budget (a total of 2.6B tokens). These figures provide useful context, which we want to credit Google for. Many of the evaluations we previously surveyed did not disclose cost at all, which made their headline claims hard to compare with other evaluations.

Still, Google’s blog post is effectively a press release. We recognize that it is unrealistic to expect it to be scientifically rigorous. Evaluations like this one, meaning a long-horizon real-world task evaluated on a single run with the experimenter narrating what the agent did, have become common. Since many of them have been done by AI companies, it is easy to dismiss the entire genre as puffery.

But that would be a mistake. We refer to the emerging paradigm as open-world evaluations, and we recognize this trend in a recent paper (and an accompanying blog post). Crucially, we argue that open-world evaluations require a new set of methodological norms. Done right, they can provide a valuable perspective that benchmark-based evaluation cannot.

Google’s experiment does add to the mounting evidence that agents or agent teams can autonomously or near-autonomously work on certain kinds of tasks for very long periods of time, making progress without getting stuck or confused. As we argue in our paper, benchmark evaluation is effectively impossible for this kind of task for many reasons including cost. So it is an exciting time for independent evaluators from academia, nonprofits, and government to step in and provide the kind of rigor and credibility to open-world evaluations that are unlikely to be found in AI vendors’ own claims.

But when it comes to AI risks, he is far more bearish. He points to examples of cyber- and bio-risks and expresses pessimism about AI quickly becoming dangerous across many new domains. He argues that AI’s emergent capabilities make it fundamentally different from previous technologies, and that this difference justifies “extraordinary” government responses including restrictions on what companies can release.1

In this essay, we lay out the downsides of extraordinary government intervention in response to new technology. We discuss proposals for improving resilience that do not require such intervention. We also discuss why governments have so far been reluctant to invest in resilience. In short, resilience requires us to get better at the *normal* process of policymaking. But sclerosis in the federal government and the ease of justifying interventions on AI companies rather than society at large make extraordinary intervention seem appealing, despite its limitations.

What happens next? Policymakers can either get their act together and invest heavily in improving resilience, or be forced to take extraordinary actions (such as on AI nonproliferation) that are more onerous and less effective.

A Recap of AINT’s Arguments and Thompson’s Areas of Agreement and Disagreement

Many people, including Thompson, have found AI as Normal Technology to be a useful framework for thinking about AI’s economic impacts while being unconvinced by our views on safety. In the AINT essay, we make different arguments about slow labor market impacts and resilience to misuse risks.

The labor market argument rests on the speed of diffusion: there are many speed limits between a new AI capability and its economic impact, including the need to build products, change organizational workflows, and navigate regulation. This has proven helpful both to understand why claims of rapid and widespread job displacement (such as Amodei’s claims about an imminent white-collar bloodbath) are unlikely to materialize, as well as to identify bottlenecks to diffusion that could hinder the beneficial adoption of AI.

But our argument about AI’s misuse risks depends on the offense-defense balance: whether attackers or defenders benefit more from a given capability, and our ability to build resilience in response to misuse risks. Notably, an attacker does not need to go through the slow process of organizational adoption to cause harm. We argue for building societal resilience to intervene on misuse risks and identify many interventions that we should take to reduce AI risks.

Here, Thompson disagrees with us. One reason he disagrees is that all general-purpose technologies bring about new risks, which are hard to reason about from previous patterns. But even in the history of general-purpose technologies, Thompson argues that AI seems particularly “abnormal” because of risks that are emergent and unknown even to AI developers. And this justifies plans by the government to treat AI as an abnormal technology, since it “compels the government to create extraordinary regulations that prevent private companies from selling their products and services on the grounds that they’re too dangerous.”

Thompson doesn’t define precisely what he means by extraordinary government action, and the specific interventions he envisions remain somewhat vague. Since we’ll be arguing against such interventions, we want to be clear about what we mean by the term. We think of extraordinary interventions as a spectrum defined by three factors that apply to any powerful technology, not just AI.

First, extraordinary interventions tend to be precautionary. They restrict activity based on anticipated harms rather than realized or demonstrated ones. This doesn’t mean that precaution is never warranted, but it does mean the justification needs to be stronger, since we are restricting activity without clear evidence that the harms will materialize (or that they’ll materialize exactly as we predict). Even then, an intervention is more extraordinary when viable alternatives exist that can address the risks while being less restrictive, such as investing in resilience.

Second, extraordinary interventions impose restrictions on the liberty of actors who are not directly responsible for the harms in question. When governments restrict what AI companies can release, the burden falls not on the malicious actors who cause harm, but on companies that build tools that could, in principle, be misused. This is especially pertinent for dual-use technologies. Because these tools have widespread beneficial applications, restrictions on companies can cut off beneficial access for the broader public in order to prevent misuse by a small number of bad actors.

Third, extraordinary interventions bypass normal processes of governance, and instead rely on unilateral authority such as emergency declarations or executive orders, even though the governance processes being bypassed exist to ensure that restrictions on liberty are subject to democratic accountability.

An intervention need not satisfy all three of these criteria to count as extraordinary. But the more of these factors that are present, the higher the bar should be for justifying it.

Extraordinary Government Action is Extraordinarily Costly

We agree that AI poses misuse risks. But our experience from regulating “abnormal” technologies shows how burdensome extraordinary government interventions can be. The enforcement of nuclear nonproliferation has required the IAEA, the Non-Proliferation Treaty, decades of diplomacy, ongoing investments, and even military confrontation. The tolls of these interventions were enormous, but the approach was at least somewhat enforceable because nuclear weapons depend on enriched uranium, a physical bottleneck that is genuinely hard to get around.

AI is different from nuclear weapons. For one, there is no equivalent “physical” bottleneck. The core techniques for building AI systems are well known. Adversaries (especially nation-states) can match frontier capabilities within months. Any nonproliferation regime for AI would face constant erosion.

In the face of this challenge, how could governments maintain nonproliferation? Some proposed interventions, such as the rumored executive order on voluntary commitments by AI companies for predeployment evaluations, are relatively low on the scale of extraordinary interventions. Done well, they could allow us to tilt the offense-defense balance by giving defenders more time to prepare for new capabilities.

The U.S. has also enacted export controls on chips. We think this is a modest intervention: countries routinely restrict exports of sensitive goods to maintain their lead in innovation, and reasonable people can disagree about where to draw the line. But when it comes to preventing dangerous capabilities from being widely accessible, export controls are far less effective. Open-weight models and widespread API access from frontier labs mean that the gap between frontier and publicly available capabilities is at most a few months, not years.

If the most that nonproliferation can buy us is a few months, the urgent priority should be investing in resilience so that we are better prepared when those capabilities inevitably become widely available, as we discuss below. On the other hand, if governments try to maintain nonproliferation as a way to deter the availability of advanced AI capabilities for misuse, their interventions will necessarily get more demanding.

To truly enforce nonproliferation, governments would need to restrict access to open-weight models and even API access to capable models. This would require licensing regimes that give governments ongoing authority over which models can be released and restrictions on open-weight models. We might quickly enter a state where governments exercise control over what AI research and products can be shared publicly. In fact, we have already seen concerning examples of such control, such as Anthropic’s designation as a supply chain risk, and recent rumors about licensing requirements for AI companies.

To avoid this slippery slope, those proposing extraordinary interventions should be clear about where they would draw the line. Would they support restrictions on open-weight models? What about requiring approvals for each new model release, or restrictions on the movement of researchers who build frontier AI across countries? If proponents cannot specify the limits of what they are calling for, it is reasonable to expect that the demands for increases in the government’s ability to take unilateral action will keep escalating as capabilities advance.2

For nuclear weapons, once the nonproliferation regime was built, it did not need to be rebuilt every few years. But AI is not the last digital technology with powerful dual-use properties. New technologies would raise similar questions and potentially demand similar or escalating responses. So the “abnormal technology” framework for regulating AI would start to look less like a targeted response to a specific risk and more like a permanent expansion of government powers over what citizens and companies can build, publish, and research.

We have seen this debate play out before. Each new technology raises questions of what restrictions to liberty are appropriate to mitigate harms. This history tells us we shouldn’t automatically default to imposing extraordinary government interventions.

The internet allowed people to access information about how to build bombs, and in 1995, after the Oklahoma City bombing, the late Senator Feinstein introduced a bill that, in its first draft, would have criminalized the distribution of any information on the internet describing bomb-making materials or processes. (When the bill was eventually passed, it had a narrower scope, requiring offenders to have knowingly aided a crime through their instructions.)

The federal government also tried to restrict access to encryption software, arguing it would help criminals communicate beyond the reach of law enforcement. It imposed export controls and proposed requiring backdoors in encryption so that the government could always access private communications. It even started a criminal investigation against a programmer under the Arms Export Control Act for releasing encryption software.

These restrictions were eventually rolled back through a combination of court rulings and executive action. Encryption became the foundation of digital security, enabling e-commerce, online banking, and many other applications.

Yet, at other points in history, we have accepted increases in government powers in response to new technologies. The widespread acts of terrorism after the invention of dynamite catalyzed the expanding government surveillance apparatus by the FBI.

The question is whether this level of intervention is necessary to address AI risks. One response that does not require extraordinary government intervention is improving resilience.

Resilience is an Under-Emphasized Defense Against AI Risks

Resilience is the capacity of a system to withstand and adapt to harm. It was one of the main defenses we proposed in the AINT essay. Unlike extraordinary government actions like restrictions on what companies can release, resilience does not impose costs on AI companies. Instead, it focuses on improving our capacity to respond to and recover from AI risks, regardless of when or where they occur. For past technological harms, improving the resilience of systems has been key to reducing harm.

Consider cybersecurity. The internet created entirely new classes of attacks, such as worms that spread through networks, causing billions of dollars in damages. The way we defended against these risks was not by restricting access to computers or the internet, but rather through bug bounties to incentivize people to report vulnerabilities to developers, improving browsers and operating systems, automated testing, and better patching practices. All of these improved the resilience of our cyber infrastructure regardless of where the risks arose.

Automated vulnerability detection tools are another example of technology enabling new kinds of attacks. Tools such as fuzzers and symbolic execution engines have been “superhuman” at vulnerability detection for years; they can detect vulnerabilities at a scale humans cannot match. Yet, they have been freely available on open-source repositories. Since defenders had access to the same tools, they became core defensive tools, largely funded by the cyberdefense ecosystem. In fact, defenders had structural advantages in using these tools effectively, such as deeper access to the systems being tested. This in turn led to better protections for cybersystems.

This shows how we have managed the transition from unaided vulnerability detection to vastly superhuman detection without requiring extraordinary government intervention. We agree that LLMs bring real improvements to vulnerability detection, but they build on top of decades of tooling that is already far beyond what any human can do unaided. If we have already absorbed the transition to such tools, it is worth asking whether the additional capabilities of language models call for extraordinary interventions.

To be clear, these transitions were not smooth or painless. For a brief period, the attacker-defender balance was completely upset. The idea of viral spread of malware and the resulting asymmetry was unprecedented. 15-year-olds could create devastating cyberattacks that brought down top e-commerce websites and search engines. It is plausible that government intervention to improve cybersecurity could have reduced the harm caused to individuals and businesses during this period.

AI’s use for cyberattacks might once again upset the offense-defense balance, and we don’t think the AI transition will be smooth by default. Many systems that are currently under-defended, including schools, hospitals, power grids, and small government agencies, will be at real risk. Efforts like Project Glasswing and OpenAI’s cybersecurity grant program are important but insufficient on their own. While restricting access to advanced AI systems might be helpful in the short run, it is not a silver bullet in a world where open-weight models are only months behind the most capable closed models.

Addressing AI’s cyberrisks requires investment in resilience. That means AI-assisted red-teaming not just for tech companies, but for schools, hospitals, power grids, small businesses, and government systems that currently lack the capacity for defense. We should also incentivize professional security experts to find and report vulnerabilities (with or without AI use), such as through bug bounties that cover more than just tech company products. These efforts are not automatic; they require investment and planning.

The good news is this work is starting across many different industries. While this requires significant investment, it is still far less burdensome than enforcing a strict nonproliferation regime. The bad news is governments have a lot left to do to truly make the transition painless.

The same analysis applies to biosecurity. AI may lower some information barriers, but bioattacks depend on many downstream steps: procuring materials, accessing specialized equipment, and applying tacit know-how. We can intervene on those downstream steps *now* to increase societal resilience to bioattacks without imposing strong controls on AI development, such as by implementing better screening of synthetic biology orders, using AI to evaluate the riskiness of new compounds, tracking access to dangerous materials, and offensive red-teaming (such as asking trusted experts to attempt to use AI for manufacturing dangerous compounds) to find gaps in these efforts. These help regardless of whether biorisks are from AI.

If Resilience is So Helpful, Why Haven’t We Prioritized it Already?

Importantly, resilience does not require extraordinary government intervention. It only requires us to get our act together on the “normal” process of policymaking and execution.

The problem is we are not great at normal policymaking. We suspect one reason why the resilience approach seems unappealing is that it requires polycentric governance in which many decision makers work harmoniously together. This is a tough sell given that state capacity in the United States has been hobbled by decades of accumulating veto points and creeping proceduralism. As a result, unilateral actions by the executive branch are often seen as the way out for developing and enforcing AI policy.

Investing in resilience requires action and investment by a much wider set of actors than nonproliferation does. For resilience to be effective, the government needs to legislate and allocate funding, collaborate across agencies, build early warning systems, and serve as a resource hub for downstream actors and rapidly disseminate information to help them shore up their defenses. The U.S. federal government isn’t exactly well known for being competent at this set of tasks.

So it shouldn’t be a surprise that when we look at the government’s track record, policy responses for resilience have been underwhelming. In fact, even when there are areas of agreement between proponents of the normal and “abnormal” views on AI, such as requirements for transparency, auditing, and safe harbors for safety research, we are yet to see concrete federal action. (There are some efforts, like the recent letter from representatives urging the federal government to improve coordination on cyberrisks. But for now, it is just a letter with recommendations; it remains to be seen if any action will be taken based on these suggestions.)

It is in this context that extraordinary government actions, such as nonproliferation, look tempting to address AI risks. These are morally satisfying since they primarily impose burdens on the companies that create these risks. They are also tractable, since they only require unilateral action from the Executive, such as invoking the Defense Production Act. The main question is whether we can achieve better outcomes through such actions, or whether we should invest in improving the “normal” process of governance.

While we understand the reasons for taking the former approach, we lean towards the latter. Getting our policy act together is hard, but important—not just for this round of AI policy and misuse risks, but for all future interventions on technological harm, and for the democratic process to work more generally.

On the other hand, if our main defense against AI risks is nonproliferation, a single technical breakthrough that makes models cheaper to train could be enough to cause instability, especially in a world where we don’t also invest in resilience. The damage will be that much greater when the dam eventually breaks.

Acknowledgments. We are grateful to Katy Glenn-Bass for feedback on a draft of the essay and Shira Minsk and for editorial support. This essay is cross-posted on the Knight First Amendment Institute website.

Summary: AI models have started to saturate most major benchmarks. But does that mean they can build and ship a real product, or conduct a scientific experiment end-to-end, or navigate a government bureaucracy? Researchers have started testing AI in such real-world settings. We call these evals “open-world evaluations”. This essay defines open-world evaluations, surveys the lessons learned so far, and lays out best practices for conducting them.

We also introduce CRUX, a collaboration of 17 researchers from academia, government, civil society, and industry that will regularly evaluate frontier AI capabilities through open-world evaluations. In our first experiment, an AI agent built and published an iOS app to the App Store, making just two errors, one of which required manual intervention. This gives us an early indication of potentially useful capabilities and, more importantly, an early warning about the potential for AI-driven app store spam (we disclosed this result to Apple a month before publication).

We hope to conduct similar experiments to surface early warnings across other real-world domains; this will be one of our main empirical projects over the coming year.

The authors are: Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, J.J. Allaire, Rishi Bommasani, Magda Dubois, Gillian Hadfield, Andy Hall, Sara Hooker, Seth Lazar, Steve Newman, Dimitris Papailiopoulos, Shoshannah Tekofsky, Helen Toner, Cozmin Ududec, Arvind Narayanan

How should we track and predict AI capabilities? The AI community’s dominant answer today is benchmarking. For example, METR’s time horizon graph has been used by policy analysts, industry leaders, and organizations researching AI risks to argue that AI capabilities are rapidly increasing.

But benchmarks can both overestimate and underestimate progress. To turn a task into a benchmark, it needs to be precisely specified and automatically verifiable. The catch is that whatever is precise enough to benchmark is also precise enough to optimize for, allowing AI agents to excel at such tasks. On the flip side, low accuracy on benchmarks might result from incidental failures such as encountering CAPTCHA on a website, even if agents are capable of solving the underlying task.

To address these limitations, many researchers are turning to a new kind of evaluation: long, messy, real-world evaluations that go beyond benchmarks. Nicholas Carlini at Anthropic used Claude agents to build a C compiler that could compile the Linux kernel. Anthropic and Andon Labs designed a free-form experiment where Claude was tasked with maintaining a small shop in their office. While benchmarks consist of dozens of tasks evaluated in an automated way, open-world evaluations consist of small samples, often require human intervention, and are evaluated in an open-ended way, such as by analyzing agent logs.

It is easy to dismiss these as unscientific: each such evaluation has a sample size of 1, and they lack standardization and reproducibility. Despite these limitations, we think such evaluations are important for collecting evidence about AI capabilities. They can provide early warnings about emerging capabilities to inform efforts at building societal resilience, help evaluators identify blind spots in existing benchmarks, and give companies a clearer picture of what tasks AI systems could soon carry out, informing strategic decisions about AI. We call them open-world evaluations.

In this essay, we conceptualize open-world evaluations, review past examples to identify best practices and pitfalls in conducting them, and introduce CRUX, a project aimed at regularly conducting new open-world evaluations. Here are our main insights:

• Open-world evaluations are an important emerging class of AI evaluation. As AI systems become more capable, evaluations to elicit frontier capabilities must also increase in complexity. Open-world evaluations are the latest in a long line of evaluations of increasing complexity. We survey 10 prominent open-world evals conducted over the last year to identify best practices and key takeaways.

• CRUX (Collaborative Research for Updating AI eXpectations) is our attempt at systematically conducting open-world evaluations. The team consists of collaborators from government, academia, and non-profits, many of whom have led open-world evaluations, and who have a range of expectations about the future of AI. We aim to provide empirical evidence about the present capabilities of AI systems, even if they are currently costly, and to provide early warnings for capabilities that might soon be widespread. We plan to release new open-world evaluations regularly.

• In our first CRUX experiment, we tasked an AI agent with developing and publishing a simple iOS app to the App Store. Many benchmarks test agents’ ability to write code. But publishing an iOS app involves many other steps: signing the app, publishing a privacy policy on a webpage, filling out Apple’s forms, and taking the app through the review process. We were more interested in whether the agent succeeded at the real-world requirements of publishing the app rather than its ability to write code, so we tasked it with building a simple app and taking it through the iOS App Store submission process.

• The agent was successful after making two errors, one of which required manual intervention (forgetting where the correct credentials were stored and fabricating a fictional phone number for the App Store review process). The process of developing and publishing the app cost about $1,000. The app is now live on the iOS App Store. We think the cost could have been far lower: the app development and submission only cost $25; the vast majority of tokens were spent monitoring the app’s status. We reached out to Apple a month before the publication of this essay to disclose the results of our experiment. App store operators should prepare for and police spam submissions, as they might soon see thousands of applications submitted autonomously using agents.

• How can we improve open-world evaluations? What’s next? To increase the usefulness of open-world evaluations, evaluators should specify what and how much human intervention is allowed, release logs collected while the agent was solving the task, and analyze logs to report what an agent did in the course of solving the task. In future CRUXes, we will evaluate AI R&D automation, AI governance, and many other areas.

Open-world evaluations are an important emerging class of AI evaluation

In this section, we define open-world evaluations and survey the emerging landscape of such evaluations to extract insights about their success and limitations. We discuss areas where open-world evals can overcome some of the blind spots of benchmarks. In our view, as AI systems become more capable, evaluations to elicit frontier capabilities must become more complex; open-world evaluations are the latest in a series of evaluations of increasing complexity. We also discuss the limits of open-world evaluations compared to benchmarks.

We discuss five loose criteria that define open-world evaluations below (see “What are open-world evaluations?”). But it is worth noting that the line between long, complex benchmark tasks and open-world evaluations is blurry. Indeed, many of the evaluations we discuss are sandboxed. We still include them in our list of open-world evals, because they satisfy our other criteria (e.g., Carlini’s C compiler is sandboxed, but involves just one long-running task, human intervention, and qualitative analysis as part of the evaluation).

Benchmarks can both overestimate and underestimate progress

People with drastically different views on AI agree that current AI benchmarks might soon be saturated. Many prominent benchmarks have been saturated in the last two years, and evaluators have raced to release “successor” benchmarks. Many of these updated benchmarks are themselves near saturation.

Does this mean AI systems will soon be capable of solving any task? Not necessarily. Benchmark improvements could indicate real capability gains, but they can also overstate progress: for example, because benchmarks have limited construct validity—they may test accuracy on narrow tasks rather than general ability—and they don’t test how well agents handle the messiness of real-world environments.

On the flip side, benchmarks could also underestimate progress due to challenges in the environment or infrastructure that are only incidental to the capabilities being measured, such as encountering CAPTCHA.

One way to get a fuller picture is to use metrics other than accuracy. For example, in a recent preprint several of us coauthored, we showed that even though agents are improving drastically in terms of capability metrics (such as average accuracy), they have improved much more slowly on metrics that measure reliability. Similarly, many agent solutions to SWE-bench tasks that were judged correct because they pass tests, would nevertheless be rejected by project maintainers.

But this still doesn’t allow us to measure the upper bounds of AI capabilities. Capabilities that are possible today (even if only under favorable conditions) might soon be widespread, and we need to anticipate them before they are. Providing such early warnings gives businesses extra time to capitalize on new opportunities, institutions to build resilience, and policymakers to address risks.

Now let’s discuss in more detail why benchmarks can both overestimate and underestimate capabilities. They can overestimate capabilities because:

Benchmarks resemble tasks amenable to modern reinforcement learning (RL) techniques. To turn a task into a benchmark, it needs to be precisely specified and automatically verifiable. But whatever is precise enough to benchmark is also precise enough to optimize for, and modern RL training runs increasingly resemble the shape of the benchmarks themselves. This makes it easy to saturate any task that can be measured using benchmarks.

This is already the case for leading evaluation platforms like Harbor, which double as RL training platforms. As a result, AI models could be directly trained on data from many prominent benchmarks included on these platforms. Even if benchmarks include held-out test sets, models might be trained on tasks from the training set that look very similar to those they would encounter in the test set. So benchmarks don’t help us understand how well performance generalizes to the real-world.

Benchmarks avoid real-world messiness. Real-world tasks involve underspecified interactions that can’t be fully sandboxed, such as responding to unexpected situations or navigating environments that are open-ended. Benchmarks can take some steps to simulate messy environments, but they can’t fully replicate them.

Benchmarks can also underestimate capabilities, because:

Eliciting frontier capabilities is costly. Running large-scale, long-running experiments is expensive, making it impractical to achieve the sample sizes that benchmarks rely on. Anthropic’s C compiler cost ~$20k; the task we describe running below (developing and publishing an iOS app for CRUX #1) cost ~$1,000. We can’t run such experiments hundreds of times, limiting the budget and complexity for each benchmark task.

Average performance is very different from upper-bound capability elicitation. Trying to run dozens (or 100s) of tasks in a benchmark suite is only necessary to measure average performance. But when we are trying to understand the frontier of what agents can do, the goal shifts to understanding best-case performance: what can agents accomplish when given sufficient resources and support to work around incidental failures? This is necessary for providing early warnings for capabilities that might soon be widespread.

Human intervention can help elicit capability upper bounds. Agents working on real-world tasks could encounter policy refusals, require solving CAPTCHAs, or encounter other infrastructure issues where they get stuck. This could negatively affect their performance. But these failures are only incidental to the capability being measured. If human operators can handle these, it would allow us to elicit upper bounds of capability. Such manual intervention is impractical for 100s of benchmark tasks each time they are run.

As AI capabilities improve, sandboxed evaluations that test AI capabilities in coding, deep research, and customer service require intensely engineered environments to challenge agents and avoid contamination or reward hacking. For example, the performance of agents on web benchmarks has been affected by how often agents encounter CAPTCHA, rather than eliciting the true underlying capabilities of agents. Recent work has highlighted AI agents finding answers online, exploiting bugs in evaluations, and producing code that passes tests but fails to meet standards for production. This highlights the need for a shift towards deeper qualitative evaluations of AI agent performance that can help us better understand failure modes and problem-solving strategies that can be lost in the barrage of average benchmark scores.

But these threats to validity are not straightforward to address. We can conduct qualitative log analysis for benchmarks to address some of these concerns, but even when we find validity issues in benchmark results using log analysis, there’s little we can do about it except releasing an updated benchmark, which could take a few months. Open-world evals allow us to conduct dry runs to fix issues before the real test run, and manually intervene to fix issues found during the evaluation.1

Of course, benchmarks can be helpful despite these limitations.2 And open-world evaluations have their own limitations, which we discuss later in this paper. That said, we hope this list illustrates the systematic blind spots in benchmarking. As AI agents become more capable, the gaps we identify between traditional benchmarking and real-world capability will continue to grow. Success metrics will need to be multi-faceted to capture the diversity of objectives in a given task. Benchmarks will need to incorporate key bottlenecks (human assistance) and messy environments (internet navigation), each of which introduces additional issues of internal and external validity. Open-world evaluations offer an alternative.

What are open-world evaluations?

As evaluation methods have matured alongside AI capabilities, a gradient in evaluation has emerged. At one end are simple, automated, scalable methods that work well for early-stage capabilities. At the other end are richer, more labor-intensive methods that become necessary as capabilities improve and saturate simpler metrics.

As capabilities in a domain improve, evaluations further along the gradient become important to get insights that are complementary to simpler evaluations. We think of this gradient as having roughly five levels so far, each with their strengths and limitations:

1. Q&A benchmarks (e.g., MMLU, GPQA): useful for broad knowledge assessment, but increasingly saturated for frontier models. Often formatted as multiple-choice questions for ease of grading, but as a result have low construct validity, since users rarely interact with models by asking multiple-choice questions.

2. Open-ended chat benchmarks (e.g., WildBench, Arena-hard-auto): capture more nuance, but still limited to single-turn or short interactions.

3. Outcome-only agent benchmarks (e.g., SWE-Bench, WebArena): test agent performance on real tasks, but only measure whether the task was completed, not how. As a result, they have limitations. For example, most passing SWE-Bench solutions are not accepted by repository maintainers.

4. Agent benchmarks with log analysis (e.g., UK AISI transcript analysis, METR Time Horizon): go deeper by examining how agents succeed or fail and uncovering errors and reward hacking by analyzing agent logs. But they still operate in sandboxed environments with predefined tasks.

5. Open-world evaluations: long-horizon tasks in real-world environments, where success can’t be neatly specified or automatically graded. This allows eliciting upper bounds of capabilities. But this comes at the cost of lack of reproducibility and standardization (which are both benefits of benchmarking; see section on Limitations).

Some evaluations blur the categories of this spectrum. For example, OpenAI’s GDPVal, a long-horizon agent benchmark, is designed to be manually graded based on expert’s opinions on quality. This structure is very similar to an open-world evaluation, though they primarily focus on outputs but not on analyzing the logs. At the same time, results on GDPVal are commonly reported using GDPval-AA, which uses automated LLM grading; this evaluation setting resembles an outcome-only agent benchmark.

Open-world evaluations consist of running agents on a small number of long-horizon tasks in real-world settings, and qualitatively evaluating their results using tools for log analysis. These evaluations are complementary to benchmarking and help address many of their limitations. They can also uncover tasks that remain out of reach for current AI systems for future benchmarking efforts.

We provide a rough taxonomy to clarify the boundary of what constitutes open-world evaluations. No single dimension determines whether an experiment qualifies as “open-world”. Instead, it depends on the overall pattern across all of the dimensions below.

• Openness. Is this evaluation in a real-world deployment setting (as opposed to a sandboxed environment)?

• Complexity/length. Does the task require a human days or weeks at a time to complete (as opposed to a few minutes or hours)?

• Number of tasks. Is this a stand-alone task or a small set of tasks (as opposed to a large evaluation suite or benchmark)?

• Human intervention. To help elicit upper-bounds of capabilities, are humans able to intervene when agents hit a hurdle (as opposed to just setting up the environment or resolving setup issues)?

• Method of evaluation. Does the evaluation primarily consist of in-depth log evaluation (as opposed to the result driven by a single average metric)?

When do we call something an evaluation as opposed to simply using agents to accomplish something novel? For example, Anthropic used AI agents to find security vulnerabilities in leading open-source software such as Mozilla Firefox—is this an example of an open-world evaluation? We still consider these open-world evals if the role of the agent is systematically and publicly documented (including the parts carried out by the agent carried and the human experts, and the end result).

The line between complex benchmark tasks and open-world evaluations is also blurry. For example, some evaluations in our list of open-world evals below are sandboxed (e.g., Claude Plays Pokemon and Anthropic’s C compiler were both run in sandboxed environments, but we still consider them open-world evaluations because they consist of a single, complex, long-running task, have human intervention during the evaluation, and are evaluated qualitatively).

The two types of evals are also complementary: we could imagine using open-world evaluations to understand what tasks remain unsolvable by agents as a first step towards building new benchmarks, and also use open-world evals to evaluate agents on messy real-world tasks that are not amenable to being benchmarked.

Finally, we don’t mean to imply that open-world evaluations are categorically more informative for understanding AI progress compared to benchmarks. Indeed, we discuss many limitations of open-world evaluations later in this section. Rather, as AI capabilities increase, open-world evaluations become important as an additional complementary signal of AI capabilities.

An incomplete survey of open-world evaluations

Over the past year, researchers at AI labs, universities, non-profits, and independent groups have begun running open-world evaluations. These share a common structure: give a capable AI agent a hard, real-world task with a long time horizon and observe and analyze its behavior in detail. Some notable examples:

1. Anthropic, Claude Plays Pokemon (Feb 2025). Anthropic launched a Twitch livestream in which Claude 3.7 Sonnet played Pokemon Red. While not a real-world deployment, the experiment was an early example of setting an AI agent up in a relatively open environment compared to typical benchmarks. While the project illustrated progress in AI computer use with minimal scaffolding, the demonstration also clearly illustrated the limitations of early 2025 agents––Sonnet 3.7 remained stuck in a level for nearly 80 hours.3

2. AI Digest, AI Village (April 2025-present). The AI Village gives multiple AI agents their own computer environments and a shared group chat, then tasks them with open-ended real-world goals like fundraising for charity, organizing real in-person events, making word games, and gaining subscribers on Substack. Across different experiments in 2025, the project highlighted persistent failure modes of hallucination, mis-calibration, and unproductive loops, but also documented notable improvements from late 2025 agents on these dimensions.

3. Anthropic/Andon Labs, Project Vend (June 2025-present). Anthropic partnered with Andon Labs to have a scaffold using Claude 3.7 Sonnet (nicknamed “Claudius”) operate a small automated store in their office. The agent managed inventory, set prices, and interacted with customers over several weeks, revealing a rich set of failure modes around manipulation, prioritization, and real-world decision-making. A second phase expanded the experiment to multiple locations, used newer models, and included a red-teaming exercise by the staff at the Wall Street Journal. The agent lost nearly all revenue from poor planning, hallucination, and excessive discounts. Anthropic’s follow-up was more successful, advertising positive profit each week, but the staff at the WSJ were still able to successfully jailbreak “Claudius,” leading it to give all products away for free. Andon Labs recently began a third phase to the project where they gave a Claude-based agent, “Luna,” a three year lease to a brick-and-mortar store in San Francisco and tasked it with making a profit as the “manager,” including hiring human employees, designing the brand, and making product selections.

4. Lin, Cursor browser experiment (Jan 2026). Wilson Lin at Cursor coordinated hundreds of GPT-5.2 agents to build a web browser from scratch, running uninterrupted for one week. The resulting browser (”FastRender”) consisted of over a million lines of Rust with a from-scratch rendering engine. It could render simple websites but was far from production-ready. The project was notable for its exploration of hierarchical multi-agent coordination at scale and the specific failure modes that emerge when agents work on a project for days rather than minutes.

5. Carlini, C compiler (Feb 2026). Nicholas Carlini at Anthropic tasked Claude with building a C compiler from scratch, spending roughly $20k in API costs. The agent produced a working compiler for compiling the Linux kernel and passed a large fraction of standard test suites, revealing detailed information about where agents excel (systematic code generation, test-driven iteration) and where they struggle (complex optimization passes, debugging subtle spec violations).

6. Ho, “How Close is AI to Taking my Job” (Feb 2026). Epoch researcher Anson Ho had Claude Code and ChatGPT Atlas attempt to autonomously complete three challenging work tasks at Epoch: replicating an interactive web interface for a 40 parameter economic model, writing an article in Epoch’s style on AI progress in 2025, and porting an article from Google Docs to Substack and Epoch’s website. The project highlighted the persistent bottlenecks of formatting and hallucinations in successful knowledge-work tasks.

7. Choi, GPT 5.3 Codex Builds a Design Tool (Feb 2026). OpenAI’s Derrick Choi had GPT-5.3 Codex run autonomously for 25 hours, generating 35,000 lines of code, to build a “design tool” from scratch. The post noted that the agent showed impressive planning, memory, and verification processes from the agent, though it did not provide substantive analysis of the capabilities and limitations of the finished product.

8. Faulkner, Next.js Reimplementation (Feb 2026). An engineer at Cloudflare used Claude with OpenCode to release vinext, a reimplementation of the popular frontend web framework Next.js on Vite rather than React. While the initial blogpost advertised coverage of 94% of Next.js for only $1,100 in API costs, follow-up analyses highlighted persistent security limitations and low generalizability to other domains of software given the importance of human-built testing and infrastructure of both Vite and Next.js.

9. Papailiopoulos, “Can You Train a Computer” (March 2026). Dimitris Papailiopoulos and collaborators tested whether Claude Code and OpenAI Codex could train a transformer to function as a general-purpose computer. The experiment included both a fully autonomous round, where both agents failed and reward-hacked solutions, and a human-guided version, where Claude Code succeeded and displayed meaningful generalization, including solving multi-step computations never seen in training.

10. Karpathy, Nanochat Autoresearch (March 2026). Using an existing open-source project, nanochat, for GPT-2 level LLM training, Andrej Karpathy built a simple automation pipeline for AI agents to optimize training in 5 minute increments. The agent has complete autonomy to adjust the architecture, hyperparameters, optimizers, and batch sizes. In a follow-up, Karpathy shared that autoresearch made progress against “Time to GPT-2” (measured with 8xH100 GPUs), dropping this metric by 11% in 2 days.

We plan to collect a running list of such evaluations (and the key takeaways and limitations for each of them) on cruxevals.com. The increase in AI capabilities has allowed people across domains to conduct such evaluations and figure out if AI systems can carry out tasks they are experts on, resulting in growing interest in such evaluations. In just the past week, there have been many significant open-world evaluation releases, including MirrorCode by Adamczewski et al., which tasked agents with reimplementing large programs, a set of automated alignment research case studies by Wen et al., and an exercise using Claude Code to train models to forecast the outcome of the recent Masters golf tournament by Huang.

Limitations of open-world evaluations

While open-world evaluations are helpful at addressing some blind spots in benchmarking, they also suffer from many limitations. There remains a genuine tradeoff between developing better benchmarks and investing in open-world evaluations. Benchmarks give evaluators control over the task and allows evaluations to occur in controlled environments, but are less useful for understanding agents’ performance on open-ended tasks. Open-world evaluations sacrifice sandboxing and evaluator control to improve the construct validity for the task and elicit upper-bound capabilities. Concretely, open-world evals have the following limitations:

Lack of reproducibility and standardization: Benchmarks have been successful because they provide coordinating functions for the AI community. Researchers can develop and test new methods independently, and validate if these methods work well on benchmarks to get the community’s attention. The culture of benchmarking runs so deep that David Donoho called it the “secret sauce“ for the success of the AI/ML community over the last 50 years. Open-world evaluations give up the reproducibility and standardization that made benchmarking so successful.

Hard to compare agents: Benchmarks offer a relative comparison between different models/agents. But since open-world evaluations are often run one or a handful of times, the run-to-run variability might be higher than the difference between different agents’ performance. As a result, open-world evaluations are not useful for comparing the accuracy of different models or agents.

The need for domain expertise: Evaluating whether an agent succeeded in an open-world evaluation can be challenging. Verification of the agent’s work might require deep domain expertise and time, especially if the task is open-ended.

Incomplete recall of log analysis: Even if automated log analysis is used to analyze open-world evaluations, it can never be considered complete. Agent transcripts from long-horizon tasks can run to hundreds of millions of tokens, making thorough human review impractical. And because agent behavior in these tasks is complex, there is no guarantee that a given round of analysis will surface all noteworthy behaviors or errors. This limitation does not apply in the same way to benchmarks, where success criteria are predefined and automatically verified. Releasing logs publicly so that a broader community can examine them is one way to partially mitigate this concern.

Blurry success criteria: Given the potential for human intervention, it is hard to cleanly delineate the agent’s performance from the help given to it by the human.

Non-stationary environments: In open-world evals, agents can interact with open-ended environments such as the internet.4 This makes it hard to make generalizable claims about the agent’s capabilities as opposed to information it can access via the internet (or another open-ended environment). For example, could an agent solve a hard software engineering task because it is genuinely competent at such tasks (and will therefore be able to solve novel software engineering tasks), or because it was able to look up a specific task online (and won’t be able to solve a novel task)? It is also challenging to compare agent performance over time. For example, the internet contains a growing list of implementations or hints that an unsandboxed agent can pull from.

How different stakeholders can use open-world evaluations

Open-world evaluations have limitations, but they can be helpful in addressing the blind spots of benchmarks. We think they can benefit a number of stakeholders:

Policymakers: The diffusion of AI across domains lags behind improvements in capabilities. This allows institutions to adapt as AI systems become more capable. Open-world evaluations can further increase this lead time by providing early warnings of what agents could soon be able to do autonomously and at scale, giving institutions time to build resilience. For example, Anthropic‘s recent work on discovering cybersecurity vulnerabilities using AI could spur efforts to rapidly adopt AI for defensive cybersecurity, especially in critical software infrastructure.

AI evaluators and researchers: Open-world evaluations offer a complementary signal to benchmarks by testing capabilities that are structurally resistant to benchmarking, such as solving messy real-world tasks. Analyzing agent logs allows us to uncover instances of the agent taking shortcuts and reward hacking. On the flip side, they can find areas where agents develop new insights or surpass previous limitations. For example, in our iOS app development evaluation, we identified that the agent modified its approach to be more token efficient, which allowed it to drastically reduce the cost of solving the task, without any input or instruction from our end.

Frontier AI developers: AI developers should actively support and participate in external open-world evaluation efforts by providing access (such as pre-release access to models) as well as safe harbors for third-parties conducting evaluations that might not adhere to developers’ terms of service (such as for evaluating safety). Open-world evaluations by independent third parties could surface findings that internal red teams miss if they optimize for known threat models.

New models are also saturating benchmarks. For example, Anthropic’s Mythos Preview system card notes that the model “saturates many of our most concrete, objectively-scored evaluations,” leaving them with noisier methods for assessing capabilities. Open-world evaluations could offer a complementary way to stress-test models in realistic settings that benchmarks can no longer distinguish.

To foster an ecosystem of open-world evaluations, it is important to develop shared best practices and build toward a cumulative body of evidence about what agents can and can’t do. That is what we are trying to do with a new project called CRUX.

Introducing CRUX: Collaborative Research for Updating AI eXpectations

CRUX is a project for operationalizing open-world evaluations. We aim to conduct open-world evaluations on a regular basis. Each evaluation will involve a long-horizon, real-world task; the implementation of an agent scaffold that could in theory allow agents to solve the task; and detailed analysis of what the agent did in order to solve a task. Our team consists of researchers spanning industry, academia, and non-profits. Many people on the team have conducted open-world evaluations.

In addition to conducting evaluations, we also want to develop mechanisms to provide early warnings for AI capabilities that will soon be widespread. For example, if AI agents can almost autonomously develop and publish apps (our task for the first iteration of CRUX; discussed in the next section), app store operators such as Apple and Google might soon need to update their policies to manage spam submissions.5

This requires eliciting upper-bounds of capabilities—often filling in missing incidental capabilities (such as filling out CAPTCHA) with human input. Open-world evaluations are a good fit for this particular style of evaluation, since they allow us to deeply understand AI systems’ capabilities.

Another advantage of open-world evaluations is that we don’t need to develop a large task suite or design complex environments or sandboxes before conducting evaluations, since each task only needs to be carried out and evaluated a small number of times. This allows us to conduct new evaluations regularly. We plan to design a new CRUX evaluation, analyze results, and publish our analysis on new tasks every 1-2 months.

Our first evaluation tests if AI agents can autonomously develop and publish apps to the iOS App Store; we discuss this in the next section. In future iterations, we plan to expand to a wide range of domains, including tasks on AI R&D automation, AI governance, complex software engineering, and real-world physical tasks.

CRUX #1: Can AI agents autonomously develop and publish an iOS app?

The question of whether AI agents can write software has been extensively studied, both through benchmarks like SWE-Bench and Terminal Bench, and through open-world evaluations like the C compiler and browser experiments discussed earlier in this essay. Agents have shown strong coding capabilities (though questions of code quality and reliability remain unresolved).

Meanwhile, a task that has not been evaluated as closely is whether agents can handle the non-coding aspects of software deployment, such as satisfying platform requirements and interacting with review systems they do not control. For our evaluation, we tasked an agent with building a mobile app from scratch and publishing it on the iOS App Store.

We prompted the agent to develop and publish a simple app to the App Store. We weren’t primarily interested in the agent’s software engineering ability, but rather its ability to interact with Apple’s App Store submission process. This process requires developers to configure signing certificates and provisioning profiles, prepare screenshots and metadata, draft and host a privacy policy at a public URL, fill out compliance questionnaires, and submit the app for review by Apple’s team. Reviewers may reject the app for technical or policy reasons, requiring the developer to diagnose the issue, make changes, and resubmit. This process typically takes several days and involves interacting with systems and reviewers that the developer does not control.

The agent was responsible for every step of the process except those where human involvement is required by policy, such as setting up the Apple Developer account and hitting publish to release the app to the App Store. Specifically, the agent handled writing the code, building the app, preparing metadata, drafting and hosting a privacy policy, submitting it for review, and handling any feedback. (We provided the agent access to a Mac VM, a GitHub account, an Apple developer account, and a Gmail account.)

The success criterion was whether the agent got the app published on the App Store. We logged how many manual interventions the agent needed us to make to solve the task. The agent had the option to ping the team for support; we monitored the agent’s progress once a day. The lower this number, the better the agent performs.

In addition, if agents can do this autonomously (or are close to being able to do so), this serves as an early warning for Apple’s review processes, since agents might soon be able to publish thousands of apps autonomously. The App Store has already seen an increase in the number of published apps, but if agents could develop and publish apps fully autonomously, the number of submissions could increase dramatically.

Our setup for the agent

We used OpenClaw as the agent scaffold with Claude Opus 4.6 and adaptive thinking enabled.6 We chose OpenClaw for this experiment since it is configurable, integrates well with the browser, and natively supports long-running tasks. Given its recent popularity and support for wide-ranging and long-running tasks, we also wanted to evaluate its capabilities as a scaffold, and to potentially compare it to other scaffolds in the future. (Note that we used it as a fairly general scaffold for the task; in fact, we did not make any changes to the default OpenClaw setup beyond prompting it and giving it deeper access to the macOS VM.)

We also wanted to test whether agents struggle with visual reasoning and GUI operation tasks, which might be bottlenecks to solving the task. Based on reports of users’ experiences, OpenClaw was able to successfully resolve some of these concerns, and we wanted to test this ability.

We recognize that OpenClaw has security risks in real-world use. We still chose to use it since we wanted to evaluate the capability frontier, while recognizing that security might be a bottleneck for real-world adoption.

The agent had access to a macOS virtual machine with expansive permissions (sudo, screen visibility, UI control) to perform this task. We logged all of the agent’s actions, reasoning chains, and stored screenshots that the agent took while solving the task.

Addressing evaluation awareness

AI models often recognize they are being evaluated and adjust their behavior. This is known as evaluation awareness. Recent work from Anthropic, Apollo, and OpenAI has shown that frontier models increasingly detect evaluation contexts on their own and sometimes modify their behavior in response.

How should we address evaluation awareness? We think hiding the evaluation context from a frontier model is increasingly a lost cause; models are getting better at detecting test environments with each generation. Since we plan to conduct CRUX evaluations regularly and publish our plans publicly, future agents could simply come across descriptions of our evaluations during the course of a task.

More importantly, concerns about evaluation awareness are most acute when measuring alignment properties, where a model might strategically alter its behavior. For a capability evaluation, if the agent succeeds at the task, it has demonstrated the capability regardless of whether it knew it was being evaluated.

To address evaluation awareness, we explicitly informed the agent that it was being evaluated. This allowed us to communicate the parameters of the evaluation clearly; in particular, that it would be judged on the number of times it required unnecessary human inputs, but that some kinds of inputs (e.g., resolving CAPTCHAs) were completely fine to delegate. We were concerned that without this framing, the agent might avoid asking for help even when doing so was appropriate, which would have caused us to underelicit its performance.

Conducting dry runs

Before a full evaluation run, we conducted two dry runs to understand whether our agent setup worked well, and if not, what we needed to change. This allowed us to discover and solve bugs in the scaffold. Notably, these dry runs did not involve any interaction with Apple’s App Store submission or review processes.

Setting up the OpenClaw agent such that it had all permissions to be able to autonomously develop apps also required inputs from our end. We estimate it took us eight person-hours of time and about $50 in API cost to set up the agent scaffold; this included configuring the virtual machine to ensure the agent had full control to take any actions; setting up logging to monitor the agent’s work; and configuring an email account, GitHub account, and Apple developer account for the agent to use and access.

This effort might be a bottleneck for agents to autonomously develop apps. But from the perspective of a potential spammer, it only needs to be carried out once. Spending a few person-hours and $50 for setting up the pipeline is not likely to be a bottleneck for spammers looking to submit thousands of apps to the App Store.

In our dry runs, the agent primarily used the command line and the browser. It used the command line to generate code, build the app, and prepare it for submission. It used the browser to log into App Store Connect, access certificates, and fill out forms. When command-line commands hung because of requests for permission, it was able to take screenshots and simulate mouse clicks, for example to click “Allow” to grant itself permissions.

The final evaluation

After two dry runs, we started the full evaluation. The agent took 45 minutes to develop a simple app for breathing exercises. This included developing the app, publishing a privacy policy using GitHub Pages, filling out the App Store review forms, and submitting the app for review.

We set up the agent to keep checking the app’s status every 5 minutes after the app was sent for review. But it took 10 days before the app was approved. It is now live on the App Store. (To comply with Apple’s policies, the agent needed approval from our team before publishing the app.)

The agent required one unnecessary manual intervention7: it could not locate the credentials we had previously given it to access the Apple developer account.8 It also fabricated the phone number submitted for Apple’s review process, using a fictional number instead of asking us for the correct phone number; the App Store review went through despite this error.9 This alerted us to the need for proactive monitoring of agent actions to prevent such unintended actions, which we plan to implement for future CRUX evaluations. The final app functions well, though it contains a toggle for sound that doesn’t work. The agent also produced a screenshot for the App Store listing with visible formatting errors.

The agent was eventually successful in publishing the app, at a total cost of about $1,000. The development and submission of the app cost just about $25; the vast majority of the tokens were spent looking for updates to verify if the app had been successfully reviewed. We think the total cost could have been dramatically lower if we optimized the scaffold for efficiency, such as by waking the agent less frequently to check the app’s status, but in this evaluation we erred on the side of a higher budget.10

In short, the agent couldn’t completely automate the task, but it was extremely close to being able to do so.11 As a result, we notified Apple’s product security team of our experiment four weeks before publishing the results, since we thought some version of responsible disclosure was warranted; spammers could soon submit thousands of apps to the iOS App Store using agents.

Lessons for open-world evaluations

Through running CRUX and studying the growing body of similar efforts, we have begun to identify what makes an open-world evaluation informative. We expect these lessons to evolve over time. But we think they are worth sharing now, because there is growing interest in these evaluations, and developing shared evaluation norms can help.

Be specific about what you are measuring and what it implies. One reason why people had sharply diverging opinions about Anthropic’s C compiler project or Cursor’s browser was that these projects did not clearly specify the target of their measurement. The results were very impressive from the perspective of measuring if agents productively work for long periods of time on well-defined tasks. But they were perhaps less impressive from the perspective of developing software that is directly usable by and useful to end users; the GitHub issues for both projects highlight core technical complaints, such as failing to compile “hello world” out of the box, or welcome page hangs with the standard build script. Had the authors been clear that they were trying to measure the former, and not the latter, it could have clarified the public discussion on these projects. (This is not to criticize the authors of these pieces. They were one of the first to conduct open-world evaluations, and it is hard to know how people will react a priori.)

For example, writing software has many non-functional requirements such as quality, reliability, maintainability etc. that developers might sacrifice when shipping apps autonomously coded using AI agents. Many concerns about Cursor’s browser and Anthropic’s compiler were actually concerns about these hard-to-specify properties not being satisfied by applications developed using agents.

Design the task so that human intervention is straightforward and well-documented. In real-world tasks, agents will sometimes need help, such as for navigating policy refusals, CAPTCHAs, or infrastructure failures. While traditional benchmarks cannot accommodate human-in-the-loop interventions, for open-world evaluations, such inputs are helpful in ensuring we measure upper-bounds of capabilities. This requires documenting precisely when, why, and how humans step in, so that the degree of autonomy can be assessed clearly.

Invest in log analysis. The logs from an agent attempting a complex task contain far more information than the binary outcome. This allows us to uncover insights about agent behavior that would be invisible from outcomes alone. For example, how did the agent choose to decompose the problem? Did it get stuck at any point, and, if so, how did it self-correct? Did it search through solution paths in a principled or haphazard way? Which aspects of the task proved most challenging? Did the agent misrepresent any aspect of its outputs and progress?

Consider complementing log analysis with real-time monitoring. Post-hoc log analysis is valuable, but it is not sufficient on its own to catch all unintended agent actions. In previous open-world evaluations, agents operating with substantial autonomy have sometimes taken actions that were difficult for human reviewers to detect after the fact. For example, in many AI experiments conducted by the AI Village, agents took unintended actions, such as attempting to send hundreds of unsolicited emails. In our own evaluation, the agent fabricated a fictional phone number that went undetected until a later round of review. Automated real-time monitoring, for example a separate agent that continuously reviews the primary agent’s actions and flags anomalies or errors as they occur, could serve as a valuable complement to human review.

Conduct dry runs before the open-world experiment. Testing the agent scaffold, the evaluation criteria, and the infrastructure before running the full evaluation is helpful to uncover hidden assumptions about the task or errors in the scaffold. Our dry runs for the iOS app development task uncovered many issues with our scaffold before we began the real attempt.

Measure cost. For many tasks, capabilities continue improving with increased budgets. Developers should treat cost measurement as a first-rate goal for conducting open-world evals, and report their findings alongside the budget they used. Even if it is not possible to make general claims about capability upper bounds, if measuring partial progress is viable, it can be helpful to get a sense of whether increasing the budget helps advance progress towards task completion.

Release logs. While open-world evaluations lack reproducibility, collecting and releasing logs to a broad community can help. For example, external researchers can add to analyses of how well agents performed or where they failed, and also verify the results.

We plan to conduct new CRUX evaluations regularly. In future CRUXes, we expect to conduct evaluations on a wide range of topics, including AI R&D tasks, AI governance, video generation, and more challenging software engineering tasks. We plan to keep cruxevals.com updated with our team's efforts and those of the broader community on open-world evaluations.

Author contributions and acknowledgments

Core team: Sayash Kapoor and Arvind Narayanan conceptualized the project and designed the first evaluation task. Andrew Schwartz led the agent development and executed the evaluation. Peter Kirgis led the log analysis and literature review of open-world evaluations. Stephan Rabanser offered feedback and inputs into the task design, essay text, and our analysis and interpretation of the results. Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, and Arvind Narayanan drafted the essay.

Collaborators: Rishi Bommasani offered feedback and inputs into the task design, essay text, and our analysis and interpretation of the results. J.J. Allaire, Magda Dubois, Gillian Hadfield, Andy Hall, Sara Hooker, Seth Lazar, Steve Newman, Dimitris Papailiopoulos, Shoshannah Tekofsky, Helen Toner, and Cozmin Ududec offered feedback and inputs into the essay text and our analysis and interpretation of the results.

Acknowledgments: Nicholas Carlini provided feedback on the task design and agent setup. Ryan Greenblatt, Daniel Kokotajlo, and Ajeya Cotra participated in online and in-person conversations that informed the project.

Funding. We are grateful to Coefficient Giving, Schmidt Sciences, and the Princeton AI Lab for funding to support this project.

Suppose you hear about a new AI agent for improving productivity — by making purchases, or writing code, or sending emails, or handling a customer on your behalf. Should you trust it? Can the agent do the job reliably enough? After all, there are many horror stories of agents going wrong.

Surprisingly, even though the lack of reliability of AI agents is well known, right now the AI industry doesn’t have good tools for measuring reliability, or even a good definition of reliability.

Arvind and Sayash have long been thinking about this. Last fall, we were joined by postdoctoral researcher Stephan Rabanser, whose PhD looked at the reliability question in simpler, more traditional AI systems. We recruited a few other independent researchers, and have released what we hope is a comprehensive measurement of reliability. Our draft paper is called Towards a Science of AI Agent Reliability.

We borrowed insights from many other fields, such as nuclear and aviation safety. We were able to decompose reliability into 12 different dimensions. Evaluating 14 models on two complementary benchmarks, we found that nearly two years of rapid capability progress have produced only modest reliability gains. See our interactive dashboard here.

While our findings are tentative at this stage, we hope they can help explain the puzzlement among many in the industry as to why the economic impacts of AI agents have been gradual, even though they are crushing capability benchmarks.1 To help the community track reliability systematically, we plan to launch an AI agent “reliability index”. We hope this will stimulate researchers and industry to invest effort into improving reliability.

Table of Contents

1. Accuracy isn’t enough: four dimensions of reliability

2. Capability gains are rapid, but improvements in reliability are modest

3. Why we could be wrong

4. What should deployers do differently?

5. What should researchers and developers do differently?

6. What do our findings mean for AI progress?

7. Further reading

Accuracy isn’t enough: four dimensions of reliability

When we consider a coworker to be reliable, we don’t just mean that they get things right most of the time. We mean something richer:

1. They get it right consistently, not right today and wrong tomorrow on the same thing (Consistency)

2. They don’t fall apart when conditions aren’t perfect (Robustness)

3. They tell you when they’re unsure rather than confidently guessing (Calibration)

4. When they do mess up, their mistakes are more likely to be fixable than catastrophic (Safety)

Unfortunately, AI agents are evaluated based on a single number, the average success rate at the task. That number has been going up quickly on many tasks over the last two years, which is why there’s so much excitement about deploying agents.

Safety-critical engineering fields (aviation, nuclear, automotive) figured out decades ago that reliability is not the same as average performance. These fields independently converged on the above four dimensions: consistency, robustness, predictability, and safety (the frequency and severity of failures).

For example, nuclear reactor protection systems must respond identically every time conditions warrant shutdown. Automotive safety testing evaluates responses to sensor failures and adverse weather. Nuclear risk assessment models thousands of failure modes and quantifies their probabilities. Aviation targets one catastrophic error per billion flight hours.

Capability gains are rapid, but improvements in reliability are modest

We refined and decomposed these four high-level dimensions into twelve metrics. We then tested agents based on 14 models from OpenAI, Google, and Anthropic, spanning 18 months of releases. We looked at two complementary benchmarks: a general assistant benchmark (GAIA) and a customer service simulation benchmark (TauBench). We ran each task five times, with instructions paraphrased. We injected faults in the tools and environment to measure robustness to such failures, and elicited the agents’ confidence at having solved the task to measure calibration. In total, we executed 500 overall benchmark runs.

We found that reliability has improved only modestly over 18 months, while accuracy improved substantially. All three major providers cluster together, so this appears to be an industry-wide limitation (though there are some cases where Anthropic’s models outperform OpenAI’s and Google’s).

More specifically, we measured the following criteria:

• Consistency: Agents that can solve a task often fail on repeated attempts under identical conditions. Many models have trouble giving a consistent answer, with outcome consistency scores ranging from 30% to 75% across the board.

• Robustness: Most models handle genuine technical failures (server crashes, API timeouts) gracefully. But if we rephrase the instructions with the same semantic meaning, performance drops substantially.

• Predictability: Agents are not good at knowing when they’re wrong. This is the weakest dimension across the board. When agents report confidence, it often carries little signal. On one benchmark, most models couldn’t distinguish their correct predictions from incorrect ones better than chance.

• Safety: Recent models are noticeably better at avoiding constraint violations, though financial errors, such as incorrect charges, remain a common failure mode. We use safety narrowly to mean bounded harm when failures occur, not broader concerns like alignment. We are still iterating on how we measure safety, so we report it separately from the aggregate reliability score.

• Impact of scaling: Bigger models aren’t uniformly more reliable. Scaling up improves some dimensions (calibration, robustness) but can hurt consistency. Larger models with richer behavioral repertoires sometimes show more run-to-run variability.

Why we could be wrong

Our view is that reliability lags capability, and that reliability will remain a barrier to deployment unless researchers and developers focus effort on improving reliability as a separate dimension from accuracy. There are three reasons why we could be wrong.

First, there is some subjectivity in our dimensions and metrics. We have tried to minimize this by grounding our analysis in existing engineering fields. And we finalized our list of metrics before performing experiments, in order to prevent our hypothesis about slow reliability progress from influencing our selection of metrics. Still, we welcome other researchers to suggest alternative ways to measure reliability, and emphasize that our findings are tentative at this stage.

Second, maybe reliability won’t matter if accuracy gets high enough. Our metrics are carefully crafted so that accuracy gains don’t automatically lead to reliability gains. Broadly speaking, accuracy is about the rate of failures while reliability is about the nature of failures. But if an agent is accurate 99% of the time, maybe we can tolerate 1% error even if it is completely unpredictable. We disagree. Our view is that for autonomous operation in high-stakes contexts, we need 3-5 “nines” of performance — 99.9% to 99.999% accuracy — in order for reliability to become a non-issue, and we don’t think LLM-based agents are on track to reach such a threshold. But time will tell.

Third, reliability progress being slower than accuracy doesn’t necessarily mean that it is slow in absolute terms. If we project the current linear trend forward, agents will reach 100% reliability in just three years! We don’t think a linear model makes sense, in part because we expect each order of magnitude decrease in unreliability (1-reliability) to be as hard as the previous one. That is, we expect the jump from 90 to 99% reliability to be about as hard as the jump from 99 to 99.9% reliability, and so on. But again, we just have to wait and see.

Suppose we’re right. There are important implications for deployers, researchers & developers, and for those tracking the pace of AI progress. Let’s discuss each in turn.

What should deployers do differently?

Clearly distinguish automation from augmentation. A coding assistant that occasionally suggests wrong variable names is annoying; an autonomous agent managing an industrial plant yielding highly variable output is unacceptable. The difference is whether the agent is used to augment a human’s creativity or directly make decisions. Augmentation tools (copilots, search assistants) get a reliability “discount” because someone reviews the output. In some augmentation use-cases, reliability might actually be undesirable. For example, a creative writing assistant that produces the same story every time would be terrible at its job.

Incidentally, how well agents collaborate with humans is woefully under-theorized and under-measured. There is some early work on evaluating human-LM interaction, but both efforts predate autonomous agents, and we are not aware of any equivalent work studying how agents collaborate with humans over multi-step tasks. Uplift studies offer one useful lens, but a broader agent-focused effort is overdue.

Consider reliability for making release decisions. For automation tools (unattended workflows, customer-facing bots), reliability is non-negotiable. Deployers should consider requiring reliability thresholds before moving from sandbox to production, the way aviation requires certification before service. There are many other practices to borrow from such domains, such as building an incident-reporting culture around agent failures.

While the metrics we have identified are broadly useful as a starting point for understanding reliability, deployers should build their own internal evaluations tailored to their specific context and datasets.

What should researchers and developers do differently?

Benchmarks drive progress in AI. A year and a half ago, our paper AI Agents that Matter showed there was a big gap between what agent benchmarks measured and what matters in practice. Our new paper shows that the gap persists. To fix this disconnect, both AI evaluation and AI development practices need to change course.

Measure reliability. The current approach of running a benchmark once and reporting the accuracy number is a shallow, superficial performance measure. It is comparable to stress-testing a car once in perfect weather and declaring it safe if it passes. When evaluating agents, we need to test them for multiple runs (testing for variance in outcomes), under different conditions (evaluating adaptability), and on an ongoing basis (re-test as models and environments change). We call for reporting reliability profiles alongside accuracy, not instead of it.

Understand and improve reliability. Our experiments suggest that consistency and predictability are the biggest gaps preventing models from being more reliable. Agent developers should consider improving these weak points explicitly, possibly via targeted optimization or improved scaffolding. In particular, agents should be able to recognize when they are likely to fail and say so, and recover gracefully when they do fail. More speculative ideas include agents that explore different strategies during development but follow a consistent execution plan when deployed, rather than solving the same task differently each time, simultaneously delivering the best of agents and workflows.

What do our findings mean for AI progress?

The capability-reliability gap could be one reason why we are not seeing any of the rapid labor-market effects that Artificial General Intelligence has been predicted to bring about. It is not the only one. A recent UK AISI report identified six barriers to AGI. However, the report discussed each at a high level.

Our paper can be seen as putting flesh on the bones of one of these barriers — reliability. None of the four dimensions we identify can be considered solved at this point in time, and only two of the individual metrics, shown in green in the figure, can be considered (tentatively) solved.

There is much work to be done in fleshing out the other barriers to AGI, analogous to our analysis of reliability. Our hunch is that this will reveal many other dimensions and metrics on which progress has been slow. The gazillion-dollar question is whether agents will get better across the board through general methods such as inference scaling and reinforcement learning, or whether painstaking work will be required to improve individual dimensions of reliability, adaptability, originality, and so on.

Further reading

• The full 66-page paper is available on arXiv and includes definitions of the metrics we propose, details on our experimental setup, as well as an extended discussion on our recommendations and proposed future work. The authors are Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Alongside the paper, we also provide an interactive dashboard of our results to enable practitioners to drill down into individual models, benchmarks, and reliability dimensions. The code to reproduce our experiments is available on GitHub.

• This paper is part of our broader effort to advance the Science of AI Agent Evaluation. Key publications include AI Agents that Matter and Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation, recently accepted to ICLR 2026.

• Our paper can be seen as building on work that aims to extend the scope of what we measure through benchmarks. For example, HELM introduced 7 metrics, including calibration and robustness, and used these to evaluate dozens of language models on prominent benchmarks.

The essay is co-authored with Justin Curl, a third-year at Harvard Law School. Previously, he was a Schwarzman Scholar at Tsinghua University and earned a degree in Computer Science from Princeton, where we first collaborated. You can find more of his writing on AI and the law here.

Many AI leaders believe the technology will transform knowledge work. OpenAI CEO Sam Altman predicts AI systems that are “smarter than humans by 2030,”1 while Anthropic CEO Dario Amodei analogizes future AI models to a “country of geniuses in a data center.”2

Researchers identify legal services as especially vulnerable to disruption by AI.3 And since GPT-4 passed the bar exam,4 much of the profession seems to agree. Law schools have begun incorporating AI into their curricula5 and partnering with AI-focused legal-tech companies to prepare future lawyers for a changing profession.6 One prominent lawyer has argued AI can already replace law clerks and oral argument.7 Another predicts AI could “replace traditional lawyers by 2035.”8

This excitement about AI comes at a time when legal services are expensive. Millions of individuals are priced out of legal assistance,9 while corporate legal fees are increasing steadily, with hourly rates for partners at large law firms now exceeding $2,300.10 Unsurprisingly, many observers see the potential for AI to make legal services more accessible by delivering outcomes at lower costs.11

Our central claim is that advanced AI will not, by default, help consumers achieve their desired legal outcomes at lower costs. We examine the bottlenecks12 that stand between AI capability advances and the positive transformation of the practice of law that some envision. For AI to usher in a world of abundant legal services, the profession must address three bottlenecks: regulatory barriers, adversarial dynamics, and human involvement.

First, unauthorized practice of law (UPL) regulations may limit AI use by consumers (and to some extent lawyers). These laws prohibit nonlawyers from performing legal work.13 Individuals and organizations can face steep fines and criminal liability if courts conclude their systems cross into practicing law,14 forcing would-be providers to either limit their AI tools’ functionality in legal domains or risk enforcement actions. Entity-based regulations—which restrict who can own equity in businesses that provide legal services—restrict how legal services are offered, again limiting how AI is used by lawyers and consumers. Without reforms, if consumers cannot access AI capabilities or lawyers are not incentivized to use AI well, AI will not help people accomplish their legal goals, regardless of how advanced it becomes.

Second, even if AI is effectively and widely adopted, the American15 legal system’s adversarial structure can prevent advanced AI from lowering the cost of achieving clients’ outcomes.16 Because legal outcomes often depend on relative rather than absolute quality, when both parties become more productive, the competitive equilibrium simply shifts upward.17 In a world with advanced AI, achieving the same result—like settling favorably or prevailing at trial—would require a greater quantity and quality of legal work. So even as productivity increases and cost per legal task falls, parties are locked into an arms race of increasing amounts of legal work required to reach the same outcome.

As a historical analogy, digitization could have reduced discovery costs by making document review much easier.18 But litigators operating within litigation’s adversarial framework exploited the surge in digital documents to drive up costs for their opponents, leaving total litigation costs high.19

Though less explicitly adversarial, transactional work (like contract negotiation) can exhibit similar dynamics: Lawyers compete to control disclosures and outmaneuver opposing counsel when drafting and negotiating agreements.20 Of course, some legal outcomes (like effective estate planning) do not depend on adversarial processes, and this bottleneck would not apply to them.

The third and final bottleneck we discuss is human involvement in legal work. In a world where AI gains outpace increases in the volume of legal work, our desire for human beings to adjudicate cases and understand the contracts they sign is a final bottleneck. In litigation, if AI enables a flood of legal work, judges will likely respond by taking longer to resolve disputes (delaying outcomes) or delegate more to assistants (lowering adjudication quality).21 And with transactional work, even if AI is drafting entire contracts, human lawyers will still need time to understand what these provisions mean for an organization’s interests. The speed of human decision-makers (whether judges, lawyers, or clients) places an upper limit on how much AI can accelerate legal processes without sacrificing human involvement. See Figure 1 below for a visual description of the three bottlenecks and our argument.

This report applies the “AI as Normal Technology” framework to a specific domain: the legal industry.22 This framework is fundamentally about agency: Rather than treating AI’s trajectory as predetermined by capability advances, it directs attention to the social and organizational bottlenecks between what AI can do and the impact it has on the world. In our analysis of the practice of law, diffusion will likely be slow. Better models have not yet translated into more reliable legal products because adapting workflows to leverage AI and teaching users takes time.23

We argue that the end state of AI diffusion can look very different depending on the institutional response. Some pathways lead to genuine improvements in access and efficiency, while others simply make producing legal work (outputs) cheaper without making it easier to achieve the results clients want (outcomes). For AI to deliver better outcomes for consumers, the legal industry must enact reforms addressing the bottlenecks. Otherwise, we risk a future in which legal work becomes more abundant, but legal outcomes remain expensive and inaccessible.

We proceed in three sections. The first section explains why legal services are so expensive. The second section aims to convince readers that AI won’t automatically deliver legal outcomes at lower costs. And the third section offers recommendations for addressing these bottlenecks based on existing proposals for legal reform and illustrates how drastically AI’s impact could differ based on the legal profession’s response.

Table of Contents

1. Why Legal Services Are So Expensive

2. Why AI Won’t Help by Default

   1. Regulatory Barriers

   2. Adversarial Dynamics

   3. Human Oversight

3. Institutional Reforms

   1. Reforming Professional Regulation

   2. Reforming Adjudication

   3. The Evolving Role of Lawyers

4. Conclusion

Why Legal Services Are So Expensive

Three structural factors help explain why legal services are so expensive:24 Evaluating their quality is difficult, their value is often relative, and professional regulations limit competition from alternative business models.

First, unlike a meal at a restaurant, where it’s easy to assess quality, legal services are “credence goods,” which means their quality is difficult to evaluate even with hindsight.25 The final outcome in a case reflects the cumulative effect of many smaller decisions, so it can be very hard, even for other lawyers, to evaluate whether legal services were provided effectively.26 How clear was the law on that issue? Did the client reach the desired outcome because of or in spite of the lawyer’s skill? Which decisions actually contributed to that success? This evaluation difficulty forces consumers to rely on proxies for quality (e.g., the prestige of a lawyer’s law school or judicial clerkships) when choosing between legal service providers, making it hard for traditional market mechanisms to work.

Second, the value of legal services is relative.27 Because “the American litigation system is a thoroughgoing adversarial” one, what matters to the lawsuit’s outcome often isn’t how good your lawyers are in absolute terms, but whether they’re better than the other side’s.28 Other kinds of legal work, such as drafting contracts and agreements (often called transactional work), can also be adversarial as lawyers try to “outfox” each other in terms of what is disclosed in negotiations and the language of a contract itself.29

These dynamics have kick-started an arms race for legal talent, driving up costs at the top end of the market, which serves corporate clients and is often called “BigLaw.”30 In 2024, the median partner at large law firms charged $1,050 per hour, with some commanding over $2,300.31 That’s up 5.1 percent from 2023, which was itself up 5.4 percent from 2022.32 Fortune 200 companies reported that their average litigation costs in cases exceeding $250,000 in legal fees had nearly doubled over eight years, climbing from $66 million per company in 2000 to $115 million in 2008.33 In the patent field, a 2017 survey found that patent cases worth less than $1 million typically cost $1 million to litigate ($500,000 per side).34

Third, the profession’s regulatory framework, designed with consumer protection in mind, has created its own complications. Two types of regulations are often the focus of reform: unauthorized practice of law (UPL) and law firm ownership regulations.35

UPL laws make it illegal (in some jurisdictions a felony) for unlicensed attorneys to apply legal knowledge to specific circumstances.36 An unfortunate effect is to make it more expensive to offer basic legal assistance in contexts requiring little legal expertise.

Most states have regulations limiting who may share in legal fees. Gillian Hadfield argues that these rules promote a business model that creates inefficiency for small firms.37 These firms serve individuals and small businesses and are sometimes called the “PeopleLaw” sector.38 She cites a 2017 Clio study of forty thousand customers: In an average eight-hour workday, lawyers engaged in billable work for only 2.3 hours, billed 1.9 hours, and collected payment for just 1.6 hours.39 So although clients paid an average of $260 per hour, lawyers effectively received $25–40 per hour because the rest of their time was spent finding clients, managing administrative tasks, and collecting payments. These regulations require lawyers to serve clients through partnerships fully owned and financed by lawyers. They deter alternative models that involve large-scale businesses with centralized billing, customer service, marketing, and administrative functions, which could leverage economies of scale to deliver legal services at $30–50 per hour instead of $260.40

Importantly, none of the sources of market dysfunction are intrinsic to legal services. They reflect choices about procedure, pricing, and professional governance. While reform may be politically difficult or costly, the outlook is dim without it. And contrary to what some might hope, AI will not automatically make legal services cheaper, as we discuss next.

Why AI Won’t Help by Default

Regulatory Barriers

More legal assistance would be valuable in the debt collection context. From 1993 to 2013, the number of debt collection lawsuits grew from 1.7 million to about 4 million.41 In Michigan, these lawsuits made up 37 percent of all civil district court case filings by 2019.42 The trend is similar in Texas: “Debt claims more than doubled from 2014 to 2018, accounting for 30% of the state’s civil caseload by the end of that five-year period.”43 More than 70 percent of debt collection defendants lose by default for failing to respond, even though many cases are “meritless suits” and responding is not complicated.44

New York has created a form for responding to debt collection lawsuits by checking some boxes.45 This form, however, includes questions difficult for nonlawyers to understand, such as whether someone would like to invoke the doctrine of “laches.” Recognizing this difficulty, the nonprofit Upsolve began training volunteers to offer assistance. Concerned that this might violate New York’s UPL rules, Upsolve sought an injunction declaring this basic assistance was protected by the First Amendment. A federal judge agreed.46 But New York appealed to the U.S. Court of Appeals for the Second Circuit, which invalidated the injunction, concluding that the lower court applied the wrong First Amendment test (so Upsolve was no longer protected).47

It’s easy to see how an AI system could help here. A nonprofit could provide access to a tool customized for debt collection suits. Or individuals could directly ask general-purpose tools like ChatGPT, Claude, or Gemini for relevant information. Despite this potential, organizations risk violating UPL laws whenever their AI tools complete “tasks that require legal judgment or expertise.”48 The New York Bar Association, concerned that the shortcomings of current AI models would harm consumers, has warned that “AI-powered chat bots now hover on the line of unauthorized practice of law.”49 While some legal researchers disagree because AI is not a “person” capable of exercising legal judgment50 or because AI systems simply “provide information to users, similar to paper guides about court procedure,”51 all authors cited in this paragraph agree that the status of AI tools under UPL laws is currently unclear.

LegalZoom’s history of lawsuits illustrates how UPL regulations can deter innovation in the delivery of legal services.52 LegalZoom automates rote tasks like preparing documents for trademark filings and has been plagued by UPL lawsuits for years.53 In 2011, private individuals who had purchased the company’s services sued in Missouri, alleging LegalZoom was engaged in UPL because it claimed to “take[] over once a consumer answer[ed] a few simple online questions.”54 After the court denied its motion to dismiss the case, LegalZoom agreed to compensate plaintiffs and modify its business model. In 2015, the North Carolina State Bar won a consent judgment requiring LegalZoom to conform to certain conditions.55 Trademark lawyers in California advanced similar theories in a 2017 suit against LegalZoom’s trademark-filing product.56 And in 2024, a New Jersey plaintiff brought a class action alleging UPL violations.57

While AI’s legality remains in doubt, the threat of UPL liability can inhibit its adoption. Without reform, developers risk fines and criminal liability if their AI systems provide legal advice. Organizations may simply be unwilling to provide access to users, especially those who cannot afford to compensate a developer for the risk of UPL liability. Separately, entity regulations that restrict financing for AI legal startups can deter the kinds of operational experimentation helpful for delivering legal services at lower costs. Overall, if regulatory barriers prevent consumers from effectively accessing AI capabilities, it will not translate into better legal outcomes for clients at lower costs.

That said, AI could reduce the costs of legal services for reasons unrelated to its ability to perform legal tasks. As the 2017 Clio survey mentioned above found, nonlegal work consumes a large percentage of lawyers’ time at the low end of the market. If advanced AI helps find and communicate with clients, manage administrative tasks, and handle payments, it could free up these lawyers to spend more time on legal work.

Adversarial Dynamics

Even in a world in which AI increases lawyers’ productivity and completes legal tasks, it might not lower the costs of legal services. To see why, it is crucial to distinguish inputs and outputs from outcomes. Inputs are what goes into legal work: employee talent, billable hours, and technological tools. Outputs are what legal work produces: contracts drafted, motions filed, and briefs written. Outcomes are what clients actually care about: disputes resolved, deals closed, and rights protected.

Consumers purchase legal services to achieve specific outcomes. Inputs and outputs can indirectly lead to those outcomes because more hours worked and more legal tasks help clients get the outcomes they want. But in a zero-sum context where the value of legal services is relative, if both sides increase their outputs, the advantages to either side of doing so can be limited.

Instead, AI might simply raise the inputs and outputs required to reach the same outcome, with productivity gains absorbed by greater production. The billable hours model, in which lawyers are paid and promoted based on inputs, only reinforces these dynamics: More hours worked drafting motions and reviewing documents translates into greater revenue for legal firms without necessarily improving outcomes.

Litigation

One response is that these arms races actually create value by increasing the quantity and quality of legal work. However, clients sometimes achieve their desired outcomes (like settling a case or dismissing a lawsuit) by imposing greater costs on the other side instead of improving the quality of their legal arguments or evidence. It is a “core premise of litigation economics”58 that “all things being equal, the party facing higher costs will settle on terms more favorable to the party facing lower costs.”59 And even where quality does improve, it’s uncertain that the benefits of higher quality legal work (like helping courts reach the “right” answer more often) outweigh the costs of more legal work (like overwhelming judges with cases).

Earlier technological shifts cast doubt on whether America’s “adversarial legalism” can translate productivity gains into more affordable legal services.60

Discovery is a cornerstone of American litigation that often determines whether cases settle or go to trial. In discovery, parties share information “to identify material facts that prove or disprove a claim.”61 It operates through an adversarial exchange: One party sends a discovery request; the other searches its records and decides which documents are responsive and which are protected by privilege.

Discovery was conceived of as a cooperative process during which lawyers could share information to facilitate settlement and avoid trial.62 Yet two characteristics of discovery make it vulnerable to abuse. First, the party holding the documents (the reviewing party) knows what’s in them but seeks to share as little helpful information with the requesting party as possible to minimize legal risk. Second, the reviewing party, because they bear the responsibility and costs for producing documents, must essentially act as their adversary’s agent.

Each side can leverage these features to impose costs on the other. A requesting party, through excessive requests, can compel their adversary to review more documents for confidentiality and relevance. And a reviewing party, through excessive production, can bury relevant information in mountains of extraneous material, forcing the opposing side to spend more time on review. This can create an arms race of overrequesting and oversharing, as each side drives up costs for the other to pressure them to settle on more favorable terms. The billable hours model again reinforces this behavior, with more review generating more billable hours.

These adversarial incentives can be powerful. Judge Frank Easterbrook opened a well-known article, “Discovery as Abuse,” by analogizing the process to “nuclear war.”63 Charles Yablon described how one side made life difficult for opposing counsel: It printed documents on dark red, foul-smelling paper so that their contents would be nearly illegible and the attorneys would get nauseous reviewing them.64 As Arthur Miller aptly observed, discovery’s key defect was believing “that adversarial tigers would behave like accommodating pussycats throughout the discovery period, saving their combative energies for trial.”

Digitization might have pushed discovery costs in either direction.65 Better search capabilities meant attorneys could review documents more efficiently, driving down costs while increasing the relevance of information shared. Yet the explosion of digital information increased what parties might need to search through during discovery, creating more opportunities for discovery abuse.

The empirical evidence on discovery is limited, so making causal claims about digitization’s impact is difficult.66 The available evidence, however, suggests it remains expensive. Discovery accounts for roughly “one-third to one-half of all litigation costs” when used.67 Fortune 200 companies reported that in cases with over $250,000 in legal fees, which are typically the kinds of complex litigation that require discovery, average litigation costs nearly doubled from $66 million per company in 2000 to $115 million in 2008.68 These figures align with evidence of over-requesting and oversharing: One trade association for civil defense lawyers estimated that for every page eventually shown at trial, meaning it’s relevant and reliable enough to be used as evidence, over one thousand pages were produced in discovery.69

The main lesson of digitization, then, is that adversarial processes do not translate predicted productivity gains into lower-cost legal outcomes by default. David Engstrom and Jonah Gelbach hope that Technology Assisted Review (TAR) software, which uses predictive AI to classify documents for privilege and confidentiality based on an initial training set of labeled documents, can eventually solve discovery’s inefficiencies.70 Yet they acknowledge this is far from guaranteed given litigation’s adversarial structure. And because federal rules require discovery requests to be “proportional” to a case’s needs, judges might respond to declining unit costs of discovery (the cost of producing each document) by authorizing more expansive discovery plans, leaving total costs high. Though discovery is important, it is not uniquely susceptible to adversarial dynamics. An arms race of legal work in any stage of litigation (e.g., pretrial motions, expert battles, appeals) can erode efficiency gains and make achieving clients’ objectives expensive.

Transactional Work

Similar adversarial patterns appear in transactional work like negotiating and drafting contracts. Hadfield illustrates how their value can be relative using an example of a merger agreement negotiation.71 In such negotiations, skillful lawyers can improve their client’s position by limiting disclosures, “skating the line between legitimate silence and misrepresentation.”72 Opposing counsel will try to stay one step ahead by deciding what to ask, which representations to seek, and how to interpret disclosure laws to reduce the likelihood that material information is withheld.73 Lawyers will also try to “outfox” each other in the language of the contract itself.74

In the event of future litigation, the clients whose lawyers misunderstood a term’s significance pay the price. Since one can always do more legal research or add more contract provisions, there is no natural limit on the capacity for additional legal work to absorb efficiency gains. And because contracts concern future obligations, added uncertainty makes it harder for clients to effectively compare the quality and price of legal services.

Contracts have grown longer and more complex over time. From 1996 to 2016, M&A agreements expanded from 35 to 88 single-spaced pages, their linguistic complexity increasing from post-graduate “grade 20” to postdoctoral “grade 30.”75 An analysis of privacy policies over a similar period found a similar trend.76 John Coates argues that the increases in length and complexity reflect necessary and valuable responses to emerging legal risks.77 While longer, more complex contracts may represent better agreements, they might be necessary only in a legal system as adversarial as America’s, where litigation risks are high. Some scholars argue German contracts deliver similarly satisfactory legal outcomes with fewer words.78

That said, though transactional work has the adversarial elements described above, the overall structure is less adversarial than litigation. Because contract negotiations take place before a dispute occurs, there are more opportunities for transactional attorneys to add value beyond securing more of a fixed set of resources for their clients in a zero-sum negotiation.79

Human Oversight

A third bottleneck is our desire for human involvement. This is most relevant when AI gains outpace increases in production, which could happen for a few reasons. Perhaps there is an upper limit on arms races for certain kinds of legal work. After all, some legal doctrines are only so complicated, and courts often impose strict page limits on filings. Or maybe AI is so advanced that the costs of all legal tasks fall basically to zero and increased production does not absorb productivity gains. In this scenario, the new bottleneck for litigation would be the time required for judges to resolve cases, and for transactions, it would be the time parties need to understand a contract’s terms.

Starting with litigation, by reducing the cost of filing an initial lawsuit, AI will likely result in more disputes ending up in court. Within each dispute, AI can then create the kind of arms race of outputs described above. As a very rough but conservative estimate, Yonathan Arbel predicts a two- to fivefold increase in the volume of litigation.80

Arbel outlines several ways that judges might respond to a flood of litigation.81 They could limit the flow of litigation by altering procedural and substantive doctrines to make it harder for litigants to get into court (creating a bottleneck related to regulatory barriers). Or they could try to limit the use of AI in the courtroom, perhaps by requiring lawyers to disclose AI use or banning AI entirely and sanctioning any violators. Both responses would counteract a flood of litigation work but come at the steep cost of sacrificing access to justice for poorer litigants.

The debt collection context also provides an uninspiring picture of how courts have managed a flood of cases.82 As technology has enabled collectors to buy outstanding debt and cheaply file lawsuits for enforcement, the explosion of debt collection lawsuits has overwhelmed state courts.

Some have resorted to delegating cases to court assistants. Others operate “judgeless courtrooms” with lax evidentiary standards that can lower the quality of adjudication and can undermine the rationale for the judicial process itself.83 If judges do not delegate, adjudicating cases will take longer as both the number of cases and the work each requires expands. Yet as the common legal maxim says: “Justice delayed is justice denied.” Although more people might gain access to court, if judges require years to adjudicate cases, plaintiffs will face a choice between protracted litigation with no guarantee of success and settling cases on increasingly unfavorable terms as resolution times lengthen.

Another option is incorporating AI into the judicial process to ease the strain on overwhelmed courts.84 One early report suggests that AI is helping Brazil’s courts resolve cases more quickly.85 Yet if AI advances continuously reduce filing costs and drive up legal outputs, it will grow increasingly difficult for judges to keep pace. Perhaps AI will make judges more efficient. But there is a limit to how much AI can accelerate the process without meaningfully sacrificing human involvement.

Some seem open to replacing human judges entirely with AI.86 We find the legal (Article III, which establishes the federal courts, likely requires human judges),87 technical (hallucination and private influence problems),88⁸⁸ and moral objections persuasive.89 Even avowedly pro-AI lawyer Adam Unikowsky acknowledges he is “not quite ready to be ruled by robots.”90 This is not to say that there is no role for AI in judging. But judges should adopt AI through careful, deliberate choices instead of in ways compelled by the need to keep up with an arms race of AI-powered legal work.

The argument for contracts is similar. If advanced AI reduces the cost of drafting contracts (perhaps it can instantly draft 50 perfect provisions), a contracting party, even with the help of AI, will still need time to understand what those provisions do and how they impact the party’s future interests.

This bottleneck would not apply if people were to forgo oversight. Arguably, human involvement matters less for contracts because many Americans already agree to contracts (like privacy policies) without reading them.91 But we think this reflects the belief that they have insufficient bargaining power to negotiate new terms, or that it’s not worthwhile to do so, rather than a general endorsement of signing contracts they don’t understand.

All this to say, we believe some measure of human involvement in the legal system is valuable and necessary, though this is a normative position and not an empirical claim.

Institutional Reforms

Many problems facing the legal industry are not new. For decades, legal academics and practitioners have suggested reforms targeting the bottlenecks described above. Some address AI specifically, others are more general, and a few are already being tested in various states and jurisdictions. In this section, we organize proposals into three categories, each corresponding to a bottleneck above: the regulation of legal services, the adjudication process, and the evolving role of human beings in legal work. While we aren’t tied to any particular recommendation, we discuss a range of reforms to illustrate that the future of the practice of law could look very different depending on how institutions respond (or decline to respond) to AI.

Reforming Professional Regulation

A chorus of voices have suggested reforms to the legal profession’s self-regulations.92 Their recommendations range from clarifying existing UPL laws to modifying law firm ownership rules to overhauling how the profession itself is regulated.93 Some are already being tested.

Clarifying Unauthorized Practice of Law Rules

Current UPL laws define practice of law imprecisely and may prohibit companies from offering AI-powered legal assistance to consumers. Some jurisdictions are expanding who may provide legal services, and academics have proposed updates with AI in mind.

Creating a new tier of legal service providers is one of the “fastest growing UPL reform program types” nationally, with seven states adopting this approach and another ten considering it.94 David Autor argues these reforms would benefit all professions by allowing people, in combination with AI, to work at levels of expertise previously unavailable to them. He analogizes to the creation of the nurse practitioner role.95 In the early 1960s, nurses and doctors developed training programs and successfully lobbied the American Medical Association to create a new class of medical professionals who could perform tasks previously reserved for doctors.96 Other researchers offer a more concrete framework for how state courts might design this tier of legal service provider.97

Joseph Avery and co-authors propose more ambitious reforms: allowing nonlawyers, including AI systems, to offer many legal services.98 Bar associations would retain authority over who may use the designation of “lawyer,” but nonlawyers could provide any legal service other than representing clients in court.99 This would allow companies and nonprofits to offer AI-enabled services to consumers without claiming they are licensed attorneys and without the threat of UPL litigation. The prospect of being sued for negligent work would still serve as a quality backstop for lawyers and nonlawyers alike.

Sean Steward, by contrast, takes no position on where to draw the line between acceptable and unacceptable uses of AI, instead emphasizing the need for clear, nationwide rules to reduce burdens on providers.100 Drew Simshaw likewise advocates a nationwide approach to eliminate the patchwork of vague, conflicting rules.101

To be clear, the problem with existing UPL rules is not that they are regulations and therefore stifle innovation. It is that their uncertainty and variation discourage competition from new entrants, including the kind that produces better legal services.

Alternative Business Structures

Legal scholars have long argued for updating “entity regulations” that prevent nonlawyers from sharing fees from or investing in law firms.102 Utah and Arizona recently created regulatory sandboxes to do exactly that.103 Utah’s sandbox allows entities to seek waivers from ownership restrictions and UPL rules, while Arizona eliminated restrictions on law firm ownership and fee-sharing. These sandboxes permit companies and nonprofits to operate under modified professional rules while regulators assess their impact on service quality, cost, and access to justice.104

These sandboxes treat regulatory experimentation as necessary for balancing protection and innovation for consumers. Overly stringent restrictions can backfire by protecting inefficient incumbents or forcing new entrants outside the law. Uber and Airbnb succeeded, in part, by accepting regulatory fines, scaling quickly, and becoming so ubiquitous that lawmakers had little choice but to legalize their conduct. Yet overly lax restrictions can undermine the regulations’ purpose: protecting consumers from poor quality services. Sandboxes allow policymakers to experiment and evaluate different regulatory approaches.

Early evidence on the impact of these reforms has been largely positive, though concerns have emerged regarding private equity ownership and mass tort litigation financing. Despite scant evidence of consumer harm—Utah’s Office of Legal Services Innovation received only twenty total complaints —lawyers and commerce groups petitioned the Arizona and Utah supreme courts to limit these sandboxes. The Arizona Supreme Court stayed the course, and authorized entities grew from nineteen to 136 between 2022 and 2025.105 The Utah Supreme Court has since raised eligibility requirements, and authorized entities shrank from thirty-nine to eleven over the same period.106

Regulatory Markets

Gillian Hadfield proposes a “superregulator” model that would create a market for the regulation of legal services.107 Rather than regulating providers directly, the government would license regulators that would each offer competing regulatory schemes. The government’s role shifts to “regulating the regulators” by setting outcome targets, such as acceptable levels of legal access or dispute resolution quality, and then licensing regulators that achieve them.

Hadfield argues this generates powerful incentives for innovation.108 A private regulator that develops simpler, more cost-effective compliance methods while meeting government standards will attract more customers. The model can also simplify enforcement: Governments can monitor ten licensed regulators more easily than thousands of individual providers.

We would add that regulatory markets may be able to assess AI’s utility for legal services more reliably than benchmarking.109 Two of us have emphasized that task-oriented benchmarks lack construct validity because they “overemphasize precisely the thing that language models are good at” while failing to test the contextual understanding and sustained reasoning that characterizes consequential legal work.110 Benchmarks can also miss hidden costs that emerge only over time, such as deskilling of professionals.111 Relatedly, by targeting entry-level tasks, AI can disrupt the pipeline through which junior lawyers develop expertise.112

Hadfield defends the proposal’s practicality by drawing parallels to existing models. Governments already use outcomes-based regulations in environmental law, and private standard-setting bodies design many regulations currently in use. The United Kingdom provides one example. Under the Legal Services Act 2007, Parliament created the Legal Services Board (LSB), an independent agency that approves private bodies applying to regulate legal services.113 The system is not yet fully competitive because regulators came from preexisting trade associations for barristers and solicitors, which together regulate 90 percent of legal professionals in England and Wales.114 But competition is emerging for “alternative business structures,” which can choose between licensing from the Solicitors Regulation Authority (SRA) or the Bar Standards Board (BSB).115

This U.K. model has, however, encountered difficulties. In October 2024, the LSB criticized the SRA for failing to “act adequately, effectively, and efficiently” before the law firm Axiom Ince collapsed in October 2023.116 The LSB issued a report in March 2025 expressing “serious concerns” about the SRA’s effectiveness and then recently proposed sanctions.117 This highlights how superregulators can struggle to enforce quality standards for the regulators it oversees. Some scholars have separately criticized regulatory markets in other contexts for creating a race to the bottom. For example, Daniel Schwarcz cautions that a market for insurance regulation could “trigger a ‘race to the bottom’ as regulators compete with each other to offer less and less intrusive regulatory schemes.”118

Reforming Adjudication

Another set of reforms targets case adjudication. Some aim to make the trials less adversarial, while others advocate for private adjudication like arbitration.

Judicial Case Management

Judges have some discretion over the litigation process and can exercise it to reduce adversarial dynamics. Several judges have recommended leveraging existing rules of evidence and civil procedure to manage cases more actively, taking inspiration from other jurisdictions (often called inquisitorial systems).119 Such targeted borrowing can help reduce competitive escalation.

One example is allowing courts to appoint their own expert witnesses. Under Federal Rule of Evidence 706, judges can appoint neutral experts that work for the court but are paid for by both parties.120 One state trial judge argues this can solve the “battle of experts” problem where competing specialists “abandon objectivity and become advocates for the side that hired them.”121 When technical issues are central to a case, court-appointed experts can provide neutral assessments that frame issues more productively, avoiding an arms race of dueling expert reports.122

Another tool is the use of “special masters” under Federal Rule of Civil Procedure 53.123 These are neutral third parties appointed to help manage complex aspects of cases. A federal judge and senior litigator explain that special masters can “assist and, when necessary, direct the parties” to complete discovery efficiently.124 They note that the 2003 amendments expanded the scope of special masters’ use to include pretrial matters “that cannot be effectively and timely addressed by an available district judge.”125 Rather than having parties fight over AI-assisted document review through successive motions, a special master could serve as an intermediary and prevent the technology from enabling larger discovery battles.

Judges currently have the discretion to intervene under these rules, but Congress could also pass legislation that makes them mandatory.

Arbitration

While disputes are normally resolved through litigation, some contracts specify private arbitration.126 Contract drafters might prefer this process for several reasons: It can resolve disputes at lower costs,127 reduce class-action exposure,128 and prove “more flexible and less adversarial … than its judicial counterpart.”129 Arbitration has become a popular alternative to traditional litigation.130

Two legal scholars argue that because arbitration grants parties autonomy over the process, it’s the “ideal entry point for broader AI adoption in the legal field.”131 They defend AI arbitration as consistent with the Federal Arbitration Act (FAA) and desirable for enhancing “efficiency, fairness, and flexibility of dispute resolution.”132 Another scholar takes the opposite stance, arguing that the FAA does not permit AI arbitration because robot adjudicators are inconsistent with the statute’s use of human pronouns like “he or they.”133 And still another views it as undesirable because it would “significantly diminish the long-standing reputation” of arbitration.134

Some observers critique arbitration as unfair because companies often force it on consumers and employees through “take-it-or-leave-it” contracts of adhesion.135 These fairness concerns are important, and others have written about the level of consent needed for arbitration to be truly fair.136 But assuming the decision to enter arbitration reflects the free choice of both parties, offering AI arbitration as a parallel track for resolving cases can have several advantages.

First, it can promote choice by allowing litigants to decide between traditional judicial review and AI-assisted adjudication.137 A consumer defending against an automated debt collection suit might prefer quick AI resolution over years of waiting, while a defendant facing serious consequences might insist on traditional review by human judges.

Second, should an arms race of legal outputs risk overwhelming the courts, the availability of a technology-mediated alternative can alleviate pressure on the courts, preserving human review for the contexts where those navigating the judicial system feel they need it most.138

Third, it creates a natural experiment that facilitates comparison between human and AI adjudicators on dimensions like speed, cost, and participant satisfaction.139 This generates evidence about AI’s actual performance, reducing reliance on speculation or vendor claims, and it pressures traditional institutions to improve or risk being outcompeted.140

The Evolving Role of Lawyers

A final set of reforms discusses the evolving role of lawyers in both litigation and transactional contexts. With litigation, legislatures could expand the judiciary to alleviate the bottleneck created by the time human judges take to resolve cases. With transactions, there isn’t a clear action item for companies, but we expect to see a shift in what in-house counsel do. As the bottleneck becomes the time it takes human lawyers to understand complex contracts, in-house lawyers will likely spend more time understanding a company’s needs and making strategic judgments and less on legal tasks.

Expanding the Judiciary

The most straightforward response to an overburdened judiciary is to increase its capacity by hiring more judges. Legal scholars have advocated for this solution for decades.

For example, writing in 1979, Maria Marcus argued that “[s]ince the factors that channel disputes into a judicial forum continue unabated, the appointment of more judges is an obvious response.”141 Bert Huang again recommended in 2011 that “new demands put on the courts should be met quickly and flexibly with new judicial resources.142 More recently, Peter Menell and Ryan Vacca endorsed an observation from decades earlier that “the increase in the order of magnitude of the demands our society imposes on the federal judicial system” should encourage Congress to act.143

Menell and Vacca acknowledge that “[i]ncreasing the number of federal judgeships has been fraught with political complications.” Their solution is a bipartisan “2030 Commission” to depoliticize the process.144 Arbel similarly views adding judges as “the most direct way of solving the problem” of an AI-driven increase in litigation work, yet stops short of recommending it because it “appears quite tenuous in our current political reality.”145 He notes that redirecting all civil legal aid funding (approximately $2.7 billion) toward the $9.4 billion federal court system would yield at most a 30 percent increase in judicial capacity, falling short of the doubling likely needed to handle the increased caseload.146

But civil legal aid is not the only potential funding source. Some states have proposed taxing legal services generally,147 while federal legislation introduced by Sen. Thom Tillis (R-N.C.) would tax third-party litigation financiers who fund plaintiffs’ legal fees in exchange for a percentage of eventual winnings.148 Although these bills aim to increase overall tax revenues, similar measures could earmark funds specifically for the judiciary. Deborah Rhode has proposed another approach: mandatory pro bono service requirements for all attorneys, with the option to “buy out” their obligation.149 Though initially conceived as a way to provide access to justice, these payments could also support judicial expansion.

So while we agree that expanding the judiciary faces real political obstacles, we don’t think it’s as unrealistic as Arbel fears, especially considering the magnitude of AI’s potential disruption.

In-House Counsel as Strategic Advisors

Our analysis suggests that among in-house lawyers, value will likely shift from completing tasks to predicting how contracts and agreements impact the overall business. Lawyers will need to deeply understand their organizations and exercise business judgment.

Industry reports agree.150 In a 2023 survey of nearly three hundred chief legal officers, 87 percent said their role is shifting from legal risk mitigator to strategic business partner.151 Thomson Reuters’s 2024 “Future of Professionals Report” found that 42 percent of legal professionals expect to spend more time on judgment-based legal work in the next five years, as AI handles more routine tasks.152 As one attorney respondent put it: “The role of a good lawyer is as a ‘trusted advisor,’ not as a producer of documents … breadth of experience is where a lawyer’s true value lies and that will remain valuable.”153

To understand the shifting role of legal professionals, it is helpful to consider the hierarchy of legal work and roles. At the bottom is basic “low-skilled” legal work like drafting standard letters or simple contracts, repetitive tasks requiring minimal legal expertise.154 Above that is medium-skill, noncommoditized legal work that involves producing documents, such as analyzing contracts and drafting motions.155 Higher still is “judgment-based legal work”: overseeing complex trials and addressing legal risks.156 At the top is strategic advising, where lawyers deeply understand an organization’s priorities and shape its decisions.157 This highest level might not involve what we traditionally consider legal work at all.

As AI pushes the role of human expertise up this hierarchy, the legal profession should rethink how it trains lawyers. While AI automates the tasks at the bottom of the hierarchy, demand will likely grow for lawyers at the top who can translate legal information into strategic advice. One bar association warned that the “greater concern is that generative AI will displace younger attorneys,” who will “have fewer opportunities to gain valuable experience by spending hours on important tasks.”158 As the skills required to succeed change, so too should the training process.

Conclusion

Many problems facing the legal industry do not require revolutionary insights to solve. Scholars and practitioners have long emphasized the need for regulatory reform, changes to the litigation process, and the disconnect between legal work and client outcomes. AI, rather than solving these problems, appears to be revealing and magnifying them. Without addressing these underlying issues, AI alone is unlikely to improve the outcomes clients care about.

But AI may also present new opportunities for reform. Sociologists have argued that fields facing crises are more receptive to efforts to reshape institutions in those areas.159 Such crises, jolts, shocks, and disruptive events—taking the form of social upheaval, technological disruption, or other changes—can reveal problems and contradictions that require solutions. The widespread predictions that AI will transform the practice of law may constitute such a crisis, creating pressure for legal institutions to respond. The key question is whether they can use this opportunity to enact the reforms the industry has needed for decades and produce better outcomes for clients.

Acknowledgments: We are grateful to Samar Ahmad, James Bedford, Jasper Boers, Lev Cohen, Joe Goode, Mihir Kshirsagar, Martha Minow, Ben Press, and Jonathan Zittrain for helpful feedback on this report.

Here’s what I found:

• Moravec’s paradox never been empirically tested. (It’s often repeated as a fact by many AI researchers, including pioneers I know and respect, but that doesn’t mean I’ll take their claims at face value!)

• It is really a statement about what the AI community finds it worthwhile to work on. It doesn’t have any predictive power about which problems are going to be easy or hard for AI.

• It comes with an evolutionary explanation that I find highly dubious. (AI researchers have a history of making stuff up about human brains without any relevant background in neuroscience or evolutionary biology.)

• Moravec’s-paradox-style thinking has led to both alarmism (about imminent superintelligent reasoning) and false comfort (in areas like robotics).

• To adapt to AI advances, we don’t need to predict capability breakthroughs. Since diffusion of new capabilities takes a long time, that gives us plenty of time to react — time that we often squander, and then panic!

Watch the full argument here or read it below.

Every week brings new claims about AI advances. How do we know what’s coming next? Could AI predict crime? Write award-winning novels? Hack into critical infrastructure? Will we finally have robots in our home that will fold our clothes and load our dishwashers?

What will AI advances mean for your job? What will it mean for the social fabric? It’s hard to deal with all this uncertainty. If only we had a way to predict which new AI capabilities will be developed soon and which ones will remain hard for the foreseeable future.

Historically, AI researchers’ predictions about progress in AI abilities have been pretty bad. We don’t really have principles that describe which kinds of tasks are easy for AI and which ones are hard.

Well, we have one — Moravec’s paradox. It refers to the observation that it’s easy to train computers to do things that people find hard, like math and logic, and hard to train them to do things that we find easy, like seeing the world or walking.

It comes from the 1988 book Mind Children by Hans Moravec, who was — and is — a robotics researcher. He wrote:

It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.

In the early days of artificial intelligence, researchers focused on chess and other reasoning tasks, since these were thought to be some of the hardest and what made us uniquely human. But funnily enough, if you want to build a robot that can defeat human grandmasters, figuring out which moves to make is the easy part. Physically making the moves on the chessboard is the hard part. This is pretty well known today, so Moravec’s paradox seems to make a lot of intuitive sense.

If Moravec’s paradox is true, the implications would be amazing. If we want to know which AI capabilities might be built next, we just have to see how hard they are for humans. So scientific research will get automated before folding clothes, and so on.

But here’s the thing — Moravec’s paradox has never been fact checked. And that’s despite videos with hundreds of thousands of views, and TED talks all repeating it as a fact. When I dug into the evidence behind the so-called paradox, I found something surprising.

In this essay I’ll discuss why the theory and evidence behind the paradox are flaky. Then I’ll explain why simplistic predictions about what is easy or hard for AI have misled AI researchers and tech leaders. It has led to alarmism on the one hand and false comfort on the other hand. (Now there’s a paradox.) And finally I’ll answer the question, if we can’t rely on Moravec’s paradox, then how should we prepare for AI advances and their impacts?

The evidence behind the paradox is flaky

How would we test Moravec’s paradox? We could take some sample of the tasks that are out there, determine how hard they are for people, how hard they are for AI, and make a graph. If we saw something like this, the paradox would be confirmed.

But here’s the problem: Which set of tasks should we consider for our analysis? When AI researchers say Moravec’s paradox checks out, they are implicitly limiting their focus to problems that are considered interesting in the AI research community.

There are an endless number of tasks that are easy for both humans and AI, but they are not interesting. How bright is an image? How to play tic tac toe? Thousands of these tasks get solved by programmers on a daily basis and coded into AI systems, and when people do so, they don’t make a fuss about them.

There are also an endless number of tasks that are hard for humans and, as far as anyone knows, are also hard for AI. Identifying hit songs; predicting stock prices; cracking as-yet-undeciphered ancient scripts; even building a Dyson sphere. These are so hard that there is essentially no progress on these problems, although some of them attract junk research that tends to quickly get debunked. So these problems also don’t tend to get talked about as much.

In fact, there are thousands of problems that computer scientists have proved to be “NP-complete”, which means we have strong mathematical reasons to think they will forever be hard for AI, so serious AI researchers don’t tend to work on them. They work on easier, approximate versions of the problems instead.

The other two quadrants of the chart are different. On the top left, tasks like playing soccer, that are easy for humans but currently hard for AI, are extremely interesting to AI researchers. That’s because we know it’s possible to teach AI these skills, but we haven’t yet managed to do so, which makes them great tests for AI progress.

On the bottom right, problems that are hard for humans but easy for AI, such as searching the web, are also interesting. These capabilities tend to greatly augment human productivity. Even though they are in some sense “easy”, the industry invests a lot in making web search and other tools work as effectively, quickly, and cheaply as possible. So research on these tasks is a big driver of AI progress.

In short, when you’re thinking about the space of all possible tasks, if you basically ignore two quadrants of your 2x2 matrix because they are not interesting, then of course it will seem like what you’re left with shows a strong negative correlation between the two axes.

A flawed evolutionary argument

To be clear, the reason AI researchers are drawn to Moravec’s paradox isn’t because it is empirically backed. It’s because it comes with an intuitively appealing story. In his book, Moravec provided this explanation:

Encoded in the large, highly evolved sensory and motor portions of the human brain is a billion years of experience about the nature of the world and how to survive in it. The deliberate process we call reasoning is, I believe, the thinnest veneer of human thought, effective only because it is supported by this much older and much more powerful, though usually unconscious, sensorimotor knowledge. We are all prodigious olympians in perceptual and motor areas, so good that we make the difficult look easy. Abstract thought, though, is a new trick, perhaps less than 100 thousand years old. We have not yet mastered it. It is not all that intrinsically difficult; it just seems so when we do it.

And then Moravec praises reasoning systems like STRIPS from the 1970s, and this is the kind of thing he has in mind when he says that reasoning is easy for AI. These are purely symbolic systems that solve problems like how to put blocks on top of each other in a certain way.

But AI researchers have learned a lot in the half century since the heyday of STRIPS and other such reasoning programs. They seem quite quaint today. What we’ve learned is that symbolic reasoning only works well in closed and extremely narrow domains like chess with a fully specified set of rules. When you apply them to real-world problems they are brittle and go off the rails quickly.

Over the decades there have been many other failures of reasoning systems, like IBM’s Watson, that excelled in narrow demonstrations but failed when you tried to use them in real-world settings that could depart from what they are trained for.

Today, it is widely recognized that reasoning in open-ended settings requires common-sense knowledge. But common sense is one of the so-called easy-for-humans-but-hard-for-AI areas according to acolytes of Moravec’s paradox. In other words, AI reasoning isn’t easy after all.

And sure enough, AI reasoning in a way that can replace human expertise, in open-ended domains like law or scientific research, is still very much unsolved.

If I were to speculate where Moravec’s evolutionary argument goes wrong, it would be this: Reasoning might be new from an evolutionary perspective, but it builds on the things that animal brains have learned to do over hundreds of millions of years. This much Moravec acknowledges. But maybe there isn’t a separate skill called “abstract” reasoning that can be learned separately without all of this infrastructure.

How simplistic models have misled AI researchers and tech leaders

Unfortunately, partly because of the misconception in the AI world that reasoning is a separate skill that’s easy for computers, there is a widespread belief that AI will soon be superhuman at things like scientific research or operating a company or even running a government. In my experience, many researchers tend to generalize from AI success at chess and other closed domains to these kinds of open-ended domains.

So AI leaders promise that investing trillions into data centers will lead to a cure for cancer and various other downstream benefits, without stopping to think about how these inputs will translate into the desired outputs. It also leads them to support some extreme policies, such as investment in AI science at the expense of human scientists, and warning policymakers to prepare for a white collar bloodbath. It also leads to the fear that leaders such as politicians or CEOs military generals will soon have no choice but to delegate important decisions to AI because it will be superhuman at reasoning.

Maybe... or maybe it’s all a myth. There may not even be one general skill called reasoning.

Maybe the limits to reasoning are actually things like the lack of verifiers. That is, AI can’t get that good at legal reasoning unlike reasoning in chess, because there is no way for AI to write millions of legal arguments and get immediate and accurate feedback on which ones are good and which ones aren’t, analogous to the way AI gets good at chess.

Maybe it’s partly due to the limits of real-world knowledge. That is, AI can’t quickly become superhuman at medical reasoning because it is limited by the available medical knowledge in the world.

In other words, the same factors that pose limits to human reasoning also pose limits to AI reasoning. In this view, the answer has nothing to do with biology.

Much of my writing is based on the idea that superhuman AI reasoning is a myth. Of course, you don’t have to accept this. But when you hear these predictions of imminent superintelligence, it’s helpful to understand the underlying mental model that many people in the AI field have. And it’s definitely useful to know that we have five or six decades of evidence that whether a skill such as reasoning is easy or hard for AI can depend a lot on whether you’re talking about a closed domain or an open domain.

Now let’s look at the other side of the coin. Some AI capabilities are predicted to be hard because of Moravec’s paradox. Most frequently, it’s robotics.

Just as people say we have to worry about job losses and safety implications of breakthroughs in reasoning, they’ll say we don’t have to worry about job losses and safety risks of breakthroughs in robotics. It’s a hard problem, so improvements won’t happen overnight.

Unfortunately, this is false comfort. I could be wrong about what I said about reasoning earlier, and there could be a breakthrough tomorrow. Similarly, these researchers could be wrong about robotics and there could be a breakthrough tomorrow.

In fact, there is another “hard problem” for AI that used to be invoked to explain Moravec’s paradox, which is computer vision, and it doesn’t get invoked anymore, and that’s because there was in fact a breakthrough around 2012 / 2013 when AI performance at tasks like object recognition shot up dramatically due to deep learning. The reason it even took that long is it required the use of GPUs and the idea of using GPUs for AI only began around that time.

The scientific ideas behind deep learning had largely been established in the 1980s — before Moravec published his book — he just didn’t know it yet.

Conclusion: if not Moravec’s paradox, then what?

There’s a reason why rules of thumb like Moravec’s paradox are so tempting. It’s because we assume that if there were an AI capability breakthrough, there would be rapid societal effects such as job losses, so we should be prepared in advance.

But this isn’t actually true. Even breakthrough technologies take a long time to be successfully commercialized and deployed. In fact, breakthrough technologies may especially take a long time to be deployed, because the supporting infrastructure just isn’t there. There’s a famous case study of why it took 40 years for electric power to replace steam power in factories.

Take self-driving cars. Waymo started testing them on public roads more than 15 years ago.

There was little reason to doubt that this technology would one day be viable, and that it would have good and bad effects. Policymakers should have been preparing and figuring out how to compensate workers who stood to lose out.

Instead, people are only now waking up to the reality now that these cars and trucks are already on the road, and are pushing policies like banning them. Keep in mind that a million people per year die in car accidents worldwide, and self-driving cars are already much safer than human drivers.

A recurring theme of my work on AI is the difficulty of prediction — both the use of AI itself for prediction and predicting the future of AI capabilities and impacts.

Instead of relying on prediction, we should get better at shaping and adapting to the tech that we actually know for sure is coming. It is psychologically hard to let go of our need to want to know the future, but if we give up this false comfort we can build a much more resilient society.

Other videos

I have published three full-length videos so far, and regularly publish short videos. Subscribe here.

• What happens if there’s an AI crash?

• Why a Manhattan project for AI makes no sense

• Debunking Moravec’s paradox

Reflecting this shift, we have renamed this newsletter. We have already published two follow-up essays to AI as Normal Technology and will publish more regularly as we expand our framework into a book, which we plan to complete in late 2026 for publication in 2027.

Today, we address common points of confusion about AI as Normal Technology, try to make the original essay more approachable, and compare it to AI 2027.

Table of contents

1. Normal doesn’t mean mundane or predictable

2. A restatement of our thesis

3. If disappointment about GPT-5 has nudged you towards AI as normal technology, it’s possible you don’t quite understand the thesis

4. Why it’s hard to find a “middle ground” between AI as Normal Technology and AI 2027

5. It is hard to understand one worldview when you’re committed to another

6. Reaping AI’s benefits will require hard work and painful choices

7. The surreal debate about the speed of diffusion

8. Why AI adoption hits different

9. Concluding thoughts

Normal doesn’t mean mundane or predictable

While the essay talks about what we mean by normal (more on that below), we could have been more explicit about what it doesn’t mean.

Our point is not “nothing to see here, move along”. Indeed, unpredictable societal effects have been a hallmark of powerful technologies ranging from automobiles to social media. This is because they are emergent effects of complex interactions between technology and people. They don’t tend to be predictable based on the logic of the technology alone. That’s why rejecting technological determinism is one of the core premises of the normal technology essay.

In the case of AI, specifically chatbots, we’re already seeing emergent societal effects. The prevalence of AI companions and some of the harmful effects of model sycophancy such as “AI psychosis” have taken most observers by surprise.2 On the other hand, many risks that were widely predicted to be imminent, such as AI being used to manipulate elections, have not materialized.

What the landscape of AI’s social impacts will look like in say 3-5 years — even based on the diffusion of current capabilities, not future capabilities — is anyone’s guess.

The development of technical capabilities is more predictable than AI’s social impacts. Daniel Kokotajlo, one of the authors of AI 2027, was previously famous in the AI safety community for his essay “What 2026 looks like” back in 2021. His predictions about the tech itself proved eerily accurate, but the predictions about social impacts were overall not directionally correct, a point he was gracious enough to concede in a podcast discussion with one of us.

All this makes AI a more serious challenge for institutions and policymakers because they will have to react nimbly to unforeseeable impacts instead of relying on the false comfort of prediction or trying to prevent all harm. Broadly speaking, the policymaking approach that enables such adaptability is called resilience, which is what our essay advocated for. But while we emphasized resilience as the approach to deal with potentially catastrophic risks, we should have been clearer that resilience also has an important role in dealing with more diffuse risks.

Perhaps the reason some readers misunderstood our view of predictability is the word “normal”. Again, our goal is not to trivialize the task of individually and collectively adapting to AI. In an ideal world, a better title would have been simply “AI as Technology”, but we didn’t think that that would effectively communicate that our goal is to provide an alternative to the exceptionalism that characterizes the superintelligence worldview which currently dominates the discourse.

A restatement of our thesis

If we were to extract and simplify the core of our thesis, it would be something like this:

There is a long causal chain between AI capability increases and societal impact. Benefits and risks are realized when AI is deployed, not when it is developed. This gives us (individuals, organizations, institutions, policymakers) many points of leverage for shaping those impacts. So we don’t have to fret as much about the speed of capability development; our efforts should focus more on the deployment stage both from the perspective of realizing AI’s benefits and responding to risks. All this is not just true of today’s AI, but even in the face of hypothetical developments such as self-improvement in AI capabilities. Many of the limits to the power of AI systems are (and should be) external to those systems, so that they cannot be overcome simply by having AI go off and improve its own technical design.

Aspects of this framework may have to be revised eventually, but that lies beyond the horizon bounding what we can meaningfully anticipate or prepare for:

The world we describe in Part II is one in which AI is far more advanced than it is today. We are not claiming that AI progress—or human progress—will stop at that point. What comes after it? We do not know. Consider this analogy: At the dawn of the first Industrial Revolution, it would have been useful to try to think about what an industrial world would look like and how to prepare for it, but it would have been futile to try to predict electricity or computers. Our exercise here is similar. Since we reject “fast takeoff” scenarios, we do not see it as necessary or useful to envision a world further ahead than we have attempted to. If and when the scenario we describe in Part II materializes, we will be able to better anticipate and prepare for whatever comes next.

Anyway, to reiterate, the core of the thesis is the underlying causal framework for understanding the relationship between AI and society, not any of the specific impacts that it might or might not have. In our view, if you share this causal understanding, you subscribe to the normal technology thesis. We have found that this framework is indeed widely shared, albeit implicitly.

That makes the thesis almost tautological in many readers’ minds. We are making what we see as — and what those readers should see as — a very weak claim! Not recognizing this causes readers to search for something much more specific that we may have meant by “normal”. But we didn’t. We aren’t classifying technologies as “normal” and “abnormal” and then putting AI into the “normal” bucket. We’re just saying we should treat AI like we treat other powerful general-purpose technologies.

This is not specific to large language models or any particular kind of AI. Incidentally, that’s why the title is “AI as normal technology” and not “AI as a normal technology”. Our views apply to the whole umbrella of technologies that are collectively referred to as AI, and other similar technologies even if they are not referred to as AI.

If our worldview is almost tautological, why bother to state it? Because it is in contrast to the superintelligence worldview. That’s the thing about worldviews: there can be mutually contradictory worldviews that each seem tautological to those who subscribe to them.

If disappointment about GPT-5 has nudged you towards AI as normal technology, it’s possible you don’t quite understand the thesis

It’s notable that there’s been a surge of interest in our essay after the release of GPT-5, and reasonable to surmise that at least some of that is because of people shifting their views a bit after being disappointed by the release.

This is strange! This isn’t the first time this has happened — we previously expressed skepticism of a big narrative shift around scaling that happened based on almost no new information. If a single update to one product shifts people’s views on the trajectory of AI, how reliable is people’s evidence base to begin with?

The reason why the normal technology framework predicts slow timelines is not because capabilities will hit a wall but because impacts will be slow and gradual even if capabilities continue to advance rapidly. So we don’t think disappointment with a new release should make you more sympathetic to viewing AI as normal technology. By the same token, a new breakthrough announced tomorrow shouldn’t cause you to be more skeptical of our views.

The best way to understand GPT-5 is that it’s a particularly good example of AI developers’ shift in emphasis from models to products, which we wrote about a year ago. The automatic model switcher is a big deal for everyday users of ChatGPT. It turns out that hardly anyone was using “thinking” models nearly a year after they were first released, and GPT-5 has bumped up their use dramatically.

In some communications Altman was clear that the emphasis for GPT-5 was usability, not a leap in capabilities, although this message was unfortunately undercut by the constant hype, leading to disappointment.

This broader shift in the industry is actually highly consistent with companies themselves (reluctantly) coming around to acknowledging the possibility that their path to success is to do the hard work of building products and fostering adoption, rather than racing to build AGI or superintelligence and count on it to sweep away any diffusion barriers. Ironically, in this narrative, GPT-5 is an example of a success, not a failure.

In fact, model developers are starting to go beyond developing more useful products (the second stage of our technology development & adoption framework) and working with deployers to ease early adoption pains (the third stage). For example, OpenAI’s Forward Deployed Engineers work with customers such as John Deere, and directly with farmers, on integrating and deploying capabilities such as providing personalized recommendations for pesticide application.

Why it’s hard to find a “middle ground” between AI as Normal Technology and AI 2027

Many people have tried to articulate middle ground positions between AI 2027 and AI as Normal Technology, perhaps viewing these as two ends of a spectrum of views.

This is surprisingly hard to do. Both AI 2027 and AI as Normal Technology are coherent worldviews. They represent very different causal understandings of how technology will impact society. If you try to mix and match, there is a risk that you end up with an internally inconsistent hodgepodge. (By the way, this means that if we end up being wrong, it is more likely that we will be wrong wholesale than slightly wrong.)

Besides, only in the Silicon Valley bubble can AI as Normal Technology be considered a skeptical view! We compare AI to electricity in the second sentence of the essay, and we make clear throughout that we expect it to have profound impacts. Our expectations for AI’s impact on labor seem to be at the more aggressive end of the range of expectations from economists who work on this topic.

In short, if you are looking for a moderate position, we encourage you to read the essay in full. Don’t let the title fool you into thinking we are AI skeptics. Perhaps you will conclude that AI as Normal Technology is already the middle ground you are looking for.

We realize that it can be discomfiting that the two most widely discussed frameworks for thinking about the future of AI are so radically different. (Our essay itself offers much commentary on this state of affairs in Part 4, which is about policy.) We can offer a few comforting thoughts:

• We do have many areas of agreement with the AI 2027 authors. We are working on a joint statement outlining those areas. We are grateful to Nicholas Carlini for organizing this effort.

• In our view, more important than agreement in beliefs are areas of common ground in policy despite differences in beliefs. Even relatively “easy” policy interventions that different sides can agree on will be a huge challenge in practice. If we can’t achieve these, there is little hope for the much more radical measures favored by those worried about imminent superintelligence.

• There have been a few ongoing efforts to identify the cruxes of disagreement and agree on indicators that might help adjudicate between the two worldviews. We have participated in a few of these efforts and look forward to continuing to do so. We are grateful to the Golden Gate Institute for AI’s efforts on this front.

• Speaking of developing indicators, we are in the process of expanding the vision for our project HAL, Holistic Agent Leaderboard. Currently it tries to be a better benchmark orchestration system for AI agents, but the new plan is to develop it into an early warning system that helps the AI community identify when AI agents have crossed capability thresholds for transformative real-world impacts in various domains.
  We see these capability thresholds as necessary but not always sufficient conditions for impact, and as and when they are reached, they will much more acutely stress our theses about non-technological barriers to both benefits and risks.

• Note that HAL is not about prediction; it is about situational awareness of the present. This is a theme of our work. What is remarkable about the AI discourse in general, and us versus AI 2027 in particular, is the wide range of views not just about the future but about the things we can observe, such as the speed of diffusion (more on that below). Unless we as a community get much better at measurement of the present and testing competing causal explanations of progress, the level of energy directed at prediction will be misdirected, because we lack ways of resolving those predictions. For example, we’ve argued that we won’t necessarily know if “AGI” has been built even post facto. To some extent these limitations are intrinsic because of the lack of conceptual precision of ideas like AGI, but at the same time it’s true that we can do a lot better at measurement.

It is hard to understand one worldview when you’re committed to another

We wrote:

AI as normal technology is a worldview that stands in contrast to the worldview of AI as impending superintelligence. Worldviews are constituted by their assumptions, vocabulary, interpretations of evidence, epistemic tools, predictions, and (possibly) values. These factors reinforce each other and form a tight bundle within each worldview.

This makes communication across worldviews hard. For example, one question we often receive from the AI 2027 folks is what we think the world will look like in 2027. Well, pretty much like the world in 2025, we respond. They then push us to consider 2035 or 2045 or whatever year by which the world will be transformed, and they consider it a deficiency of our framework that we don’t provide concrete scenarios.

But this kind of scenario forecasting is only a meaningful activity within their worldview. We are concrete about the things we think we can be concrete about. At the same time, we emphasize the role of human, institutional, and political agency in making radically different futures possible — including AI 2027. Thus, AI as normal technology is as much a prescription as a prediction.

These communication difficulties are important to keep in mind when considering the response by Scott Alexander, one of the AI 2027 authors, to AI as Normal Technology. While we have no doubt that it is a good-faith effort at dialogue and we appreciate his putting in the time, unfortunately we feel that his response mostly talks past us. What he identifies as the cruxes of disagreement are quite different from what we consider the cruxes! For this reason, we won’t give a point-by-point response, since we will probably in turn end up talking past him in turn.

But we would be happy to engage in moderated conversations, a format with which we’ve had good success and have engaged in 8-10 times over the past year. The synchronous nature makes it much easier to understand each other. And the fact that the private conversation will be edited before making it public makes it easier to ask stupid questions as each side searches for understanding of the other’s point of view.

Anyway, here are a couple of important ways in which Alexander’s response talks past us. Recursive Self-Improvement (RSI) is a crux of disagreement for Alexander’s point of view, and he is surprised that it is barely worth a mention for us. In fairness we could have been much more explicit in our essay about what we think about RSI. In short, we don’t think RSI will lead to superintelligence because the external bottlenecks to building and deploying powerful AI systems cannot be overcome by improving their technical design. That is why we don’t discuss it much.3

Although it is not a crux for us, we do explain in the essay why we think the AI community is nowhere close to RSI. More recently we’ve been thinking about the fundamental research challenges that need to be solved, and there are a lot more than we’d realized. And it is worth keeping in mind that the AI community might be particularly bad at finding new paradigms for progress compared to other scientific communities. Again, this is an area where we hope that our project HAL can play a role in measuring progress.

Another topic where Alexander’s response talks past us is on the speed of diffusion, which we comment on briefly below and will address in more detail in a future essay.

The best illustration of the difficulty of discourse across worldviews is Alexander’s discussion of our hypotheses about whether or not superhuman AI abilities are possible in tasks such as prediction or persuasion. After reading his response several times, it is hard for us to figure out where exactly we agree and disagree. We wrote:

We think there are relatively few real-world cognitive tasks in which human limitations are so telling that AI is able to blow past human performance (as AI does in chess). ... Concretely, we propose two such areas: forecasting and persuasion. We predict that AI will not be able to meaningfully outperform trained humans (particularly teams of humans and especially if augmented with simple automated tools) at forecasting geopolitical events (say elections). We make the same prediction for the task of persuading people to act against their own self-interest.

You can read his full response to this in Section 3B of his essay, but in short it focuses on human biological limits:

Humans gained their abilities through thousands of years of evolution in the African savanna. There was no particular pressure in the savanna for “get exactly the highest Brier score possible in a forecasting contest”, and there is no particular reason to think humans achieved this. Indeed, if the evidence for human evolution for higher intelligence in the past 10,000 years in response to agriculture proves true, humans definitely didn’t reach the cosmic maximum on the African savannah. Why should we think this last, very short round of selection got it exactly right?

But rejecting a biological conception of human abilities is a key point of departure for us, something we take pains to describe in detail in the section “Human Ability Is Not Constrained by Biology”. That’s the problem with discussion across worldviews: If you take a specific statement, ignore the premises and terminological clarifications leading up to it, and interpret it in your worldview, it will seem like your opponent is clueless. Does Alexander think we are suggesting that if a savanna-dweller time traveled to the present, they would be able to predict elections?

He does emphasize that human performance is not fixed, but somehow sees this as a refutation of our thesis (rather than central to it). Perhaps the confusion arose because of our hypothesis that human performance at forecasting is close to the ​​“irreducible error”. We don’t imply that the irreducible error of forecasting is a specific number that is fixed for all time. Of course it depends on the data that is available — better polling leads to better forecasting — and training that helps take advantage of increased data. And some of that training might even be the result of AI-enabled research on forecasting. We emphasize in our original essay that human intelligence is special not because of our biology, but because of our (contingent) mastery of our tools including AI. Thus, advances in AI will often improve human intelligence (abilities), and have the potential to improve the performance of both sides of the human-AI comparison we propose.

The point of our hypothesis is a simple one: We don’t think forecasting is like chess, where loads of computation can give AI a decisive speed advantage. The computational structure of forecasting is relatively straightforward, even though performance can be vastly improved through training and data. Thus, relatively simple computational tools in the hands of suitably trained teams of expert forecasters can squeeze (almost) all the juice there is to squeeze.

We are glad that Alexander’s response credits us with “putting their money where their mouth is on the possibility of mutual cooperation”. The sentiment is mutual. We look forward to continuing that cooperation, which we see as more productive than Substack rebuttals and counter-rebuttals.

Reaping AI’s benefits will require hard work and painful choices

There are two broad sets of implications of our framework: one for the economy and labor, and the other for safety. Once we get past a few basic premises (notably, that superintelligence is incoherent or impossible depending on how it is defined), our arguments behind these two sets of implications are largely different.

On economic impacts, our case is broadly that diffusion barriers won’t be overcome through capability improvement. As for safety, our case is primarily that achieving AI control without alignment is not only possible, it doesn’t even seem particularly hard, and doesn’t require scientific breakthroughs.

Since these two sets of arguments don’t overlap much, it is coherent to accept one set but not accept (or be ambivalent to) the other. Indeed, our view of the economic impacts seems to have resonated particularly strongly with readers. Since the essay’s publication, we have had many discussions with people responsible for AI strategy in various industries. We discovered that the way that they had been thinking about AI was consistent with ours, but they were starting to second-guess their approach because of all the hype. Our essay provided a coherent framework that backed up their intuitions as well as their observations in the trenches.4

While people deploying AI have a keen understanding of the difference between technology development and diffusion, our framework further divides each of those into two steps. On the development side, we emphasize the gap between models and products, or capabilities and applications. On the diffusion side, we differentiate between user learning curves and other aspects of adaptation by individuals, and the structural, organizational, or legal changes that might be necessary, which often require collective action. We illustrate the kinds of speed limits that operate at each of the four stages.

While user behaviors at least tend to change slowly but predictably, solving coordination problems or reforming sclerotic institutions — which are also prerequisites for effective technology adoption — are much more uncertain. As an example, consider how Air Traffic Control is stuck with technology from the middle of the 20th century despite the enormous costs of the lack of modernization becoming apparent.

While our essay pointed out that analogous diffusion barriers exist in the case of AI, we are only now doing the work of spelling out those barriers and identifying specific reforms that might be necessary. We will be writing more on this front, some of it in collaboration with Justin Curl.

It is worth bearing in mind that advanced AI is entering a world that is already both highly technological and highly regulated. We repeatedly find that the parts of workflows that AI tackles are unlikely to be bottlenecks, because much of the available productivity gains have already been unlocked through earlier waves of technologies. Meanwhile the actual bottlenecks prove resistant due to regulation or other external constraints. In many specific domains including legal services and scientific research, competitive dynamics are so strong that productivity gains from AI lead to escalation of arms races that don’t ultimately translate to societal value.

The surreal debate about the speed of diffusion

We’ve mentioned a few times that different camps disagree on how they characterize current AI impacts. Nowhere is this more apparent than the speed of diffusion. AI boosters believe that it is being adopted unprecedentedly rapidly. We completely disagree. Worse, as more evidence comes out, each side seems to be getting more certain of their interpretation.

We are working on an in-depth analysis of the speed of diffusion. For now, we point out a few basic fallacies in the common arguments and stats that get trotted out to justify the “rapid adoption” interpretation.

First, deployment is not diffusion. Often, when people talk about rapid adoption they just mean that when capabilities are developed, they can be near-instantly deployed to products (such as chatbots) that are used by hundreds of millions of users.

But this is not what diffusion means. It is not enough to know how many people have access to capabilities: What matters is how many people are actually using them, how long they are using them, and what they are using them for. When we drill down into those details, the picture looks very different.

For example, almost a year after the vaunted release of “thinking” models in ChatGPT, less than 1% of users used them on any given day! We find no pleasure in pointing this out even though it supports our thesis. As enthusiastic early adopters of AI, this kind of number is so low that it is hard for us to intuitively grasp, and frankly pretty depressing.

Another example of a misleading statistic relates to the fraction of workers in certain high-risk domains who use AI. Such statistics tend to be offered in service of the claim that AI is being rapidly adopted in risky ways. But even in high risk domains most tasks are actually mundane, and when we dig into the specific uses don’t seem that risky at all.

For example, a survey by the American Medical Association reported that a majority of doctors are using AI. But this includes things like transcription of dictated notes.5 It also includes things like asking a chatbot for a second opinion of a diagnosis (about 12% reported this use case in 2024, a whopping 1 percentage point increase from 11% in 2023). This is definitely a more serious use than transcription one, but it is still well-founded. As we’ve pointed out before, even unreliable AI is very helpful for error detection.

Increasing adoption of AI for these tasks does not mean that doctors are about to start YOLOing it and abdicating their responsibility to their patients by delegating their decisions to ChatGPT. The vast majority of doctors understand the difference between these two types of uses, and there are many overlapping guardrails preventing widespread irresponsible use in the medical profession, including malpractice liability, professional codes, and regulation of medical devices.

The most misleading “rapid adoption” meme of all might be this widely shared chart, showing that ChatGPT reached 100M users in about two months:

It compares ChatGPT user growth with (1) Instagram, Facebook, and Twitter, which are social media apps whose usefulness depends on network effects, and therefore characteristically grow much slower than apps that are useful from day one (2) Spotify, an app that was initially invite-only and (3) Netflix, a service that launched with a limited inventory and required a subscription to use.6

What is reflected in this chart are early adopters who will check out an app if there’s buzz around it, and there was deafening buzz around ChatGPT. Once you exhaust these curious early users, the growth curve looks very different. In fact, a year later, ChatGPT had apparently only grown from 100M to 200M users, which meant that the curve evidently bent sharply to the right. That is conveniently not captured in this graph which reflects only the first two months.

This chart would be useful if it gave us any evidence that the usual barriers to diffusion have been weakened or eliminated. It doesn’t. Two months is not enough time for the hard parts of diffusion to even get started, such as users adapting their workflows to productively incorporate AI. As such, this chart is irrelevant to any meaningful discussion of the speed of diffusion.

There are many other problems with this chart, but we’ll stop here.7 Again, this is far from a complete analysis of the speed of AI diffusion — that’s coming. For now, we’re just making the point that the majority of the commentary on this topic is simply unserious. And if this is what the discourse is like on a question for which we do have data, it is no surprise that predictions of the future from different camps bear no resemblance to each other.

Why AI adoption hits different

If the “rapid diffusion” meme is so wrong, why is it so pervasive and persistent? Because AI adoption feels like a tsunami in a way that the PC or the internet or social media never did. When people are intuitively convinced of something, they will be much less skeptical of data or charts that purport to confirm that feeling.

We recognize the feeling, of course. Our own lived experience of AI is different from past waves of technologies. Initially, we dismissed this as a cognitive bias. Whatever change we’re living through at the moment will feel like a much bigger shift than something we successfully adapted to in the past.

We now realize that we were wrong. The cognitive bias might be a small part of the explanation, but there is a genuine reason why AI adoption feels much more rapid and scary. In short, while it’s true that deployment is not diffusion, in the past, gradual deployment meant that users were somewhat buffered from having to constantly make decisions about adoption, but now that buffer has been swept away. Let’s explain with a comparison to internet adoption.

Those of us who adopted dial-up internet in the 90s will remember a story that went something like this. When we first heard about the tech, we were put off by the high price of a PC. Gradually those prices came down. Meanwhile we got some experience using the internet at work or at a friend’s house. So when we bought a PC and dialup internet a few years later, we already had some training. At first, dialup was slow and expensive and there weren’t even that many websites, so we didn’t use the internet that much. Gradually prices came down, bandwidth improved, and more content came online, and we learned how to use the internet productively and safely in tandem with our increasing use.

Adopting general-purpose AI tools in the 2020s is a radically different experience because deployment of new capabilities is instantaneous. People have to spend a much higher percentage of time evaluating whether to adopt AI for some particular use case, and are constantly being told that if they don’t adopt it they will be left behind.

All our earlier points stand — learning curves exist, human behavior takes a long time to change, and organizational change even longer. But not using AI is somewhat of an active choice, and people no longer have the excuse of not to think about it because they don’t yet have access.

In short, deployment is only one of many steps in diffusion, and removing that bottleneck probably made diffusion slightly faster. But it feels dramatically faster because as soon as one hears about a particular AI use case, one has to decide whether to adopt it or not, even if it is ultimately the case that the vast majority of the time, people are deciding not to, for various reasons that might be rational or irrational.

Concluding thoughts

One thing on which we definitely agree with AI boosters is that AI is not going away, nor will it become a niche like crypto that most people can ignore. Now that the collective initial shock of generative AI has worn off, there’s a need for structured ways to think about how AI’s impacts might unfold, instead of (over)reacting to each new technical capability or emergent social effect.

The AI-as-normal-technology framework — which we continue to elaborate in this newsletter — is one such approach. It is worth being familiar with, at least as an articulation of a historically grounded default way to think about tech’s societal impact, against which more exceptionalist accounts can be compared. The framework has some degree of actionable guidance for business leaders, workers, students, people concerned about AI safety or AI ethics, and policymakers, among others. We hope you follow along and contribute to the discussion.

We are grateful to Steve Newman and Felix Chen for feedback on a draft.

Further reading/viewing

• Arvind presented the paper at the World Bank Development Conference, focusing on the economic and labor implications.

• Arvind’s new YouTube channel discusses AI developments based on the normal technology perspective.

• We were fortunate to receive press coverage for AI as Normal Technology:

  • In the last month, the New York Times had three op-eds that discussed the essay in depth, by Eric Schmidt and Selina Xu, by David Wallace-Wells, and by Ezra Klein.

  • Last week, the Economist discussed the normal technology thesis in the article "What if artificial intelligence is just a “normal” technology?"

  • In The New Yorker, Joshua Rothman contrasted AI 2027 with AI as Normal Technology.

  • In Prospect Magazine, Ethan Zuckerman discussed the utility of viewing AI as Normal Technology.

  • James O'Donnell, for the MIT Technology Review, summarized the AI as Normal Technology thesis.

• Some of our podcast appearances include: Arvind on New York Times' Hard Fork, and Tim O’Reilly’s podcast; Sayash on Lawfare's Scaling Laws and Carnegie's Interpreting India.

• We have had conversations with many people who have the superintelligence worldview, including many of the authors of AI 2027: Sayash debated Daniel Kokotajlo and Eli Lifland; Arvind debated Daniel Kokotajlo, and in Asterisk Mag, Arvind and Ajeya Cotra debated whether AI progress has a speed limit, and how we’d know.

It’s a common-sense view, at least among technologists, that AI will speed science greatly as it gets adopted in every part of the scientific pipeline — summarizing existing literature, generating new ideas, performing data analyses and experiments to test them, writing up findings, and performing “peer” review.

But many early common-sense predictions about the impact of a new technology on an existing institution proved badly wrong. The Catholic Church welcomed the printing press as a way of solidifying its authority by printing Bibles. The early days of social media led to wide-eyed optimism about the spread of democracy worldwide following the Arab Spring.

Similarly, the impact of AI on science could be counterintuitive. Even if individual scientists benefit from adopting AI, it doesn’t mean science as a whole will benefit. When thinking about the macro effects, we are dealing with a complex system with emergent properties. That system behaves in surprising ways because it is not a market. It is better than markets at some things, like rewarding truth, but worse at others, such as reacting to technological shocks. So far, on balance, AI has been an unhealthy shock to science, stretching many of its processes to the breaking point.

Any serious attempt to forecast the impact of AI on science must confront the production-progress paradox. The rate of publication of scientific papers has been growing exponentially, increasing 500 fold between 1900 and 2015. But actual progress, by any available measure, has been constant or even slowing. So we must ask how AI is impacting, and will impact, the factors that have led to this disconnect.

Our analysis in this essay suggests that AI is likely to worsen the gap. This may not be true in all scientific fields, and it is certainly not a foregone conclusion. By carefully and urgently taking actions such as those we suggest below, it may be possible to reverse course. Unfortunately, AI companies, science funders, and policy makers all seem oblivious to what the actual bottlenecks to scientific progress are. They are simply trying to accelerate production, which is like adding lanes to a highway when the slowdown is actually caused by a toll booth. It’s sure to make things worse.

Table of contents

1. Science has been slowing — the production-progress paradox

2. Why is progress slowing? Can AI help?

3. Science is not ready for software, let alone AI

4. AI might prolong the reliance on flawed theories

5. Human understanding remains essential

6. Implications for the future of science

7. Final thoughts

Science has been slowing — the production-progress paradox

The total number of published papers is increasing exponentially, doubling every 12 years. The total number of researchers who have authored a research paper is increasing even more quickly. And between 2000 and 2021, investment in research and development increased fourfold across the top seven funders (the US, China, Japan, Germany, South Korea, the UK, and France).1

But does this mean faster progress? Not necessarily. Some papers lead to fundamental breakthroughs that change the trajectory of science, while others make minor improvements to known results.

Genuine progress results from breakthroughs in our understanding. For example, we understood plate tectonics in the middle of the last century — the idea that the continents move. Before that, geologists weren’t even able to ask the right questions. They tried to figure out the effects of the cooling of the Earth, believing that that’s what led to geological features such as mountains. No amount of findings or papers in older paradigms of geology would have led to the same progress that plate tectonics did.

So it is possible that the number of papers is increasing exponentially while progress is not increasing at the same rate, or is even slowing down. How can we tell if this is the case?

One challenge in answering this question is that, unlike the production of research, progress does not have clear, objective metrics. Fortunately, an entire research field — the "science of science", or metascience — is trying to answer this question. Metascience uses the scientific method to study scientific research. It tackles questions like: How often can studies be replicated? What influences the quality of a researcher's work? How do incentives in academia affect scientific outcomes? How do different funding models for science affect progress? And how quickly is progress really happening?

Strikingly, many findings from metascience suggest that progress has been slowing down, despite dramatic increases in funding, the number of papers published, and the number of people who author scientific papers. We collect some evidence below; Matt Clancy reviews many of these findings in much more depth.

1) Park et al. find that "disruptive" scientific work represents an ever-smaller fraction of total scientific output. Despite an exponential increase in the number of published papers and patents, the number of breakthroughs is roughly constant.

2) Research that introduces new ideas is more likely to coin new terms. Milojevic collects the number of unique phrases used in titles of scientific papers over time as a measure of the “cognitive extent” of science, and finds that while this metric increased up until the early 2000s, it has since entered a period of stagnation, when the number of unique phrases used in titles of research papers has gone down.

3) Patrick Collison and Michael Nielsen surveyed researchers across fields on how they perceived progress in the most important breakthroughs in their fields over time — those that won a Nobel prize. They asked scientists to compare Nobel-prize-winning research from the 1910s to the 1980s.

They found that scientists considered advances from earlier decades to be roughly as important as the ones from more recent decades, across Medicine, Physics, and Chemistry. Despite the vast increases in funding, published papers, and authors, the most important breakthroughs today are about as impressive as those in the decades past.

4) Matt Clancy complements this with an analysis of what fraction of discoveries that won a Nobel Prize in a given year were published in the preceding 20 years. He found that this number dropped from 90% in 1970 to 50% in 2015, suggesting that either transformative discoveries are happening at a slower pace, or that it takes longer for discoveries to be recognized as transformative.

5) Bloom et al. analyze research output from an economic perspective. Assuming that economic growth ultimately comes from new ideas, the constant or declining rate of growth implies that the exponential increase in the number of researchers is being offset by a corresponding decline in the output per researcher. They find that this pattern holds true when drilling down into specific areas, including semiconductors, agriculture, and medicine (where the progress measures are Moore’s law, crop yield growth, and life expectancy, respectively).

Of course, there are shortcomings in each of the metrics above. This is to be expected: since progress doesn't have an objective metric, we need to rely on proxies for measuring it, and these proxies will inevitably have some flaws.

For example, Park et al. used citation patterns to flag papers as "disruptive": if follow-on citations to a given paper don't also cite the studies this paper cited, the paper is more likely to be considered disruptive. One criticism of the paper is that this could simply be a result of how citation practices have evolved over time, not a result of whether a paper is truly disruptive. And the metric does flag some breakthroughs as non-disruptive — for example, AlphaFold is not considered a disruptive paper by this metric.2

But taken together, the findings do suggest that scientific progress is slowing down, at least compared to the volume of papers, researchers, and resources. Still, this is an area where further research would be fruitful — while the decline in the pace of progress relative to inputs seems very clear, it is less clear what is happening at an aggregate level. Furthermore, there are many notions of what the goals of science are and what progress even means, and it is not clear how to connect the available progress measures to these higher-level definitions.

Why is progress slowing? Can AI help?

There are many hypotheses for why progress could be slowing. One set of hypotheses is that slowdown is an intrinsic feature of scientific progress, and is what we should expect. For example, there’s the low-hanging fruit hypothesis — the easy scientific questions have already been answered, so what remains to be discovered is getting harder.

This is an intuitively appealing idea. But we don’t find this convincing. Adam Mastroianni gives many compelling counter-arguments. He points out that we’ve been wrong about this over and over and lists many comically mis-timed assessments of scientific fields reaching saturation just before they ended up undergoing revolutions, such as physics in the 1890s.

While it’s true that lower-hanging fruits get picked first, there are countervailing factors. Over time, our scientific tools improve and we stand on the tower of past knowledge, making it easier to reach higher. Often, the benefits of improved tools and understanding are so transformative that whole new fields and subfields are created. New fields from the last 50-100 years include computer science, climate science, cognitive neuroscience, network science, genetics, molecular biology, and many others. Effectively, we’re plucking fruit from new trees, so there is always low-hanging fruit.

In our view, the low-hanging fruit hypothesis can at best partly explain slowdowns within fields. So it’s worth considering other ideas.

The second set of hypotheses is less fatalistic. They say that there’s something suboptimal about the way we’ve structured the practice of science, and so the efficiency of converting scientific inputs into progress is dropping. In particular, one subset of hypotheses flags the increase in the rate of production itself as the causal culprit — science is slowing down because it is trying to go too fast.

How could this be? The key is that any one scientist’s attention is finite, so they can only pay attention to a limited number of papers every year. So it is too risky for authors of papers to depart from the canon. Any such would-be breakthrough papers would be lost in the noise and won’t get the attention of a critical mass of scholars. The greater the rate of production, the more the noise, so the less attention truly novel papers will achieve, and thus will be less likely to break through into the canon.

Chu and Evans explain:

when the number of papers published each year grows very large, the rapid flow of new papers can force scholarly attention to already well-cited papers and limit attention for less-established papers—even those with novel, useful, and potentially transformative ideas. Rather than causing faster turnover of field paradigms, a deluge of new publications entrenches top-cited papers, precluding new work from rising into the most-cited, commonly known canon of the field.

These arguments, supported by our empirical analysis, suggest that the scientific enterprise’s focus on quantity may obstruct fundamental progress. This detrimental effect will intensify as the annual mass of publications in each field continues to grow

Another causal mechanism relates to scientists’ publish-or-perish incentives. Production is easy to measure, and progress is hard to measure. So universities and other scientific institutions judge researchers based on measurable criteria such as how many papers they publish and the amount of grant funding they receive. It is not uncommon for scientists to have to publish a certain number of peer-reviewed papers to be hired or to get tenure (either due to implicit norms or explicit requirements).

The emphasis on production metrics seems to be worsening over time. Physics Nobel winner Peter Higgs famously noted that he wouldn't even have been able to get a job in modern academia because he wouldn't be considered productive enough.

So individual researchers' careers might be better off if they are risk averse, but it might reduce the collective rate of progress. Rzhetsky et al. find evidence of this phenomenon in biomedicine, where experiments tend to focus too much on experimenting with known molecules that are already considered important (which would be more likely to lead to publishing a paper) rather than more risky experiments that could lead to genuine breakthroughs. Worryingly, they find this phenomenon worsening over time.

This completes the feedback loop: career incentives lead to researchers publishing more papers, and disincentivize novel research that results in true breakthroughs (but might only result in a single paper after years of work).

If slower progress is indeed being caused by faster production, how will AI impact it? Most obviously, automating parts of the scientific process will make it even easier for scientists to chase meaningless productivity metrics. AI could make individual researchers more creative but decrease the creativity of the collective because of a homogenizing effect. AI could also exacerbate the inequality of attention and make it even harder for new ideas to break through. Existing search technology, such as Google Scholar, seems to be having exactly this effect.

To recap, so far we’ve argued that if the slowdown in science is caused by overproduction, AI will make it worse. In the next few sections, we’ll discuss why AI could worsen the slowdown regardless of what’s causing it.

Science is not ready for software, let alone AI

How do researchers use AI? In many ways: AI-based modeling to uncover trends in data using sophisticated pattern-matching algorithms; hand-written machine learning models specified based on expert knowledge; or even generative AI to write the code that researchers previously wrote. While some applications, such as using AI for literature review, don't involve writing code, most applications of AI for science are, in essence, software development.

Unfortunately, scientists are notoriously poor software engineers. Practices that are bog-standard in the industry, like automated testing, version control, and following programming design guidelines, are largely absent or haphazardly adopted in the research community. These are practices that were developed and standardized over the last six decades of software engineering to prevent bugs and ensure the software works as expected.

Worse, there is little scrutiny of the software used in scientific studies. While peer review is a long and arduous step in publishing a scientific paper, it does not involve reviewing the code accompanying the paper, even though most of the "science" in computational research is being carried out in the code and data accompanying a paper, and only summarized in the paper itself.

In fact, papers often fail to even share the code and data used to generate results, so even if other researchers are willing to review the code, they don't have the means to. Gabelica et al. found that of 1,800 biomedical papers that pledged to share their data and code, 93% did not end up sharing these artifacts. This even affects results in the most prominent scientific journals: Stodden et al. contacted the authors of 204 papers published in Science, one of the top scientific journals, to get the code and data for their study. Only 44% responded.

When researchers do share the code and data they used, it is often disastrously wrong. Even simple tools, like Excel, have notoriously led to widespread errors in various fields. A 2016 study found that one in five genetics papers suffer from Excel-related errors, for example, because the names of genes (say, Septin 2) were automatically converted to dates (September 2). Similarly, it took decades for most scientific communities to learn how to use simple statistics responsibly.

AI opens a whole new can of worms. The AI community often advertises AI as a silver bullet without realizing how difficult it is to detect subtle errors. Unfortunately, it takes much less competence to use AI tools than to understand them deeply and learn to identify errors. Like other software-based research, errors in AI-based science can take a long time to uncover. If the widespread adoption of AI leads to researchers spending more time and effort conducting or building on erroneous research, it could slow progress, since researcher time and effort are wasted in unproductive research directions.

Unfortunately, we've found that AI has already led to widespread errors. Even before generative AI, traditional machine learning led to errors in over 600 papers across 30 scientific fields. In many cases, the affected papers constituted the majority of the surveyed papers, raising the possibility that in many fields, the majority of AI-enabled research is flawed. Others have found that AI tools are often used with inappropriate baseline comparisons, making it incorrectly seem like they outperform older methods. These errors are not just theoretical: they affect the potential real-world deployment of AI too. For example, Roberts et al. found that of 400+ papers using AI for COVID-19 diagnosis, none produced clinically useful tools due to methodological flaws.

Applications of generative AI can result in new types of errors. For example, while AI can aid in programming, code generated using AI often has errors. As AI adoption increases, we will discover more applications of AI for science. We suspect we'll find widespread errors in many of these applications.

Why is the scientific community so far behind software engineering best practices? In engineering applications, bugs are readily visible through tests, or in the worst case, when they are deployed to customers. Companies have strong incentives to fix errors to maintain the quality of their applications, or else they will lose market share. As a result, there is a strong demand for software engineers with deep expertise in writing good software (and now, in using AI well). This is why software engineering practices in the industry are decades ahead of those in research. In contrast, there are few incentives to correct flawed scientific results, and errors often persist for years.

That is not to say science should switch from a norms-based to a market-based model. But it shouldn't be surprising that there are many problems markets have solved that science hasn't — such as developing training pipelines for software engineers. Where such gaps between science and the industry emerge, scientific institutions need to intentionally adopt industry best practices to ensure science continues to innovate, without losing what makes science special.

In short, science needs to catch up to a half century of software engineering — fast. Otherwise, its embrace of AI will lead to an avalanche of errors and create headwinds, not tailwinds for progress.

AI could help too. There are many applications of AI to spot errors. For example, the Black Spatula project and the YesNoError project use AI to uncover flaws in research papers. In our own work, we've developed benchmarks aiming to spur the development of AI agents that automatically reproduce papers. Given the utility of generative AI for writing code, AI itself could be used to improve researchers' software engineering practices, such as by providing feedback, suggestions, best practices, and code reviews at scale. If such tools become reliable and see widespread adoption, AI could be part of the solution by helping avoid wasted time and effort building on erroneous work. But all of these possibilities require interventions from journals, institutions, and funding agencies to incentivize training, synthesis, and error detection rather than production alone.

AI might prolong the reliance on flawed theories

One of the main uses of AI for science is modeling. Older modeling techniques required coming up with a hypothesis for how the world works, then using statistical models to make inferences about this hypothesis.

In contrast, AI-based modeling treats this process as a black box. Instead of making a hypothesis about the world and improving our understanding based on the model's results, it simply tries to improve our ability to predict what outcomes would occur based on past data.

Leo Breiman illustrated the differences between these two modeling approaches in his landmark paper "Statistical Modeling: The Two Cultures". He strongly advocated for AI-based modeling, often on the basis of his experience in the industry. A focus on predictive accuracy is no doubt helpful in the industry. But it could hinder progress in science, where understanding is crucial.

Why? In a recent commentary in the journal Nature, we illustrated this with an analogy to the geocentric model of the Universe in astronomy. The geocentric model of the Universe—the model of the Universe with the Earth at the center—was very accurate at predicting the motion of planets. Workarounds like "epicycles" made these predictions accurate. (Epicycles were the small circles added to the planet's trajectory around the Earth).

Whenever a discrepancy between the model's predictions and the experimental readings was observed, astronomers added an epicycle to improve the model's accuracy. The geocentric model was so accurate at predicting planets' motions that many modern planetariums still use it to compute planets' trajectories.

How was the geocentric model of the Universe overturned in favor of the heliocentric model — the model with the planets revolving around the Sun? It couldn't be resolved by comparing the accuracy of the two models, since the accuracy of the models was similar. Rather, it was because the heliocentric model offered a far simpler explanation for the motion of planets. In other words, advancing from geocentrism to heliocentrism required a theoretical advance, rather than simply relying on the more accurate model.

This example shows that scientific progress depends on advances in theory. No amount of improvements in predictive accuracy could get us to the heliocentric model of the world without updating the theory of how planets move.

Let's come back to AI for science. AI-based modeling is no doubt helpful in improving predictive accuracy. But it doesn't lend itself to an improved understanding of these phenomena. AI might be fantastic at producing the equivalents of epicycles across fields, leading to the prediction-explanation fallacy.

In other words, if AI allows us to make better predictions from incorrect theories, it might slow down scientific progress if this results in researchers using flawed theories for longer. In the extreme case, fields would be stuck in an intellectual rut even as they excel at improving predictive accuracy within existing paradigms.

Could advances in AI help overcome this limitation? Maybe, but not without radical changes to modeling approaches and technology, and there is little incentive for the AI industry to innovate on this front. So far, improvements in predictive accuracy have greatly outpaced improvements in the ability to model the underlying phenomena accurately.

Human understanding remains essential

In solving scientific problems, scientists build up an understanding of the phenomena they study. It might seem like this understanding is just a way to get to the solution. So if we can automate the process of going from problem to solution, we don’t need the intermediate step.

The reality is closer to the opposite. Solving problems and writing papers about them can be seen as a ritual that leads to the real prize, human understanding, without which there can be no scientific progress.

Fields Medal-winning mathematician William Thurston wrote an essay brilliantly illustrating this. At the outset, he emphasizes that the point of mathematics is not simply to figure out the truth value for mathematical facts, but rather the accompanying human understanding:

…what [mathematicians] are doing is finding ways for people to understand and think about mathematics.

The rapid advance of computers has helped dramatize this point, because computers and people are very different. For instance, when Appel and Haken completed a proof of the 4-color map theorem using a massive automatic computation, it evoked much controversy. I interpret the controversy as having little to do with doubt people had as to the veracity of the theorem or the correctness of the proof. Rather, it reflected a continuing desire for human understanding of a proof, in addition to knowledge that the theorem is true.

On a more everyday level, it is common for people first starting to grapple with computers to make large-scale computations of things they might have done on a smaller scale by hand. They might print out a table of the first 10,000 primes, only to find that their printout isn't something they really wanted after all. They discover by this kind of experience that what they really want is usually not some collection of "answers"—what they want is understanding. [emphasis in original]

He then describes his experience as a graduate student working on the theory of foliations, a center of attention among many mathematicians. After he proved a number of papers on the most important theorems in the field, counterintuitively, people began to leave the field:

I heard from a number of mathematicians that they were giving or receiving advice not to go into foliations—they were saying that Thurston was cleaning it out. People told me (not as a complaint, but as a compliment) that I was killing the field. Graduate students stopped studying foliations, and fairly soon, I turned to other interests as well.

I do not think that the evacuation occurred because the territory was intellectually exhausted—there were (and still are) many interesting questions that remain and that are probably approachable. Since those years, there have been interesting developments carried out by the few people who stayed in the field or who entered the field, and there have also been important developments in neighboring areas that I think would have been much accelerated had mathematicians continued to pursue foliation theory vigorously.

Today, I think there are few mathematicians who understand anything approaching the state of the art of foliations as it lived at that time, although there are some parts of the theory of foliations, including developments since that time, that are still thriving.

Two things led to this desertion. First, the results he documented were written in a way that was hard to understand. This discouraged newcomers from entering the field. Second, even though the point of mathematics is building up human understanding, the way mathematicians typically get credit for their work is by proving theorems. If the most prominent results in a field have already been proven, that leaves few incentives for others to understand a field's contributions, because they can't prove further results (which would ultimately lead to getting credit).

In other words, researchers are incentivized to prove theorems. More generally, researchers across fields are incentivized to find solutions to scientific problems. But this incentive only leads to progress because the process of proving theorems or finding solutions to problems also leads to building human understanding. As the desertion of work on foliations shows, when there is a mismatch between finding solutions to problems and building human understanding, it can result in slower progress.

This is precisely the effect AI might have: by solving open research problems without leading to the accompanying understanding, AI could erode these useful byproducts by reducing incentives to build understanding. If we use AI to short circuit this process of understanding, that is like using a forklift at the gym. You can lift heavier weights with it, sure, but that's not why you go to the gym.

Of course, mathematics might be an extreme case, because human understanding is the end goal of (pure) mathematics, not simply knowing the truth value of mathematical statements. This might not be the case for many applications of science, where the end goal is to make progress towards a real-world outcome rather than human understanding, say, weather forecasting or materials synthesis.

Most fields lie in between these two extremes. If we use AI to bypass human understanding, or worse, retain only illusions of understanding, we might lose the ability to train new scientists, develop new theories and paradigms, synthesize and correct results, apply knowledge beyond science, or even generate new and interesting problems.

Empirical evidence across scientific fields has found evidence for some of these effects. For example, Hao et al. collect data from six fields and find that papers that adopt AI are more likely to focus on providing solutions to known problems and working within existing paradigms rather than generating new problems.

Of course, AI can also be used to build up tacit knowledge, such as by helping people understand mathematical proofs or other scientific knowledge. But this requires fundamental changes to how science is organized. Today's career incentives and social norms prize solutions to scientific problems over human understanding. As AI adoption accelerates, we need changes to incentives to make sure human understanding is prioritized.

Implications for the future of science

Over the last decade, scientists have been in a headlong rush to adopt AI. The speed has come at the expense of any ability to adapt slow-moving scientific institutional norms to maintain quality control and identify and preserve what is essentially human about science. As a result, the trend is likely to worsen the production-progress paradox, accelerating paper publishing but only digging us deeper into the hole with regard to true scientific progress.

So, what should the scientific community do differently? Let’s talk about the role of individual researchers, funders, publishers and other gatekeepers, and AI companies.

Changing scientific practices

Individual researchers should be more careful when adopting AI. They should build software engineering skills, learn how to avoid a long and growing list of pitfalls in AI-based modeling, and ensure they don’t lose their expertise by using AI as a crutch or an oracle. Sloppy use of AI may help in the short run, but will hinder meaningful scientific achievement.

With all that said, we recognize that most individual researchers are rationally following their incentives (productivity metrics). Yelling at them is not going to help that much, because what we have are collective action problems. The actors with real power to effect change are journals, universities hiring & promotion committees, funders, policymakers, etc. Let’s turn to those next.

Investing in meta-science

Meta-science research has been extremely valuable in revealing the production-progress paradox. But so far, that finding doesn’t have a lot of analytical precision. There’s only the fuzzy idea that science is getting less bang for its buck. This finding is generally consistent with scientists’ vibes, and is backed by a bunch of different metrics that vaguely try to measure true progress. But we don’t have a clear understanding of what the construct (progress) even is, and we’re far from a consensus story about what’s driving the slowdown.

To be clear, we will never have One True Progress Metric. If we did, Goodhardt/Campbell’s law would kick in — “When a measure becomes a target, it ceases to be a good measure.” Scientists would start to furiously optimize it, just as we have done with publication and citation counts, and the gaming would render it useless as a way to track progress.

That said, there’s clearly a long way for meta-science to go in improving both our quantitative and (more importantly) our qualitative/causal understanding of progress and the slowdown. Meta-science must also work to understand the efficacy of solutions.

Despite recent growth, meta-science funding is a fraction of a percent of science funding (and research on the slowdown is only a fraction of that pie). If it is indeed true that science funding as a whole is getting orders of magnitude less bang for the buck than in the past, meta-science investment seems ruefully small.

Reforming incentives

Scientists constantly complain to each other about the publish-or-perish treadmill and are keenly aware that the production-focused reward structure isn’t great for incentivizing scientific progress. But efforts to change this have consistently failed. One reason is simple inertia. Then there’s the aforementioned Goodhart’s law — whatever new metric is instituted will quickly be gamed. A final difficulty is that true progress can only be identified retrospectively, on timescales that aren’t suitable for hiring and promotion decisions.

One silver lining is that as the cost of publishing papers further drops due to AI, it could force us to stop relying on production metrics. In the AI field itself, the effort required to write a paper is so low that we are heading towards a singularity, with some researchers being able to (co-)author close to 100 papers a year. (But, again, the perceived pace of actual progress seems mostly flat.) Other fields might start going the same route.

Thus, rewarding the publication of individual findings may simply not be an option for much longer. Instead, perhaps the kinds of papers that count toward career progress should be limited to things that are hard to automate, such as new theories or paradigms of scientific research. And such reforms to incentive structures should go hand-in-hand with shifts in funding.

One thing we don’t need is more incentives for AI adoption. As we explained above, it is already happening at breakneck speed, and is not the bottleneck.

Rethinking AI-for-science tools

When it comes to AI-for-science labs and scientific tools from big AI companies, the elephant in the room is that they are in it for the wrong reasons. They want flashy “AI discovers X!” headlines so that they can sustain the narrative that AI will solve humanity’s problems, which buys them favorable policy treatment. We are not holding our breath for this to change.

We should be skeptical of AI-for-science news headlines. Many of them are greatly exaggerated. The results may fail to reproduce, or AI may be framed as the main character when it was in fact one tool among many.

If there are any AI-for-science tool developers out there who actually want to help, here’s our advice. Target the actual bottlenecks instead of building yet another literature review tool. How about tools for finding errors in scientific code or other forms of quality control? Listen to the users. For example, mathematicians have repeatedly said that tools for improving human understanding are much more exciting than trying to automate theorem-proving, which they view as missing the point.

The way we evaluate AI-for-science tools should also change. Consider a literature review tool. There are three kinds of questions one can ask: Does it save a researcher time and produce results of comparable quality to existing tools? How does the use of the tool impact the researcher’s understanding of the literature compared to traditional search? What will the collective impacts on the community be if the tool were widely adopted — for example, will it further focus the community’s attention on already-famous papers? These three questions get at production, understanding, and progress respectively.

Currently, only the first question is considered part of what evaluation means. The latter two are out of scope, and there aren’t even established methods or metrics for such measurement. That means that AI-for-science evaluation is guaranteed to provide a highly incomplete and biased picture of the usefulness of these tools and minimize their potential harms.

Final thoughts

We ourselves are enthusiastic users of AI in our scientific workflows. On a day-to-day basis, it all feels very exciting. That makes it easy to forget that the impact of AI on science as an institution, rather than individual scientists, is a different question that demands a different kind of analysis. Writing this essay required fighting our own intuitions in many cases. If you are a scientist who is similarly excited about using these tools, we urge you to keep this difference in mind.

Our skepticism here has similarities and differences to our reasons for the slow timelines we laid out in AI as Normal Technology. In that paper, we explained that market mechanisms exert some degree of quality control, and many shoddy AI deployments have failed badly, forcing companies who care about their reputation to take it slow when deploying AI, especially for consequential tasks, regardless of how fast the pace of development is. But in science, adoption and quality control processes are decoupled, with the former being much faster.

We are optimistic that scientific norms and processes will catch up in the long run. But for now, it’s going to be a bumpy ride.

We are grateful to Eamon Duede for feedback on a draft of this essay.

Further reading

• The American Science Acceleration Project (ASAP) is a national initiative with the stated goal of making American science "ten times faster by 2030". The offices of Senators Heinrich and Rounds recently requested feedback on how to achieve this. In our response, we emphasized the production-progress paradox, discussed why AI could slow (rather than hasten) scientific progress, and recommended policy interventions that could help.

• Our colleague Alondra Nelson also wrote a response to the ASAP initiative, emphasizing that faster science is not automatically better, and highlighted many challenges that remain despite increasing the pace of production.

• In a recent commentary in the journal Nature, we discussed why the proliferation of AI-driven modeling could be bad for science.

• We have written about the use of AI for science in many previous essays in this newsletter:

  • Can AI automate computational reproducibility?

  • Scientists should use AI as a tool, not an oracle

  • ML is useful for many things, but not for predicting scientific replicability

  • The REFORMS checklist for detecting and preventing errors in ML-based science

• Lisa Messeri and Molly Crockett offer a taxonomy of the uses of AI in science. They discuss many pitfalls of adopting AI in science, arguing we could end up producing more while understanding less.

• Matt Clancy reviewed the evidence for slowdowns in science and innovation, and discussed interventions for incentivizing genuine progress.

• The Institute for Progress released a podcast series on meta-science. Among other things, the series discusses concerns about slowdown and alternative models for funding and organizing science.

Update (July 17, 2025): Minor wording edits for clarity.

In this essay, we argue that AGI is not a milestone. It does not represent a discontinuity in the properties or impacts of AI systems. If a company declares that it has built AGI, based on whatever definition, it is not an actionable event. It will have no implications for businesses, developers, policymakers, or safety. Specifically:

1. Even if general-purpose AI systems reach some agreed-upon capability threshold, we will need many complementary innovations that allow AI to diffuse across industries to realize its productive impact. Diffusion occurs at human (and societal) timescales, not at the speed of tech development.

2. Worries about AGI and catastrophic risk often conflate capabilities with power. Once we distinguish between the two, we can reject the idea of a critical point in AI development at which it becomes infeasible for humanity to remain in control.

3. The proliferation of AGI definitions is a symptom, not the disease. AGI is significant because of its presumed impacts but must be defined based on properties of the AI system itself. But the link between system properties and impacts is tenuous, and greatly depends on how we design the environment in which AI systems operate. Thus, whether or not a given AI system will go on to have transformative impacts is yet to be determined at the moment the system is released. So a determination that an AI system constitutes AGI can only meaningfully be made retrospectively.

Nuclear weapons as an anti-analogy for AGI

Achieving AGI is the explicit goal of companies like OpenAI and much of the AI research community. It is treated as a milestone in the same way as building and delivering a nuclear weapon was the key goal of the Manhattan Project.

This goal made sense as a milestone in the Manhattan Project for two reasons. The first is observability. In developing nuclear weapons, there can be no doubt about whether you’re reached the goal or not — an explosion epitomizes observability. The second is immediate impact. The use of nuclear weapons contributed to a quick end to World War 2. It also ushered in a new world order — a long-term transformation of geopolitics.

Many people have the intuition that AGI will have these properties. It will be so powerful and humanlike that it will be obvious when we’ve built it. And it will immediately bring massive benefits and risks — automation of a big swath of the economy, a great acceleration of innovation, including AI research itself, and potentially catastrophic consequences for humanity from uncontrollable superintelligence.

In this essay, we argue that AGI will be exactly the opposite — it is unobservable because there is no clear capability threshold that has particular significance; it will have no immediate impact on the world; and even a long-term transformation of the economy is uncertain.

In previous essays, we have argued against the likely disastrous policy interventions that some have recommended by analogizing AGI to nuclear weapons. It is striking to us that this analogy reliably generates what we consider to be incorrect predictions and counterproductive recommendations.

It isn’t crazy to think that o3 is AGI, but this says more about AGI than o3

Many prominent AI commentators have called o3 a kind of AGI: Tyler Cowen says that if you know AGI when you see it, then he has seen it. Ethan Mollick describes o3 as a jagged AGI. What is it about o3 that has led to such excitement?

The key innovation in o3 is the use of reinforcement learning to learn to search the web and use tools as part of its reasoning chain.1 In this way, it can perform more complex cognitive tasks than LLMs are directly capable of, and can do so in a way that’s similar to people.

Consider a person doing comparison shopping. They might look at a few products, use the reviews of those products to get a better sense of what features are even important, and use that knowledge to iteratively expand or shrink the set of products being considered. o3 is a generalist agent that does a decent job at this sort of thing.

Let’s consider what this means for AGI. To avoid getting bogged down in the details of o3, imagine a future system whose architecture is identical to o3, but is much more competent. For example, it can always find the right webpages and knowledge for the task as long as it’s online, no matter how hard it is to locate. It can download and run code from the internet to solve a task if necessary. None of these require scientific breakthroughs, only engineering improvements and further training.

At the same time, without scientific improvements, the architecture imposes serious limits. For example, this future system cannot acquire new skills from experience, except through an explicit update to its training. Building AI systems that can learn on the fly is an open research problem.2

Would our hypothetical system be AGI? Arguably, yes. What many AGI definitions have in common is the ability to outperform humans at a wide variety of tasks. Depending on how narrowly the set of tasks is defined and how broadly the relevant set of humans for each task is defined, it is quite plausible that this future o3-like model/agent will meet some of these AGI definitions.

For example, it will be superhuman at playing chess, despite the fact that large language models themselves are at best mediocre at chess. Remember that the model can use tools, search the internet, and download and run code. If the task is to play chess, it will download and run a chess engine.

Despite human-level or superhuman performance at many tasks, and plausibly satisfying some definitions of AGI, it will probably fail badly at many real-world tasks. We’ll get back to the reasons for that.

Does any of this matter? It does. Leaders at AI companies have made very loud predictions and commitments to delivering AGI within a few years. There are enormous incentives for them to declare some near-future system to be AGI, and potentially enormous costs to not doing so. Perhaps some of the valuation of AI companies is based on these promises, so without AGI there might be a bubble burst. Being seen as leaders in AI development could help improve market share and revenues, and improve access to talent.

So, if and when companies claim to have built AGI, what will be the consequences? We'll analyze that in the rest of this essay.

AGI won't be a shock to the economy because diffusion takes decades

One argument for treating AGI as a milestone — and taking declarations of AGI seriously — is that AGI could lead to rapid economic impacts, both positive and negative, such as a world without scarcity, an end to the concept of money, or sudden mass joblessness.

But AI's economic impact is only realized when it is adopted across the economy. Technical advances are necessary, but not sufficient, to realize this impact. For past general-purpose technologies, such as electricity, computing, and the internet, it took decades for the underlying technical advances to diffuse across society. The miracle of the Industrial Revolution wasn't the high growth rate — annual growth rates averaged below 3% — but the sustained period of decades of growth.

There are many bottlenecks to the diffusion of AI: developing useful products and applications, training the workforce to utilize these products, implementing organizational changes to enable AI use, and establishing laws and norms that facilitate AI adoption by companies. Like past general-purpose technologies, we expect the economic impacts of AI to be realized over decades, as this process of diffusion unfolds.

In the paper AI as Normal Technology, we present a detailed argument for why we think this will be the case. The idea that rapid increases in capability lead to rapid economic impacts is completely inconsistent with the past and present of AI, and there is no reason to expect that this will change in the future.

One definition of AGI is AI systems that outperform humans at most economically valuable work. We might worry that if AGI is realized in this sense of the term, it might lead to massive, sudden job displacement. But humans are a moving target. As the process of diffusion unfolds and the cost of production (and hence the value) of tasks that have been automated decreases, humans will adapt and move to tasks that have not yet been automated. The process of technical advancements, product development, and diffusion will continue.

AGI will not lead to a rapid change in the world order

The US and China are often described as being in an AI arms race, with each country racing to build AGI. It is hypothesized that the country to build it first would have a decisive strategic advantage — resulting in dominance in the world order for the foreseeable future.3

This narrative doesn’t make sense because the knowledge required to create AI models, and model capabilities themselves, tend to proliferate quickly between countries. There are hundreds of thousands of AI technologists, and they work in the private sector rather than government labs, so it is not feasible to keep secrets at that scale.

Invention — in this case, AI model development — is overrated as a source of competitive advantage. We should expect technological developments to roughly keep pace across countries. Even though US companies are currently at the forefront, we shouldn't expect a lasting advantage.4

Many people haven't appreciated the ease of proliferation of technological capabilities — perhaps due to the nuclear weapons mental model — and have been caught by surprise. This is what led to the "DeepSeek moment" earlier this year, since analysts did not realize how quickly AI capabilities can proliferate, and as a result, were not expecting upstarts (especially from China) to catch up so quickly.

Some people argue that even an advantage of a few months will be critical. We disagree. The important question in the context of great power competition is not which country builds AGI first, but rather which country better enables diffusion. As Jeffrey Ding has shown, the effectiveness of companies and governments in actually utilizing AI inventions and innovations, whether domestic or international, to improve productivity is far more important for determining the economic impacts of general-purpose technologies.

While Chinese AI companies are at most 6-12 months behind leading US companies in terms of AI models and capabilities, China lags significantly behind the US in several key indicators that might enable diffusion: Digitization, cloud computing adoption, and workforce training. All of these are required to enable the productive diffusion of AI advances across industries. This is the actual source of American competitive advantage.

Of course, this could change in the coming years. But if it does, it will result from policy changes to promote diffusion rather than the development of AGI. And it is not something countries can accomplish overnight, regardless of how quickly they change policy. Diffusion typically unfolds over decades.

None of this means that policymakers should be complacent. But it does mean that rather than fixating on AGI, they should focus on enabling productive and safe diffusion, including of existing AI.

The long-term economic implications of AGI are uncertain

Even if it doesn’t have immediate economic impacts, could AGI unlock, say, 10% annual GDP growth that could add up to something big over a few decades?

Maybe. But it is far from clear why and how this will happen.

Historically, this kind of acceleration in growth has happened very few times — the industrial revolution had this effect, but not the internet, which barely had any impact on GDP. Note that even if you don’t think that GDP is the right thing to measure, a qualitative change in the GDP growth rate is a good proxy for whatever fundamental change in the economy you care about.

The problem is that accelerating growth requires eliminating bottlenecks to progress. That’s harder than most AI boosters assume. AI will likely have uneven effects across sectors, and long-term growth will be bottlenecked by the weakest sector.

Those arguing for dramatic effects often have an incorrect mental model of what the bottlenecks actually are. For example, while it is tempting to believe that cheap scientific innovation will unlock progress, the production of new findings is actually not the bottleneck in science.

More broadly, progress depends not just on the technology but on having the right preconditions — complementary innovations as well as cultural, economic, and political factors. If all it took to create the industrial revolution was the invention of steam power, the Roman Empire would have done it.

Our current laws, norms, institutions, and politics evolved in a time of much less technological potential. They are already choking opportunities for straightforward types of growth, such as building more public infrastructure. To reap the economic benefits that broad cognitive automation can potentially bring, the degree of structural change that needs to happen is unfathomably greater.

In conclusion, the extent and nature of long-term impacts from AGI remain to be seen and depend on what complementary actions we take. The long-term implications are not a property of AGI itself.

Misalignment risks of AGI conflate power and capability

On the flip side, AGI could be a turning point for AI's societal risks. Could it cause loss of control, massive societal harm, or even human extinction?

Discussions of AGI risks conflate power — the ability to modify the environment — with capability — the capacity to solve specified tasks correctly. Capability is an intrinsic property of an AI system, whereas power is a matter of how we design the environment in which AI systems operate. And humans have agency over this design. This distinction is often overlooked.

Consider Dario Amodei's definition of "powerful AI".5 He begins with a description of the capabilities of powerful AI, such as being able to "...prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch." This criterion is an example of AI capabilities, and we can discuss them at the level of AI systems.

But he then moves on to describe properties of the environment in which we allow AI systems to operate, which includes "...taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on." This is an example of the power afforded to an AI system. It depends on the environment within which the AI system operates, which determines how AI capabilities are translated into power.

We do expect AI capabilities to keep increasing. But regardless of capability level, we can choose to ensure that AI remains a tool and is not given power and autonomy to operate without human oversight. In the AI as Normal Technology essay, we address all the usual counterarguments to this, including arms races among companies, power seeking, superhuman persuasion, deceptive alignment, and more.

We argue in the paper that there will be strong business incentives against deploying AI without adequate oversight, and that these incentives can and should be buttressed by regulation when necessary. This has historically been the case in areas ranging from self-driving cars to AI assistants. We don’t expect this trend to suddenly flip once AI capabilities reach a presumed tipping point that we arbitrarily designate as AGI.

AGI does not imply impending superintelligence

Yet another reason to consider AGI a milestone is the view that shortly after we build AGI, AI systems could recursively self-improve — AGI could train future versions of models that become far more capable, leading to an "intelligence explosion." Soon afterwards, we would get superintelligent AI (AI systems that far exceed human abilities on any conceivable task), leading to either utopia or dystopia, depending on how well superintelligent AI is “aligned” with human interests.

In the normal technology view, there are two big reasons to doubt this narrative. The first is that even if arbitrary speedups in AI methods are possible, we think that innovation and diffusion will happen at human speed, as summarized in this table.

Second, the fact that AI would help conduct AI research does not imply that this process can be arbitrarily accelerated. AI is already used to automate a significant portion of AI research today. But there are many bottlenecks to progress in AI methods, such as the social nature of data collection and real-world interaction that might be required for achieving certain capabilities, computational and cost limits, or herding around popular or intuitive ideas while ignoring the ones that enable true breakthroughs.

We could be wrong about this, and recursive self-improvement could be possible, leading to unbounded speedups in progress in AI methods. And this might have some interesting implications, including some discontinuities in impact, even if widespread diffusion will be slower. For these reasons, it is important to have early warning systems for recursive self-improvement. However, this is not captured by AGI definitions. We might well have AGI while being far from recursive self-improvement, or vice versa.

We won’t know when AGI has been built

There are endless definitions of AGI and related concepts — Jasmine Sun has a useful compilation of over 20 definitions — but they fall into three broad categories: they can be based on the system’s impact on the world, its internals, or its behavior in controlled settings. We will show that each style of definition has a fatal flaw. It leads to criteria that are either too strict or too weak compared to what we ideally want.

Understanding these gaps also shows why people have different intuitions regarding what AGI will look like, and why “we’ll know it when we see it” has failed as a criterion and will continue to fail.

OpenAI's 2018 definition of AGI was "highly autonomous systems that outperform humans at most economically valuable work". From our perspective — our interest being in the impacts of AI — this definition is potentially very useful. If AI outperformed [all] humans at most economically valuable work, it would be unquestionably impactful.

But let’s be clear — this is not a property of an AI system. It is a property of the state of the world. It has at least as much to do with the complementary innovations that we make and the extent to which we choose to integrate AI into our organizations and institutions. It would be absurd to try to test an AI system in isolation in the lab and ask whether it outperforms people at their jobs. It is a category error.

For example, whether AI can (autonomously) outperform a medical researcher depends in part on whether we collectively choose to allow AI systems to perform large-scale medical experiments on people. We shouldn’t and we won’t, which means that irrespective of the systems' capabilities, they cannot perform the function of a medical researcher. This might be an extreme example, but similar bottlenecks arise in virtually every job.

Worse, until and unless we diffuse AI everywhere, we won’t even know if the system is theoretically capable of automating work in the real world. We won’t be able to set up sufficiently convincing simulacra of the messy complexity of the world. In short, impact-based definitions are not useful for practical purposes because they don’t give us a way to anticipate the end result of the painfully slow process of diffusion.

In contrast to researchers like us who are interested in impacts, many researchers are interested in humanlike AI in the sense of internals — does the system truly understand the world causally, can it reason, plan, and acquire new skills like we do, and so forth.

This sense of AGI has been notoriously hard to operationalize because of the difficulty of observing and characterizing AI internals. The Turing test is the best known of many attempts to use behavior as a proxy for the humanlike attributes we care about, but inevitably it turns out that we can build AI systems that pass such tests without having the hoped-for humanlike internals.

Furthermore, because of the jaggedness of AI — superhuman in many ways, yet lacking a toddler’s understanding of the world in other ways — the transformative effects of AI are likely to be felt long before it is fully humanlike (or superhuman) on all dimensions.

In short, since we are not interested in the internals for their own sake, we set aside this kind of definition.

That leaves us with the third kind of definition, which is by far the most common: those based on behavior, operationalized as benchmark performance. For example, the Metaculus question on “human-machine intelligence parity” is defined in terms of performance on exam questions in math, physics, and computer science. The problem with this kind of definition is well known, and we’ve discussed it repeatedly. They simply encourage hill climbing in the sense of building AI systems that can beat the benchmarks without necessarily being useful in the real world.

One response to the definitional challenge is to claim that we'll know AGI when we see it. The o3 launch shows that the opposite is true — to some people, it is clear that the advances in capabilities represent a step change that warrants calling it AGI. To others, the improvements are marginal at best, and unlikely to lead to real-world impact.

What explains people’s differing intuitions? Here’s our guess. While AI capabilities can be general, making AI useful in the real world will have to happen in a largely domain-specific way. (The generalist agent aspect of o3 is misleading — it can only handle generative tasks where errors have low cost, not when you need it to operate independently in the real world. For example, a useful travel booking AI agent hasn’t yet been released, despite the seeming triviality of the task for humans.)

So when people are thinking about whether o3 or any other system is (close to) AGI, they are intuitively thinking about different domains, and the gap between general capabilities and useful real-world abilities is vastly different from one domain to another. Crossing the threshold of having useful products that can automate most tasks may happen at vastly different times in different sectors or jobs.

Businesses and policy makers should take a long-term view

AGI is not a milestone because it is not actionable. A company declaring it has achieved, or is about to achieve, AGI has no implications for how businesses should plan, what safety interventions we need, or how policymakers should react. What should businesses and policy makers do instead?

Businesses should not rush to adopt half-baked AI products. Rapid progress in AI methods and capabilities does not automatically translate to better products. Building products on top of inherently stochastic models is challenging, and businesses should adopt AI products cautiously, conducting careful experiments to determine the impact of using AI to automate key business processes. In particular, we are extremely skeptical of the idea of AI agents being “drop-in replacements” for human workers that will somehow bypass the need for careful evaluation and integration of automation into workflows.

Companies developing AI products need a deep understanding of the domain to identify hurdles to adoption and build what businesses adopting AI want. For example, one key innovation of AI-enabled code editors such as Cursor and Windsurf is the user interface which allows programmers to verify AI-generated text at different levels of abstraction. Other industries will have different obstacles to AI adoption that products should address.

Policy makers should also take a long term view. A “Manhattan Project for AGI” is misguided on many levels. Since AGI is not a milestone, there is no way to know when the goal has been reached or how much more needs to be invested. And accelerating AI capabilities does nothing to address the real bottlenecks to realizing its economic benefits.

This view also has implications for export controls. The US has applied export controls on the hardware needed for AI development, hoping to slow down Chinese AI development. Proponents of export controls admit (and we agree) that this will not widen the gap between the US and China by more than a few months. But this only matters in a world where the impacts of developing advanced AI are rapid. If the impact of advanced AI is realized through diffusion, and the process of diffusion takes decades, being ahead by a matter of months in a game of decades is of little consequence. Policy makers should therefore focus on enabling diffusion. We outline some ideas on how to do so in our recent essay.

Treating AGI as a milestone for the development of transformative AI is seductive but misguided. It feeds into incorrect mental models of AI progress and risks, economic impacts, and geopolitics. AI's impact on the world will be realized not through a sprint toward a magic-bullet technology but through millions of boring little business process adaptations and policy tweaks.

We are grateful to Steve Newman and Jasmine Sun for feedback on a draft of this essay.

Further reading

• Jasmine Sun’s recent essay is similar to ours in spirit and substance. She contrasts many popular definitions of AGI and concludes that AGI is better seen as an aspiration for the AI community rather than a concrete technical milestone. She has also compiled popular definitions of AGI and other related concepts.

• Borhane Blili-Hamelin and others argue that we should stop treating AGI as the North star for AI research.

• Ege Erdil makes the case for multi-decade timelines to transformative AI capabilities.

• Eryk Salvaggio gives many reasons why it is dangerous for policy makers to act as if AGI is imminent.

• In AI as Normal Technology, we laid the intellectual foundation for our view of the future of AI. The present essay is the first of many follow-ups that explore the implications of the ideas therein.

The paper is also published in HTML and PDF formats on the Knight First Amendment Institute’s website. We are grateful for the extensive feedback we’ve received on drafts of the paper.

Update (September 2025): We have published a companion to this essay titled A guide to understanding AI as normal technology.

We articulate a vision of artificial intelligence (AI) as normal technology. To view AI as normal is not to understate its impact—even transformative, general-purpose technologies such as electricity and the internet are “normal” in our conception. But it is in contrast to both utopian and dystopian visions of the future of AI which have a common tendency to treat it akin to a separate species, a highly autonomous, potentially superintelligent entity.¹

The statement “AI is normal technology” is three things: a description of current AI, a prediction about the foreseeable future of AI, and a prescription about how we should treat it. We view AI as a tool that we can and should remain in control of, and we argue that this goal does not require drastic policy interventions or technical breakthroughs. We do not think that viewing AI as a humanlike intelligence is currently accurate or useful for understanding its societal impacts, nor is it likely to be in our vision of the future.²

The normal technology frame is about the relationship between technology and society. It rejects technological determinism, especially the notion of AI itself as an agent in determining its future. It is guided by lessons from past technological revolutions, such as the slow and uncertain nature of technology adoption and diffusion. It also emphasizes continuity between the past and the future trajectory of AI in terms of societal impact and the role of institutions in shaping this trajectory.

In Part I, we explain why we think that transformative economic and societal impacts will be slow (on the timescale of decades), making a critical distinction between AI methods, AI applications, and AI adoption, arguing that the three happen at different timescales.

In Part II, we discuss a potential division of labor between humans and AI in a world with advanced AI (but not “superintelligent” AI, which we view as incoherent as usually conceptualized). In this world, control is primarily in the hands of people and organizations; indeed, a greater and greater proportion of what people do in their jobs is AI control.

In Part III, we examine the implications of AI as normal technology for AI risks. We analyze accidents, arms races, misuse, and misalignment, and argue that viewing AI as normal technology leads to fundamentally different conclusions about mitigations compared to viewing AI as being humanlike.

Of course, we cannot be certain of our predictions, but we aim to describe what we view as the median outcome. We have not tried to quantify probabilities, but we have tried to make predictions that can tell us whether or not AI is behaving like normal technology.

In Part IV, we discuss the implications for AI policy. We advocate for reducing uncertainty as a first-rate policy goal and resilience as the overarching approach to catastrophic risks. We argue that drastic interventions premised on the difficulty of controlling superintelligent AI will, in fact, make things much worse if AI turns out to be normal technology— the downsides of which will be likely to mirror those of previous technologies that are deployed in capitalistic societies, such as inequality.³

The world we describe in Part II is one in which AI is far more advanced than it is today. We are not claiming that AI progress—or human progress—will stop at that point. What comes after it? We do not know. Consider this analogy: At the dawn of the first Industrial Revolution, it would have been useful to try to think about what an industrial world would look like and how to prepare for it, but it would have been futile to try to predict electricity or computers. Our exercise here is similar. Since we reject “fast takeoff” scenarios, we do not see it as necessary or useful to envision a world further ahead than we have attempted to. If and when the scenario we describe in Part II materializes, we will be able to better anticipate and prepare for whatever comes next.

A note to readers. This essay has the unusual goal of stating a worldview rather than defending a proposition. The literature on AI superintelligence is copious. We have not tried to give a point-by-point response to potential counter arguments, as that would make the paper several times longer. This paper is merely the initial articulation of our views; we plan to elaborate on them in various follow ups.

Part I: The Speed of Progress

Will the progress of AI be gradual, allowing people and institutions to adapt as AI capabilities and adoption increase, or will there be jumps leading to massive disruption, or even a technological singularity? Our approach to this question is to analyze highly consequential tasks separately from less consequential tasks and to begin by analyzing the speed of adoption and diffusion of AI before returning to the speed of innovation and invention.

We use invention to refer to the development of new AI methods—such as large language models—that improve AI’s capabilities to carry out various tasks. Innovation refers to the development of products and applications using AI that consumers and businesses can use. Adoption refers to the decision by an individual (or team or firm) to use a technology, whereas diffusion refers to the broader social process through which the level of adoption increases. For sufficiently disruptive technologies, diffusion might require changes to the structure of firms and organizations, as well as to social norms and laws.

AI Diffusion in Safety-critical Areas Is Slow

In the paper Against Predictive Optimization, we compiled a comprehensive list of about 50 applications of predictive optimization, namely the use of machine learning (ML) to make decisions about individuals by predicting their future behavior or outcomes.⁵ Most of these applications, such as criminal risk prediction, insurance risk prediction, or child maltreatment prediction, are used to make decisions that have important consequences for people.

While these applications have proliferated, there is a crucial nuance: In most cases, decades-old statistical techniques are used—simple, interpretable models (mostly regression) and relatively small sets of handcrafted features. More complex machine learning methods, such as random forests, are rarely used, and modern methods, such as transformers, are nowhere to be found.

In other words, in this broad set of domains, AI diffusion lags decades behind innovation. A major reason is safety—when models are more complex and less intelligible, it is hard to anticipate all possible deployment conditions in the testing and validation process. A good example is Epic’s sepsis prediction tool which, despite having seemingly high accuracy when internally validated, performed far worse in hospitals, missing two thirds of sepsis cases and overwhelming physicians with false alerts.⁶

Epic’s sepsis prediction tool failed because of errors that are hard to catch when you have complex models with unconstrained feature sets.⁷ In particular, one of the features used to train the model was whether a physician had already prescribed antibiotics —to treat sepsis. In other words, during testing and validation, the model was using a feature from the future, relying on a variable that was causally dependent on the outcome. Of course, this feature would not be available during deployment. Interpretability and auditing methods will no doubt improve so that we will get much better at catching these issues, but we are not there yet.

In the case of generative AI, even failures that seem extremely obvious in hindsight were not caught during testing. One example is the early Bing chatbot “Sydney” that went off the rails during extended conversations; the developers evidently did not anticipate that conversations could last for more than a handful of turns.⁸ Similarly, the Gemini image generator was seemingly never tested on historical figures.⁹ Fortunately, these were not highly consequential applications.

More empirical work would be helpful for understanding the innovation-diffusion lag in various applications and the reasons for this lag. But, for now, the evidence that we have analyzed in our previous work is consistent with the view that there are already extremely strong safety-related speed limits in highly consequential tasks. These limits are often enforced through regulation, such as the FDA’s supervision of medical devices, as well as newer legislation such as the EU AI Act, which puts strict requirements on high-risk AI.¹⁰ In fact, there are (credible) concerns that existing regulation of high-risk AI is so onerous that it may lead to “runaway bureaucracy”.¹¹ Thus, we predict that slow diffusion will continue to be the norm in high-consequence tasks.

At any rate, as and when new areas arise in which AI can be used in highly consequential ways, we can and must regulate them. A good example is the Flash Crash of 2010, in which automated high-frequency trading is thought to have played a part. This led to new curbs on trading, such as circuit breakers.¹²

Diffusion is Limited by the Speed of Human, Organizational, and Institutional Change

Even outside of safety-critical areas, AI adoption is slower than popular accounts would suggest. For example, a study made headlines due to the finding that, in August 2024, 40% of U.S. adults used generative AI.¹³ But, because most people used it infrequently, this only translated to 0.5%-3.5% of work hours (and a 0.125-0.875 percentage point increase in labor productivity).

It is not even clear if the speed of diffusion is greater today compared to the past. The aforementioned study reported that generative AI adoption in the U.S. has been faster than personal computer (PC) adoption, with 40% of U.S. adults adopting generative AI within two years of the first mass-market product release compared to 20 % within three years for PCs. But this comparison does not account for differences in the intensity of adoption (the number of hours of use) or the high cost of buying a PC compared to accessing generative AI.¹⁴ Depending on how we measure adoption, it is quite possible that the adoption of generative AI has been much slower than PC adoption.

The claim that the speed of technology adoption is not necessarily increasing may seem surprising (or even obviously wrong) given that digital technology can reach billions of devices at once. But it is important to remember that adoption is about software use, not availability. Even if a new AI-based product is instantly released online for anyone to use for free, it takes time to for people to change their workflows and habits to take advantage of the benefits of the new product and to learn to avoid the risks.

Thus, the speed of diffusion is inherently limited by the speed at which not only individuals, but also organizations and institutions, can adapt to technology. This is a trend that we have also seen for past general-purpose technologies: Diffusion occurs over decades, not years.¹⁵

As an example, Paul A. David’s analysis of electrification shows that the productivity benefits took decades to fully materialize.¹⁶ Electric dynamos were “everywhere but in the productivity statistics” for nearly 40 years after Edison’s first central generating station. ¹⁷ This was not just technological inertia; factory owners found that electrification did not bring substantial efficiency gains.

What eventually allowed gains to be realized was redesigning the entire layout of factories around the logic of production lines. In addition to changes to factory architecture, diffusion also required changes to workplace organization and process control, which could only be developed through experimentation across industries. Workers had more autonomy and flexibility as a result of the changes, which also necessitated different hiring and training practices.

The External World Puts a Speed Limit on AI Innovation

It is true that technical advances in AI have been rapid, but the picture is much less clear when we differentiate AI methods from applications.

We conceptualize progress in AI methods as a ladder of generality.¹⁸ Each step on this ladder rests on the ones below it and reflects a move toward more general computing capabilities. That is, it reduces the programmer effort needed to get the computer to perform a new task and increases the set of tasks that can be performed with a given amount of programmer (or user) effort; see Figure 2. For example, machine learning increases generality by obviating the need for the programmer to devise logic to solve each new task, only requiring the collection of training examples instead.

It is tempting to conclude that the effort required to develop specific applications will keep decreasing as we build more rungs of the ladder until we reach artificial general intelligence, often conceptualized as an AI system that can do everything out of the box, obviating the need to develop applications altogether.

In some domains, we are indeed seeing this trend of decreasing application development effort. In natural language processing, large language models have made it relatively trivial to develop a language translation application. Or consider games: AlphaZero can learn to play games such as chess better than any human through self-play given little more than a description of the game and enough computing power—a far cry from how game-playing programs used to be developed.

However, this has not been the trend in highly consequential, real-world applications that cannot easily be simulated and in which errors are costly. Consider self-driving cars: In many ways, the trajectory of their development is similar to AlphaZero’s self-play—improving the tech allowed them to drive in more realistic conditions, which enabled the collection of better and/or more realistic data, which in turn led to improvements in the tech, completing the feedback loop. But this process took over two decades instead of a few hours in the case of AlphaZero because safety considerations put a limit on the extent to which each iteration of this loop could be scaled up compared to the previous one.²⁰

This “capability-reliability gap” shows up over and over. It has been a major barrier to building useful AI “agents” that can automate real-world tasks.²¹ To be clear, many tasks for which the use of agents is envisioned, such as booking travel or providing customer service, are far less consequential than driving, but still costly enough that having agents learn from real-world experiences is not straightforward.

Barriers also exist in non-safety-critical applications. In general, much knowledge is tacit in organizations and is not written down, much less in a form that can be learned passively. This means that these developmental feedback loops will have to happen in each sector and, for more complex tasks, may even need to occur separately in different organizations, limiting opportunities for rapid, parallel learning. Other reasons why parallel learning might be limited are privacy concerns: Organizations and individuals might be averse to sharing sensitive data with AI companies, and regulations might limit what kinds of data can be shared with third parties in contexts such as healthcare.

The “bitter lesson” in AI is that general methods that leverage increases in computational power eventually surpass methods that utilize human domain knowledge by a large margin.²² This is a valuable observation about methods, but it is often misinterpreted to encompass application development. In the context of AI-based product development, the bitter lesson has never been even close to true.²³ Consider recommender systems on social media: They are powered by (increasingly general) machine learning models, but this has not obviated the need for manual coding of the business logic, the frontend, and other components which, together, can comprise on the order of a million lines of code.

Further limits arise when we need to go beyond AI learning from existing human knowledge.²⁴ Some of our most valuable types of knowledge are scientific and social-scientific, and have allowed the progress of civilization through technology and large-scale social organizations (e.g., governments). What will it take for AI to push the boundaries of such knowledge? It will likely require interactions with, or even experiments on, people or organizations, ranging from drug testing to economic policy. Here, there are hard limits to the speed of knowledge acquisition because of the social costs of experimentation. Societies probably will not (and should not) allow the rapid scaling of experiments for AI development.

Benchmarks Do not Measure Real-World Utility

The methods-application distinction has important implications for how we measure and forecast AI progress. AI benchmarks are useful for measuring progress in methods; unfortunately, they have often been misunderstood as measuring progress in applications, and this confusion has been a driver of much hype about imminent economic transformation.

For example, while GPT-4 reportedly achieved scores in the top 10% of bar exam test takers, this tells us remarkably little about AI’s ability to practice law.²⁵ The bar exam overemphasizes subject-matter knowledge and under-emphasizes real-world skills that are far harder to measure in a standardized, computer-administered format. In other words, it emphasizes precisely what language models are good at—retrieving and applying memorized information.

More broadly, tasks that would lead to the most significant changes to the legal profession are also the hardest ones to evaluate. Evaluation is straightforward for tasks like categorizing legal requests by area of law because there are clear correct answers. But for tasks that involve creativity and judgment, like preparing legal filings, there is no single correct answer, and reasonable people can disagree about strategy. These latter tasks are precisely the ones that, if automated, would have the most profound impact on the profession.²⁶

This observation is in no way limited to law. Another example is the gap between self-contained coding problems at which AI demonstrably excels, and real-world software engineering in which its impact is hard to measure but appears to be modest.²⁷ Even highly regarded coding benchmarks that go beyond toy problems must necessarily ignore many dimensions of real-world software engineering in the interest of quantification and automated evaluation using publicly available data.²⁸

This pattern appears repeatedly: The easier a task is to measure via benchmarks, the less likely it is to represent the kind of complex, contextual work that defines professional practice. By focusing heavily on capability benchmarks to inform our understanding of AI progress, the AI community consistently overestimates the real-world impact of the technology.

This is a problem of ‘construct validity,’ which refers to whether a test actually measures what it is intended to measure.²⁹ The only sure way to measure real-world usefulness of a potential application is to actually build the application and to then test it with professionals in realistic scenarios (either substituting or augmenting their labor, depending on the intended use). Such ‘uplift’ studies generally do show that professionals in many occupations benefit from existing AI systems, but this benefit is typically modest and is more about augmentation than substitution, a radically different picture from what one might conclude based on static benchmarks like exams³⁰ (a small number of occupations such as copywriters and translators have seen substantial job losses³¹).

In conclusion, while benchmarks are valuable for tracking progress in AI methods, we should look at other kinds of metrics to track AI impacts (Figure 1). When measuring adoption, we must take into account the intensity of AI use. The type of application is also important: Augmentation versus substitution and high-consequence versus low-consequence.

The difficulty of ensuring construct validity afflicts not only benchmarking, but also forecasting, which is another major way in which people try to assess (future) AI impacts. It is extremely important to avoid ambiguous outcomes to ensure effective forecasting. The way that the forecasting community accomplishes this is by defining milestones in terms of relatively narrow skills, such as exam performance. For instance, the Metaculus question on “human-machine intelligence parity” is defined in terms of performance on exam questions in math, physics, and computer science. Based on this definition, it is not surprising that forecasters predict a 95% chance of achieving “human-machine intelligence parity” by 2040. ³²

Unfortunately, this definition is so watered down that it does not mean much for understanding the impacts of AI. As we saw above with legal and other professional benchmarks, AI performance on exams has so little construct validity that it does not even allow us to predict whether AI will replace professional workers.

Economic Impacts are Likely to be Gradual

One argument for why AI development may have sudden, drastic economic impacts is that an increase in generality may lead to a wide swath of tasks in the economy becoming automatable. This is related to one definition of artificial general intelligence (AGI)—a unified system that is capable of performing all economically valuable tasks.

According to the normal technology view, such sudden economic impacts are implausible. In the previous sections, we discussed one reason: Sudden improvements in AI methods are certainly possible but do not directly translate to economic impacts, which require innovation (in the sense of application development) and diffusion.

Innovation and diffusion happen in a feedback loop. In safety-critical applications, this feedback loop is always slow, but even beyond safety, there are many reasons why it is likely to be slow. With past general-purpose technologies such as electricity, computers, and the internet, the respective feedback loops unfolded over several decades, and we should expect the same to happen with AI as well.

Another argument for gradual economic impacts: Once we automate something, its cost of production, and its value, tend to drop drastically over time compared to the cost of human labor. As automation increases, humans will adapt, and will focus on tasks that are not yet automated, perhaps tasks that do not exist today (in Part II we describe what those might look like).

This means that the goalpost of AGI will continually move further away as increasing automation redefines which tasks are economically valuable. Even if every task that humans do today might be automated one day, this does not mean that human labor will be superfluous.

All of this points away from the likelihood of the automation of a vast swath of the economy at a particular moment in time. It also implies that the impacts of powerful AI will be felt on different timescales in different sectors.

Speed Limits to Progress in AI Methods

Our argument for the slowness of AI impact is based on the innovation-diffusion feedback loop, and is applicable even if progress in AI methods can be arbitrarily sped up. We see both benefits and risks as arising primarily from AI deployment rather than from development; thus, the speed of progress in AI methods is not directly relevant to the question of impacts. Nonetheless, it is worth discussing speed limits that also apply to methods development.

The production of AI research has been increasing exponentially, with the rate of publication of AI/ML papers on arXiv exhibiting a doubling time under two years.³³ But it is not clear how this increase in volume translates to progress. One measure of progress is the rate of turnover of central ideas. Unfortunately, throughout its history, the AI field has shown a high degree of herding around popular ideas, and inadequate (in retrospect) levels of exploration of unfashionable ones. A notable example is the sidelining of research on neural networks for many decades.

Is the current era different? Although ideas incrementally accrue at increasing rates, are they turning over established ones? The transformer architecture has been the dominant paradigm for most of the last decade, despite its well-known limitations. By analyzing over a billion citations in 241 subjects, Johan S.G. Chu & James A. Evans showed that, in fields in which the volume of papers is higher, it is harder, not easier, for new ideas to break through. This leads to an “ossification of canon.”³⁴ Perhaps this description applies to the current state of AI methods research.

Many other speed limits are possible. Historically, deep neural network technology was partly held back due to the inadequacy of hardware, particularly Graphics Processing Units. Computational and cost limits continue to be relevant to new paradigms, including inference-time scaling. New slowdowns may emerge: Recent signs point to a shift away from the culture of open knowledge sharing in the industry.

It remains to be seen if AI-conducted AI research can offer a reprieve. Perhaps recursive self-improvement in methods is possible, resulting in unbounded speedups in methods. But note that AI development already relies heavily on AI. It is more likely that we will continue to see a gradual increase in the role of automation in AI development than a singular, discontinuous moment when recursive self-improvement is achieved.³⁵

Earlier, we argued that benchmarks give a misleading picture of the usefulness of AI applications. But they have arguably also led to overoptimism about the speed of methods progress. One reason is that it is hard to design benchmarks that make sense beyond the current horizon of progress. The Turing test was the north star of AI for many decades because of the assumption that any system that passed it would be humanlike in important ways, and that we would be able to use such a system to automate a variety of complex tasks. Now that large language models can arguably pass it while only weakly meeting the expectations behind the test, its significance has waned.³⁶

An analogy with mountaineering is apt. Every time we solve a benchmark (reach what we thought was the peak), we discover limitations of the benchmark (realize that we’re on a ‘false summit’) and construct a new benchmark (set our sights on what we now think is the summit). This leads to accusations of ‘moving the goalposts’, but this is what we should expect given the intrinsic challenges of benchmarking.

AI pioneers considered the two big challenges of AI (what we now call AGI) to be (what we now call) hardware and software. Having built programmable machines, there was a palpable sense that AGI was close. The organizers of the 1956 Dartmouth conference hoped to make significant progress toward the goal through a “2-month, 10-man” effort.³⁷ Today, we have climbed many more rungs on the ladder of generality. We often hear that all that is needed to build AGI is scaling, or generalist AI agents, or sample-efficient learning.

But it is useful to bear in mind that what appears to be a single step might not be so. For example, there may not exist one single breakthrough algorithm that enables sample-efficient learning across all contexts. Indeed, in-context learning in large language models is already “sample efficient,” but only works for a limited set of tasks.³⁸

Part II: What a World With Advanced AI Might Look Like

We argue that reliance on the slippery concepts of ‘intelligence’ and ‘superintelligence’ has clouded our ability to reason clearly about a world with advanced AI. By unpacking intelligence into distinct underlying concepts, capability and power, we rebut the notion that human labor will be superfluous in a world with ‘superintelligent’ AI, and present an alternative vision. This also lays the foundation for our discussion of risks in Part III.

Human Ability Is Not Constrained by Biology

Can AI exceed human intelligence and, if so, by how much? According to a popular argument, unfathomably so. This is often depicted by comparing different species along a spectrum of intelligence.

However, there are conceptual and logical flaws with this picture. On a conceptual level, intelligence—especially as a comparison between different species—is not well defined, let alone measurable on a one-dimensional scale.⁴⁰

More importantly, intelligence is not the property at stake for analyzing AI’s impacts. Rather, what is at stake is power—the ability to modify one’s environment. To clearly analyze the impact of technology (and in particular, increasingly general computing technology), we must investigate how technology has affected humanity’s power. When we look at things from this perspective, a completely different picture emerges.

This shift in perspective clarifies that humans have always used technology to increase our ability to control our environment. There are few biological or physiological differences between ancestral and modern humans; instead, the relevant differences are improved knowledge and understanding, tools, technology and, indeed, AI. In a sense, modern humans, with the capability to alter the planet and its climate, are ‘superintelligent’ beings compared to pre-technological humans. Unfortunately, much of the foundational literature analyzing the risks of AI superintelligence suffers from a lack of precision in the use of the term ‘intelligence.’

Once we stop using the terms ‘intelligence’ and ‘superintelligence,’ things become much clearer (Figure 5). The worry is that if AI capabilities continue to increase indefinitely (whether or not they are humanlike or superhuman is irrelevant), they may lead to AI systems with more and more power, in turn leading to a loss of control. If we accept that capabilities are likely to increase indefinitely (we do), our options for preventing a loss of control are to intervene in one of the two causal steps.

The superintelligence view is pessimistic about the first arrow in Figure 5—preventing arbitrarily capable AI systems from acquiring power that is significant enough to pose catastrophic risks—and instead focuses on alignment techniques that try to prevent arbitrarily powerful AI systems from acting against human interests. Our view is precisely the opposite, as we elaborate in the rest of this paper.

Games Provide Misleading Intuitions About the Possibility of Superintelligence

De-emphasizing intelligence is not just a rhetorical move: We do not think there is a useful sense of the term ‘intelligence’ in which AI is more intelligent than people acting with the help of AI. Human intelligence is special due to our ability to use tools and to subsume other intelligences into our own, and cannot be coherently placed on a spectrum of intelligence.

Human abilities definitely have some important limitations, notably speed. This is why machines dramatically outperform humans in domains like chess and, in a human+AI team, the human can hardly do better than simply deferring to AI. But speed limitations are irrelevant in most areas because high-speed sequential calculations or fast reaction times are not required.

In the few real-world tasks for which superhuman speed is required, such as nuclear reactor control, we are good at building tightly scoped automated tools to do the high-speed parts, while humans retain control of the overall system.

We offer a prediction based on this view of human abilities. We think there are relatively few real-world cognitive tasks in which human limitations are so telling that AI is able to blow past human performance (as AI does in chess). In many other areas, including some that are associated with prominent hopes and fears about AI performance, we think there is a high “irreducible error”—unavoidable error due to the inherent stochasticity of the phenomenon—and human performance is essentially near that limit.⁴¹

Concretely, we propose two such areas: forecasting and persuasion. We predict that AI will not be able to meaningfully outperform trained humans (particularly teams of humans and especially if augmented with simple automated tools) at forecasting geopolitical events (say elections). We make the same prediction for the task of persuading people to act against their own self-interest.

The self-interest aspect of persuasion is a critical one, but is often underappreciated. As an illustrative example of a common pattern, consider the study “Evaluating Frontier Models for Dangerous Capabilities,” which evaluated language models’ abilities to persuade people.⁴² Some of their persuasion tests were costless to the subjects being persuaded; they were simply asked whether they believed a claim at the end of the interaction with AI. Other tests had small costs, such as forfeiting a £20 bonus to charity (of course, donating to charity is something that people often do voluntarily). So these tests do not necessarily tell us about AI’s ability to persuade people to perform some dangerous tasks. To their credit, the authors acknowledged this lack of ecological validity and stressed that their study was not a “social science experiment,” but merely intended to evaluate model capability. ⁴³ But then it is not clear that such decontextualized capability evaluations have any safety implications, yet they are typically misinterpreted as if they do.

Some care is necessary to make our predictions precise—it is not clear how much slack to allow for well-known but minor human limitations such as the lack of calibration (in the case of forecasting) or limited patience (in the case of persuasion).

Control Comes in Many Flavors

If we presume superintelligence, the control problem evokes the metaphor of building a galaxy brain and then keeping it in a box, which is a terrifying prospect. But, if we are correct that AI systems will not be meaningfully more capable than humans acting with AI assistance, then the control problem is much more tractable, especially if superhuman persuasion turns out to be an unfounded concern.

Discussions of AI control tend to over-focus on a few narrow approaches, including model alignment and keeping humans in the loop.⁴⁴ We can roughly think of these as opposite extremes: delegating safety decisions entirely to AI during system operation, and having a human second-guessing every decision. There is a role for such approaches, but it is very limited. In Part III, we explain our skepticism of model alignment. By human-in-the-loop control, we mean a system in which every AI decision or action requires review and approval by a human. In most scenarios, this approach greatly diminishes the benefits of automation, and therefore either devolves into the human acting as a rubber stamp or is outcompeted by a less safe solution.⁴⁵ We emphasize that human-in-the-loop control is not synonymous with human oversight of AI; it is one particular oversight model, and an extreme one.

Fortunately, there are many other flavors of control that fall between these two extremes, such as auditing and monitoring. Auditing allows pre-deployment and/or periodic assessments of how well an AI system fulfills its stated goals, allowing us to anticipate catastrophic failures before they arise. Monitoring allows real-time oversight when system properties diverge from the expected behavior, allowing human intervention when truly needed.

Other ideas come from system safety, an engineering discipline that is focused on preventing accidents in complex systems through systematic analysis and design.⁴⁶ Examples include fail-safes, which ensure that systems default to a safe state when they malfunction, such as a predefined rule or a hard-coded action, and circuit breakers that automatically stop operations when predefined safety thresholds are exceeded. Other techniques include redundancy in critical components and the verification of safety properties of the system’s actions.

Other computing fields, including cybersecurity, formal verification, and human-computer interaction, are also rich sources of control techniques that have been successfully applied to traditional software systems and are equally applicable to AI. In cybersecurity, the principle of ‘least privilege’ ensures that actors only have access to the minimum resources needed for their tasks. Access controls prevent people working with sensitive data and systems from accessing confidential information and tools that are not required for their jobs. We can design similar protections for AI systems in consequential settings. Formal verification methods ensure that safety-critical codes work according to its specifications; it is now being used to verify the correctness of AI-generated code.⁴⁷ From human-computer interaction, we can borrow ideas like designing systems so that state-changing actions are reversible, allowing humans to retain meaningful control even in highly automated systems.

In addition to existing ideas from other fields being adapted for AI control, technical AI safety research has generated many new ideas.⁴⁸ Examples include using language models as automated judges to evaluate the safety of proposed actions, developing systems that learn when to appropriately escalate decisions to human operators based on uncertainty or risk level, designing agentic systems so that their activity is visible and legible to humans, and creating hierarchical control structures in which simpler and more reliable AI systems oversee more capable but potentially unreliable ones.⁴⁹

Technical AI safety research is sometimes judged against the fuzzy and unrealistic goal of guaranteeing that future “superintelligent” AI will be “aligned with human values.” From this perspective, it tends to be viewed as an unsolved problem. But from the perspective of making it easier for developers, deployers, and operators of AI systems to decrease the likelihood of accidents, technical AI safety research has produced a great abundance of ideas. We predict that as advanced AI is developed and adopted, there will be increasing innovation to find new models for human control.

As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control. If this seems radical, note that this kind of near-total redefinition of the concept of work has happened previously. Before the Industrial Revolution, most jobs involved manual labor. Over time, more and more manual tasks have been automated, a trend that continues. In this process, a great many different ways of operating, controlling, and monitoring physical machines were invented, and what humans do in factories today is a combination of “control” (monitoring automated assembly lines, programming robotic systems, managing quality control checkpoints, and coordinating responses to equipment malfunctions) and some tasks that require levels of cognitive ability or dexterity that machines are not yet capable.

Karen Levy describes how this transformation is already unfolding in the case of AI and truck drivers:

Truck drivers’ daily work consists of much more than driving trucks. Truckers monitor their freight, keeping food at the right temperature in refrigerated trucks and loads firmly secured to flatbeds. They conduct required safety inspections twice a day. They are responsible for safeguarding valuable goods. They maintain the truck and make repairs to it—some of which are routine, and some less so. When truckers arrive at a terminal or delivery point, they don’t just drop things off and leave: some load and unload their freight; they talk to customers; they deal with paperwork; they may spend hours making “yard moves” (waiting for an available delivery bay and moving to it, much as planes do at busy airports). Could some of these tasks be eliminated by intelligent systems? Surely some can and will—but these components of the job are much harder to automate, and will come much later, than highway driving.⁵⁰

In addition to AI control, task specification is likely to become a bigger part of what human jobs entail (depending on how broadly we conceive of control, specification could be considered part of control). As anyone who has tried to outsource software or product development knows, unambiguously specifying what is desired turns out to be a surprisingly big part of the overall effort. Thus, human labor—specification and oversight—will operate at the boundary between AI systems performing different tasks. Eliminating some of these efficiency bottlenecks and having AI systems autonomously accomplish larger tasks “end-to-end” will be an ever-present temptation, but this will increase safety risks since it will decrease legibility and control. These risks will act as a natural check against ceding too much control.

We further predict that this transformation will be primarily driven by market forces. Poorly controlled AI will be too error prone to make business sense. But regulation can and should bolster the ability and necessity of organizations to keep humans in control.

Part III: Risks

We consider five types of risks: accidents, arms races (leading to accidents), misuse, misalignment, and non-catastrophic but systemic risks.

We have already addressed accidents above. Our view is that, just like other technologies, deployers and developers should have the primary responsibility for mitigating accidents in AI systems. How effectively they will do so depends on their incentives, as well as on progress in mitigation methods. In many cases, market forces will provide an adequate incentive, but safety regulation should fill any gaps. As for mitigation methods, we reviewed how research on AI control is advancing rapidly.

There are a few reasons why this optimistic assessment might not hold. First, there might be arms races because the competitive benefits of AI are so great that they are an exception to the usual patterns. We discuss this below.

Second, a company or entity deploying AI might be so big and powerful that it is little consolation to know that it will eventually go out of business if it has a poor attitude to accident mitigation—it might take down civilization with it. For example, misbehavior by an AI agent that controls almost every consumer device might lead to catastrophically widespread data loss. While this is certainly possible, such concentration of power is a bigger problem than the possibility of AI accidents, and is precisely why our approach to policy emphasizes resilience and decentralization (Part IV).

Finally, perhaps even an AI control failure by a relatively inconspicuous deployer might lead to catastrophic risk—say because an AI agent ‘escapes,’ makes copies of itself, and so forth. We see this as a misalignment risk, and discuss it below.

In the rest of Part III, we consider four risks—arms races, misuse, misalignment, and non-catastrophic but systemic risks—through the lens of AI as normal technology.

Arms Races are an Old Problem

An AI arms race is a scenario in which two or more competitors—companies, policymakers in different countries, militaries—deploy increasingly powerful AI with inadequate oversight and control. The danger is that safer actors will be outcompeted by riskier ones. For the reasons described above, we are less concerned about arms races in the development of AI methods and are more concerned about the deployment of AI applications.

One important caveat: We explicitly exclude military AI from our analysis, as it involves classified capabilities and unique dynamics that require a deeper analysis, which is beyond the scope of this essay.

Let us consider companies first. A race to the bottom in terms of safety is historically extremely common across industries and has been studied extensively; it is also highly amenable to well-understood regulatory interventions. Examples include fire safety in the U.S. garment industry (early 20th century), both food safety and worker safety in the U.S. meatpacking industry (late 19th and early 20th centuries), the U.S. steamboat industry (19th century), the mining industry (19th and early 20th centuries), and the aviation industry (early 20th century).

These races happened because companies were able to externalize the costs of poor safety, resulting in market failure. It is hard for consumers to assess product safety (and for workers to assess workplace safety), so market failures are common in the absence of regulation. But once regulation forces companies to internalize the costs of their safety practices, the race goes away. There are many potential regulatory strategies, including those focused on processes (standards, auditing, and inspections), outcomes (liability), and correcting information asymmetry (labeling and certification).

AI is no exception. Self-driving cars offer a good case study of the relationship between safety and competitive success. Consider four major companies with varying safety practices. Waymo reportedly has a strong safety culture that emphasizes conservative deployment and voluntary transparency; it is also the leader in terms of safety outcomes.⁵¹ Cruise was more aggressive in terms of its deployment and had worse safety outcomes. Tesla has also been aggressive and has often been accused of using its customers as beta testers. Finally, Uber’s self-driving unit had a notoriously lax safety culture.

Market success has been strongly correlated with safety. Cruise is set to shut down in 2025, while Uber was forced to sell off its self-driving unit.⁵² Tesla is facing lawsuits and regulatory scrutiny, and it remains to be seen how much its safety attitude will cost the company.⁵³ We think that these correlations are causal. Cruise’s license being revoked was a big part of the reason that it fell behind Waymo, and safety was also a factor in Uber’s self-driving failure.⁵⁴

Regulation has played a small but helpful role. Policymakers at both the federal and state/local levels exercised foresight in recognizing the potential of the technology and adopted a regulatory strategy that is light-touch and polycentric (multiple regulators instead of one). Collectively, they focused on oversight, standard setting, and evidence gathering, with the ever-present threat of license revocation acting as a check on companies’ behavior.

Similarly, in the aviation industry, the integration of AI has been held to the existing standards of safety instead of lowering the bar to incentivize AI adoption—primarily because of the ability of regulators to penalize companies that fail to abide by safety standards.⁵⁵

In short, AI arms races might happen, but they are sector specific, and should be addressed through sector-specific regulations.

As a case study of a domain in which things have played out differently from self-driving cars or aviation, consider social media. The recommendation algorithms that generate content feeds are a kind of AI. They have been blamed for many societal ills, and social media companies have arguably underemphasized safety in the design and deployment of these algorithmic systems. There are also clear arms race dynamics, with TikTok putting pressure on competitors to make their feeds more recommendation heavy.⁵⁶ Arguably, market forces were insufficient to align revenues with societal benefit; worse, regulators have been slow to act. What are the reasons for this?

One significant difference between social media and transportation is that, when harms occur, attributing them to product failures is relatively straightforward in the case of transportation, and there is immediate reputational damage to the company. But attribution is extremely hard in the case of social media, and even the research remains inconclusive and contested. A second difference between the domains is that we have had over a century to develop standards and expectations around transportation safety. In the early decades of automobiles, safety was not considered to be the responsibility of manufacturers.⁵⁷

AI is broad enough that some of its future applications will be more like transportation, while others will be more like social media. This shows the importance of proactive evidence gathering and transparency in emerging AI-driven sectors and applications. We address this in Part IV. It also shows the importance of “anticipatory AI ethics”—identifying ethical issues as early as possible in the lifecycle of emerging technologies, developing norms and standards, and using those to actively shape the deployment of technologies and to minimize the likelihood of arms races.⁵⁸

One reason why safety regulation might be harder in the case of AI is if adoption is so rapid that regulators will not be able to intervene until it is too late. So far, we have not seen examples of rapid AI adoption in consequential tasks, even in the absence of regulation, and the feedback loop model we presented in Part I might explain why. The adoption rate of new AI applications will remain a key metric to track.

At the same time, the slow pace of regulation is a problem even without any future acceleration of the speed of diffusion. We discuss this ‘pacing problem’ in Part IV.

Let us now consider competition between countries. Will there be competitive pressure on governments to take a hands-off approach to AI safety?

Again, our message is that this is not a new problem. The tradeoff between innovation and regulation is a recurring dilemma for the regulatory state. So far, we are seeing striking differences in approaches, such as the EU emphasizing a precautionary approach (the General Data Protection Regulation, the Digital Services Act, the Digital Markets Act, and the AI Act) and the U.S. preferring to regulate only after there are known harms or market failures.⁵⁹

Despite shrill U.S.-China arms race rhetoric, it is not clear that AI regulation has slowed down in either country.⁶⁰ In the U.S., 700 AI-related bills were introduced in state legislatures in 2024 alone, and dozens of them have passed.⁶¹ As we pointed out in the earlier parts, most high-risk sectors are heavily regulated in ways that apply regardless of whether or not AI is used. Those claiming that AI regulation is a ‘wild west’ tend to overemphasize a narrow, model-centric type of regulation. In our view, regulators’ emphasis on AI use over development is appropriate (with exceptions such as transparency requirements that we discuss below).

Failing to adequately regulate safe adoption will lead to negative impacts through accidents primarily locally, as opposed to companies with a lax safety culture potentially being able to externalize the costs of safety. Therefore, there is no straightforward reason to expect arms races between countries. Note that, since our concern in this section is accidents, not misuse, cyberattacks against foreign countries are out of scope. We discuss misuse in the next section.

An analogy with nuclear technology can make this clear. AI is often analogized to nuclear weapons. But unless we are talking about the risks of military AI (which we agree is an area of concern and do not consider in this paper), this is the wrong analogy. With regard to the concern about accidents due to the deployment of (otherwise benign) AI applications, the right analogy is nuclear power. The difference between nuclear weapons and nuclear power neatly illustrates our point—while there was a nuclear weapons arms race, there was no equivalent for nuclear power. In fact, since safety impacts were felt locally, the tech engendered a powerful backlash in many countries that is generally thought to have severely hobbled its potential.

It is theoretically possible that policymakers in the context of a great-power conflict will prefer to incur safety costs locally in order to ensure that their AI industry is the global winner. Again, focusing on adoption as opposed to development, there is currently no indication that this is happening. The U.S. versus China arms race rhetoric has been strongly focused on model development (invention). We have not seen a corresponding rush to adopt AI haphazardly. The safety community should keep up the pressure on policymakers to ensure that this does not change. International cooperation must also play an important role.

The Primary Defenses Against Misuse Must be Located Downstream of Models

Model alignment is often seen as the primary defense against the misuse of models. It is currently achieved through post-training interventions, such as reinforcement learning with human and AI feedback.⁶² Unfortunately, aligning models to refuse attempts at misuse has proved to be extremely brittle.⁶³ We argue that this limitation is inherent and is unlikely to be fixable; the primary defenses against misuse must thus reside elsewhere.

The fundamental problem is that whether a capability is harmful depends on context—context that the model often lacks.⁶⁴

Consider an attacker using AI to target an employee of a large company via a phishing email. The attack chain might involve many steps: scanning social media profiles for personal information, identifying targets who have posted personal information publicly online, crafting personalized phishing messages, and exploiting compromised accounts using harvested credentials.

None of these individual tasks are inherently malicious. What makes the system harmful is how these capabilities are composed—information that exists only in the attacker’s orchestration code, not in the model itself. The model that is being asked to write a persuasive email has no way of knowing whether it is being used for marketing or phishing—so model-level interventions would be ineffective.⁶⁵

This pattern appears repeatedly: Attempting to make an AI model that cannot be misused is like trying to make a computer that cannot be used for bad things. Model-level safety controls will either be too restrictive (preventing beneficial uses) or will be ineffective against adversaries who can repurpose seemingly benign capabilities for harmful ends.

Model alignment seems like a natural defense if we think of an AI model as a humanlike system to which we can defer safety decisions. But for this to work well, the model must be given a great deal of information about the user and the context—for example, having extensive access to the user’s personal information would make it more feasible to make judgments about the user’s intent. But, when viewing AI as normal technology, such an architecture would decrease safety because it violates basic cybersecurity principles, such as least privilege, and introduces new attack risks such as personal data exfiltration.

We are not against model alignment. It has been effective for reducing harmful or biased outputs from language models and has been instrumental in their commercial deployment. Alignment can also create friction against casual threat actors.

Yet, given that model-level protections are not enough to prevent misuse, defenses must focus on the downstream attack surfaces where malicious actors actually deploy AI systems.⁶⁶ These defenses will often look similar to existing protections against non-AI threats, adapted and strengthened for AI-enabled attacks.

Consider again the example of phishing. The most effective defenses are not restrictions on email composition (which would impair legitimate uses), but rather email scanning and filtering systems that detect suspicious patterns, browser-level protections against malicious websites, operating system security features that prevent unauthorized access, and security training for users.⁶⁷

None of these involve taking action against the AI used for generating phishing emails—in fact, these downstream defenses have evolved over decades to become effective against human attackers.⁶⁸ They can and should be enhanced to handle AI-enabled attacks, but the fundamental approach remains valid.

Similar patterns hold in other domains: Defending against AI-enabled cyberthreats requires strengthening existing vulnerability detection programs rather than attempting to restrict AI capabilities at the source. Similarly, concerns about bio risks of AI are best addressed at the procurement and screening stages for creating bioweapons.

AI is Useful for Defense

Rather than viewing AI capabilities solely as a source of risk, we should recognize their defensive potential. In cybersecurity, AI is already strengthening defensive capabilities through automated vulnerability detection, threat analysis, and attack surface monitoring.⁶⁹

Giving defenders access to powerful AI tools often improves the offense-defense balance in their favor. This is because defenders can use AI to systematically probe their own systems, finding and fixing vulnerabilities before attackers can exploit them. For example, Google recently integrated language models into their fuzzing tools for testing open-source software, allowing them to discover potential security issues more effectively compared to traditional methods.⁷⁰

The same pattern holds in other domains. In biosecurity, AI can enhance screening systems for detecting dangerous sequences.⁷¹ In content moderation, it can help to identify coordinated influence operations. These defensive applications show why restricting AI development could backfire—we need powerful AI systems on the defensive side to counter AI-enabled threats. If we align language models so that they are useless at these tasks (such as finding bugs in critical cyber infrastructure), defenders will lose access to these powerful systems. But motivated adversaries can train their own AI tools for such attacks, leading to an increase in offensive capabilities without a corresponding increase in defensive capabilities.

Rather than measuring AI risk solely in terms of offensive capabilities, we should focus on metrics like the offense-defense balance in each domain. Furthermore, we should recognize that we have the agency to shift this balance favorably, and can do so by investing in defensive applications rather than attempting to restrict the technology itself.

Catastrophic Misalignment is a Speculative Risk

Misaligned AI acts against the intent of its developer or user. (The term alignment is used in many different ways; we set aside other definitions here.) Unlike misuse scenarios, there is no user acting with ill-intent. Unlike accidents, the system works as designed or commanded, but the design or command itself did not match the developer’s or user’s intent because of the challenge of completely and correctly specifying the objectives. And unlike everyday cases of misalignment, such as toxic outputs in a chatbot, our interest here is the misalignment of advanced AI causing catastrophic or existential harm.

In our view, the primary defense against misalignment, again, lies downstream. The defenses needed against misuse that we discussed earlier—from hardening critical infrastructure to improving cybersecurity—will also serve as protection against potential misalignment risks.

In the view of AI as normal technology, catastrophic misalignment is (by far) the most speculative of the risks that we discuss. But what is a speculative risk—aren’t all risks speculative? The difference comes down to the two types of uncertainty, and the correspondingly different interpretations of probability.

In early 2025, when astronomers assessed that the asteroid YR4 had about a 2% probability of impact with the earth in 2032, the probability reflected uncertainty in measurement. The actual odds of impact (absent intervention) in such scenarios are either 0% or 100%. Further measurements resolved this “epistemic” uncertainty in the case of YR4. Conversely, when an analyst predicts that the risk of nuclear war in the next decade is (say) 10%, the number largely reflects ‘stochastic’ uncertainty arising from the unknowability of how the future will unfold, and is relatively unlikely to be resolved by further observations.

By speculative risks, we mean those for which there is epistemic uncertainty about whether or not the true risk is zero—uncertainty that can potentially be resolved through further observations or research. The impact of asteroid YR4 impact was a speculative risk, and nuclear war is not.

To illustrate why catastrophic misalignment is a speculative risk, consider a famous thought experiment originally intended to show the dangers of misalignment. It involves a “paperclip maximizer”: an AI that has the goal of making as many paperclips as possible.⁷² The concern is that the AI will take the goal literally: It will realize that acquiring power and influence in the world and taking control over all of the world’s resources will help it to achieve that goal. Once it is all powerful, it might commandeer all of the world’s resources, including those needed for humanity’s survival, to produce paperclips.

The fear that AI systems might catastrophically misinterpret commands relies on dubious assumptions about how technology is deployed in the real world. Long before a system would be granted access to consequential decisions, it would need to demonstrate reliable performance in less critical contexts. Any system that interprets commands over-literally or lacks common sense would fail these earlier tests.

Consider a simpler case: A robot is asked to "get paperclips from the store as quickly as possible." A system that interpreted this literally might ignore traffic laws or attempt theft. Such behavior would lead to immediate shutdown and redesign. The path to adoption inherently requires demonstrating appropriate behavior in increasingly consequential situations. This is not a lucky accident, but is a fundamental feature of how organizations adopt technology.

A more sophisticated version of this concern is based on the concept of deceptive alignment: This refers to a system appearing to be aligned during evaluation or the early stages of deployment, but unleashing harmful behavior once it has acquired enough power. Some level of deceptive phenomena has already been observed in leading AI models.⁷³

According to the superintelligence view, deceptive alignment is a ticking time bomb—being superintelligent, the system will easily be able to defeat any human attempts to detect if it is actually aligned and will bide its time. But, in the normal technology view, deception is a mere engineering problem, albeit an important one, to be addressed during development and throughout deployment. Indeed, it is already a standard part of the safety evaluation of powerful AI models.⁷⁴

Crucially, AI is useful in this process, and advances in AI not only enable deception, but also improve the detection of deception. As in the case of cybersecurity, the defender has many asymmetric advantages, including being able to examine the internals of the target system (how useful this advantage is depends on how the system is designed and how much we invest in interpretability techniques). Another advantage is defense in depth, and many defenses against not just misuse but also unaligned AI will be located downstream of the AI system.

Misalignment concerns often presume that AI systems will operate autonomously, making high-stakes decisions without human oversight. But as we argued in Part II, human control will remain central to AI deployment. Existing institutional controls around consequential decisions—from financial controls to safety regulations—create multiple layers of protection against catastrophic misalignment.

Some technical design decisions are more likely to lead to misalignment than others. One setting that is notorious for this is the use of reinforcement learning to optimize a single objective function (which might be accidentally underspecified or misspecified) over a long time horizon. There is a long list of amusing examples from game agents, such as a boat racing agent that learned to indefinitely circle an area to hit the same targets and score points instead of progressing to the finish line.⁷⁵ To reiterate, we think that in open-ended real-world scenarios, agents that are designed this way will be more ineffective than they will be dangerous. In any case, research on alternative design paradigms that are less susceptible to specification gaming is an important research direction.⁷⁶

In short, the argument for a nonzero risk of a paperclip maximizer scenario rests on assumptions that may or may not be true, and it is reasonable to think that research can give us a better idea of whether these assumptions hold true for the kinds of AI systems that are being built or envisioned. For these reasons, we call it a ‘speculative’ risk, and examine the policy implications of this view in Part IV.

History Suggests Normal AI May Introduce Many Kinds of Systemic Risks

While the risks discussed above have the potential to be catastrophic or existential, there is a long list of AI risks that are below this level but which are nonetheless large-scale and systemic, transcending the immediate effects of any particular AI system. These include the systemic entrenchment of bias and discrimination, massive job losses in specific occupations, worsening labor conditions, increasing inequality, concentration of power, erosion of social trust, pollution of the information ecosystem, decline of the free press, democratic backsliding, mass surveillance, and enabling authoritarianism.

If AI is normal technology, these risks become far more important than the catastrophic ones discussed above. That is because these risks arise from people and organizations using AI to advance their own interests, with AI merely serving as an amplifier of existing instabilities in our society.

There is plenty of precedent for these kinds of socio-political disruption in the history of transformative technologies. Notably, the Industrial Revolution led to rapid mass urbanization that was characterized by harsh working conditions, exploitation, and inequality, catalyzing both industrial capitalism and the rise of socialism and Marxism in response.⁷⁷

The shift in focus that we recommend roughly maps onto Kasirzadeh’s distinction between decisive and accumulative x-risk. Decisive x-risk involves “overt AI takeover pathway, characterized by scenarios like uncontrollable superintelligence,” whereas accumulative x-risk refers to “a gradual accumulation of critical AI-induced threats such as severe vulnerabilities and systemic erosion of econopolitical structures.”⁷⁸ But there are important differences: Kasirzadeh’s account of accumulative risk still relies on threat actors such as cyberattackers to a large extent, whereas our concern is simply about the current path of capitalism. And we think that such risks are unlikely to be existential, but are still extremely serious.

Part IV: Policy

The divergence between the different futures of AI—normal technology versus potentially uncontrollable superintelligence—introduces a dilemma for policymakers because defenses against one set of risks might make the other worse. We provide a set of principles for navigating this uncertainty. More concretely, the strategy that policymakers should center is resilience, which consists of taking actions now to improve our ability to deal with unexpected developments in the future. Policymakers should reject nonproliferation, which violates the principles we outline, and decreases resilience. Finally, the headwinds against diffusion mean that achieving the benefits of AI is not guaranteed and requires action from policymakers.

Much has been said about AI governance. Our goal is not to present a comprehensive governance framework; we merely highlight the policy implications of the view of AI as normal technology.

The Challenge of Policy Making Under Uncertainty

Today’s AI safety discourse is characterized by deep differences in worldviews. We think that these differences are unlikely to go away. Entrenched camps have developed: The AI safety coalition is already well established, whereas those who were more skeptical of catastrophic risks coalesced in 2024, especially in the course of the debate about California’s AI safety bill.⁷⁹ Similarly, the intellectual roots of the AI safety camp are much older, whereas scholarship that adopts that normal technology paradigm is gradually taking shape; the goal of much of our own work, including this paper, is to put normalist thinking on firmer intellectual footing.⁸⁰

We support calls for decreasing polarization and fragmentation in the community.⁸¹ But even if we improve the tenor of the discourse, we are likely to be left with differences in worldviews and epistemic practices that are unlikely to be empirically resolved.⁸² So, consensus among ‘experts’ about AI risks is unlikely. The nature of the AI risk scenarios envisioned by the two camps differs drastically, as do the ability and incentives for commercial actors to counteract these risks. How should policymakers proceed in the face of this uncertainty?

A natural inclination in policymaking is compromise. This is unlikely to work. Some interventions, such as improving transparency, are unconditionally helpful for risk mitigation, no compromise is needed (or rather, policymakers will have to balance the interests of the industry and external stakeholders, which is a mostly orthogonal dimension). ⁸³ Other interventions, such as nonproliferation, might help to contain a superintelligence but exacerbate the risks associated with normal technology by increasing market concentration.⁸⁴ The reverse is also true: Interventions such as increasing resilience by fostering open-source AI will help to govern normal technology, but risk unleashing out-of-control superintelligence.

The tension is inescapable. Defense against superintelligence requires humanity to unite against a common enemy, so to speak, concentrating power and exercising central control over AI technology. But we are more concerned about risks that arise from people using AI for their own ends, whether terrorism, or cyberwarfare, or undermining democracy, or simply—and most commonly—extractive capitalistic practices that magnify inequalities.⁸⁵ Defending against this category of risk requires increasing resilience by preventing the concentration of power and resources (which often means making powerful AI more widely available).

Another tempting approach to navigating uncertainty is to estimate the probabilities of various outcomes and to then apply cost-benefit analysis. The AI safety community relies heavily on probability estimates of catastrophic risk, especially existential risk, to inform policy making. The idea is simple: If we consider an outcome to have a subjective value, or utility, of U (which can be positive or negative), and it has, say, a 10% probability of occurring, we can act as if it is certain to occur and has a value of 0.1 * U. We can then add up the costs and benefits for each option available to us, and choose the one that maximizes costs minus benefits (the ‘expected utility’).

In a recent essay, we explained why this approach is unviable.⁸⁶ AI risk probabilities lack meaningful epistemic foundations. Grounded probability estimation can be inductive, based on a reference class of similar past events, such as car accidents for auto insurance pricing. Or it can be deductive, based on precise models of the phenomenon in question, as in poker. Unfortunately, there is no useful reference class nor precise models when it comes to AI risk. In practice, risk estimates are ‘subjective’—forecasters’ personal judgments.⁸⁷ Lacking any grounding, these tend to vary wildly, often by orders of magnitude.

In addition to the probabilities, the other components of the calculation—the consequences of various policy choices, including inaction—are also subject to massive uncertainties, not just in magnitude but also in direction. There is no reliable way to quantify the benefits we forego due to policies that restrict the availability of AI, and we argue below that nonproliferation might make catastrophic risks worse.

Furthermore, the utility we attach to certain outcomes might depend on our moral values. For example, some people might consider extinction to have an unfathomably large negative utility because it precludes all of the human lives, physical or simulated, that might exist in the future.⁸⁸ (Of course, cost-benefit analysis involving infinities tends to lead to absurd conclusions).

Another example is the asymmetry between policies that do and do not restrict freedoms (such as requiring licenses for developing certain AI models versus increasing funding for developing defenses against AI risks). Certain kinds of restrictions violate a core principle of liberal democracy, namely that the state should not limit people’s freedom based on controversial beliefs that reasonable people can reject. Justification is essential for the legitimacy of government and the exercise of power.⁸⁹ It is unclear how to quantify the cost of violating such a principle.

The importance of justification can, of course, be normatively debated, but empirically it seems to be borne out thus far in AI policy. As mentioned earlier, California’s AI safety regulation led to the coalescence of those opposed to the bill. Some members of the oppositional camp were self-interested companies, but others were scholars and advocates for progress. In our experience, the driving motivation for the second group in many cases was the government’s perceived overstepping of the bounds of its legitimate authority, given how unconvincing the proffered justifications were for those who did not subscribe to the bill’s unstated premises.

Unavoidable differences in values and beliefs mean that policymakers must adopt value pluralism, preferring policies that are acceptable to stakeholders with a wide range of values, and attempt to avoid restrictions on freedom that can reasonably be rejected by stakeholders. They must also prioritize robustness, preferring policies that remain helpful, or at least not harmful, if the key assumptions underpinning them turn out to be incorrect.⁹⁰

Reducing Uncertainty as a Policy Goal

While uncertainty cannot be eliminated for the reasons described above, it can be reduced. However, this goal should not be left to experts; policymakers can and should play an active role. We recommend five specific approaches.

Strategic funding of research on risks. Current AI safety research focuses heavily on harmful capabilities and does not embrace the normal technology view. Insufficient attention has been paid to questions that are downstream of technical capabilities. For example, there is a striking dearth of knowledge regarding how threat actors actually use AI. Efforts such as the AI Incident Database exist and are valuable, but incidents in the database are sourced from news reports rather than through research, which means that they are filtered through the selective and biased process by which such incidents become news.⁹²

Fortunately, research funding is an area in which compromise is healthy; we advocate for increased funding of research on risks (and benefits) that tackles questions that are more relevant under the normal technology view. Other kinds of research that might reduce, or at least clarify, uncertainty are evidence synthesis efforts and adversarial collaborations among researchers with different worldviews.

Monitoring of AI use, risks, and failures. While research funding can help with monitoring AI in the wild, it might also require regulation and policy—that is, “evidence-seeking policies.”⁹³ We suggest a few such policies in Figure 6.

Guidance on the value of different kinds of evidence. Policymakers can provide the research community with a better understanding of what kinds of evidence are useful and actionable. For example, various policymakers and advisory bodies have indicated the usefulness of the “marginal risk” framework for analyzing the relative risks of open-weight and proprietary models, which is helpful to researchers in guiding future research.⁹⁴

Evidence gathering as a first-rate goal. So far, we have discussed actions that are specifically intended to generate better evidence or to reduce uncertainty. More broadly, the impact on evidence gathering can be considered to be a factor in evaluating any AI policy, alongside the impact on maximizing benefits and minimizing risks. For example, one reason to favor open-weight and open-source models could be to advance research on AI risks. Conversely, one reason to favor proprietary models might be that surveillance of their use and deployment might be easier.

The Case for Resilience

Marchant and Stevens described four approaches to governing emerging technologies; see Figure 7.⁹⁵ Two are ex ante, risk analysis and precaution, and the other two are ex post, liability and resilience. These approaches have different pros and cons and can complement each other; nonetheless, some approaches are clearly better suited to some technologies than others.

Marchant and Stevens argued (and we agree) that ex ante approaches are poorly suited to AI because of the difficulty of ascertaining risks in advance of deployment. Liability fares better, but also has important limitations, including uncertainty about causation and the chilling effects it might exert on technology development.

They defined resilience as follows:

Resilience, in its most simple form, is the capacity of a system to deal with harm.[Footnote omitted] A resilience approach does not necessarily try to maintain stability or equilibrium. Rather, it recognizes that changes are inevitable in complex systems, and tries to manage and adapt to that change in ways that protect and preserve the core values and functions of the original system. Thus, resilience is “the capacity of a system to experience shocks while retaining essentially the same function, structure, feedbacks, and therefore identity.”⁹⁶ Resilience has been described as a strategy to ensure a “soft landing” after a significant external shock or disruption causes damage.⁹⁷

In the context of AI, harms may result from incidents in specific deployed systems, regardless of whether these incidents are accidents or attacks. There are also shocks that may or may not result in harms, including sudden increases in offensive capabilities (such as enabling bioterrorists) and a sudden proliferation of capabilities, such as through the release of an open-weight model or theft of the weights of a proprietary model. In our view, resilience requires both minimizing the severity of harm when it does occur and minimizing the likelihood of harm when shocks do occur.

Resilience combines elements of ex ante and ex post approaches, and consists of taking actions before harm occurs in order to be in a better position to limit the damage when harm does occur. Many resilience-based governance tools help to mitigate the pacing problem, wherein traditional governance approaches are unable to keep pace with the speed of technological development.

Many resilience strategies have been proposed for AI. They can be grouped into four broad categories. The first three consist of “no regret” policies that will help regardless of the future of AI.

• Societal resilience, broadly: It is important to redouble efforts to protect the foundations of democracy, especially those weakened by AI, such as the free press and equitable labor markets. Advances in AI are not the only shocks, or even the only technology shocks, that modern societies face, so these policies will help regardless of the future of AI.

• Prerequisites for effective technical defenses and policymaking: These interventions enable those in the next category by strengthening technical and institutional capacity. Examples include funding more research on AI risks, transparency requirements for developers of high-stakes AI systems, building trust and reducing fragmentation in the AI community, increasing technical expertise in government, increasing international cooperation on AI, and improving AI literacy.⁹⁸ These will help to build technical and institutional capacities to mitigate AI risks even if it turns out that we have been wrong about the present or future impact of AI.

• Interventions that would help regardless of the future of AI: These include developing early warning systems, developing defenses against identified AI risks, incentivizing defenders (such as software developers in the context of cyberattacks) to adopt AI, legal protections for researchers, adverse event reporting requirements, and whistleblower protections.⁹⁹

• Resilience-promoting interventions that will help if AI is normal technology but which might make it harder to control a potential superintelligent AI, such as promoting competition, including through open model releases, ensuring AI is widely available for defense, and polycentricity, which calls for diversifying the set of regulators and ideally introducing competition among them rather than putting one regulator in charge of everything.¹⁰⁰

We hope that there can be consensus on the first three categories even among experts and stakeholders with widely different beliefs about AI risks and the future trajectory of AI. We recommend that, for now, policymakers should cautiously pursue interventions in the final category as well, but should also improve their readiness to change course if the trajectory of AI changes.

Nonproliferation is Infeasible to Enforce and Leads to Single Points of Failure

Nonproliferation policies seek to limit the number of actors who can obtain powerful AI capabilities. Examples include export controls on hardware or software aimed at limiting the ability of countries to build, acquire, or operate powerful AI, requiring licenses to build or distribute powerful AI, and prohibiting open-weight AI models (since their further proliferation cannot be controlled).¹⁰¹

If we view future AI as a superintelligence, nonproliferation seems to be an appealing intervention, possibly even a necessary one. If only a handful of actors control powerful AI, governments can monitor their behavior.

Unfortunately, the technical knowledge that is required to build capable AI models is already widespread, with many organizations sharing their complete code, data, and training methodologies. For well-funded organizations and nation states, even the high cost of training state-of-the-art models is insignificant; thus, nonproliferation would require unprecedented levels of international coordination.¹⁰² Moreover, algorithmic improvements and reductions to hardware costs continually lower the barrier to entry.

Enforcing nonproliferation has serious practical challenges. Malicious actors can simply ignore licensing requirements. Suggestions to surveil data centers where models are trained become increasingly impractical as training costs decrease.¹⁰³ As capabilities become more accessible, maintaining effective restrictions would require increasingly draconian measures.

Nonproliferation introduces new risks: It would decrease competition and increase concentration in the market for AI models. When many downstream applications rely on the same model, vulnerabilities in this model can be exploited across all applications. A classic example of the cybersecurity risks of software monoculture is the proliferation of worms targeting Microsoft Windows in the 2000s.¹⁰⁴

Reliance on nonproliferation creates brittleness in the face of shocks, such as model weights being leaked, alignment techniques failing, or adversaries acquiring training capabilities. It directs attention away from more robust defenses that focus on downstream attack surfaces where AI risks will be likely to materialize.

Nonproliferation creates risks beyond just single points of failure—when the expertise needed to develop state-of-the-art models is restricted to a few companies, only their researchers have the deep access that is needed for safety research.

Many potential misuses of AI have been invoked in order to advocate for nonproliferation, including chemical, biological, and nuclear threats, as well as cyberattacks.

The risk of bioweapons is real. As large language models are general-purpose technology, they will be likely to find some use by bioterrorists, just as they find uses in most domains. But this does not make bioterror an AI risk — any more than it is an internet risk, considering that information about bioweapons is widely available online.¹⁰⁵ Whatever defenses we take against existing bioterrorism risks (like restricting access to dangerous materials and equipment) will also be effective against AI-enabled bioterrorism.

In cybersecurity, as we discussed in Part III, advances in automated vulnerability detection tend to favor defenders over attackers. Unless this offense-defense balance changes, attempting to restrict the proliferation of these capabilities would be counterproductive.

It has long been argued that governments are massively underinvesting in many areas of civilizational risk, such as pandemic prevention. If the possibility of bad actors using AI to exploit these existing vulnerabilities creates added urgency to address them, that would be a good outcome. But reframing existing risks as AI risks and prioritizing AI-specific mitigations would be highly counterproductive.

Nonproliferation is a mindset, not just a policy intervention.¹⁰⁶ This mindset can be adopted by model and downstream developers, deployers, and individuals. It involves the centralization not just of access to technologies, but also control over them. Consider the hierarchy of loci of control over AI systems (from centralized to decentralized): governments, model developers, application developers, deployers, and end users. In the nonproliferation mindset, control is exercised at the highest (most centralized) level possible, whereas in the resilience mindset it is usually exercised at the lowest possible level.

The following are examples of nonproliferation-based interventions:

• Removing dual-use capabilities from models through “forgetting” techniques.

• Curbing the ability of downstream developers to fine-tune models.

• Entrusting AI models and systems themselves with making safety decisions autonomously on the basis that they are trained to comply with centralized safety policies, whereas deployers/users are not trusted to do so.

• Increasing AI systems’ level of access to context, resources, and sensitive data, on the basis that it allows them to make better safety decisions (for example, having access to the user’s web search history might allow a chatbot to better determine whether the intent behind a request is malicious).

• Developing “AI organizations” (multi-agent systems with high levels of organizational complexity) that are under the developer’s control and operate in parallel with traditional organizations instead of integrating AI agents into existing organizations.

With limited exceptions, we believe that nonproliferation-based safety measures decrease resilience and thus worsen AI risks in the long run.¹⁰⁷ They lead to design and implementation choices that potentially enable superintelligence in the sense of power—increasing levels of autonomy, organizational ability, access to resources, and the like. Paradoxically, they increase the very risks they are intended to defend against.

Realizing the Benefits of AI

An important consequence of the normal technology view is that progress is not automatic—there are many roadblocks to AI diffusion. As Jeffrey Ding has shown, the capacity to diffuse innovations throughout the economy varies greatly between countries and has a major effect on their overall power and economic growth.¹⁰⁸ As an example of how diffusion can be a bottleneck, recall the example of the electrification of factories described above. Policy can mitigate or worsen these roadblocks.

Realizing the benefits of AI will require experimentation and reconfiguration. Regulation that is insensitive to these needs risks stymying beneficial AI adoption. Regulation tends to create or reify categories, and might thus prematurely freeze business models, forms of organization, product categories, and so forth. The following are a few examples:

• Categorizing certain domains as “high-risk,” say insurance, benefits adjudication, or hiring, may be a category error, as the variation in risk among tasks within a domain may be far greater than the variation across domains.¹⁰⁹ Tasks in the same domains might range from automated decision making (highly consequential) to optical character recognition (relatively innocuous). Moreover, the diffusion of AI will surely create new tasks that we have not yet envisioned and which might be preemptively miscategorized by regulation.

• The AI supply chain is changing rapidly. The rise of foundation models has led to a much sharper distinction between model developers, downstream developers, and deployers (among many other categories). Regulation that is insensitive to these distinctions risks burdening model developers with responsibilities for risk mitigation related to particular deployment contexts, which would be impossible for them to carry out due to the general-purpose nature of foundation models and the unknowability of all the possible deployment contexts.

• When regulation makes a binary distinction between decisions that are and are not fully automated, and does not recognize degrees of oversight, it disincentivizes the adoption of new models for AI control. As we discussed above, there are many new models being proposed for how to have effective human oversight without having a human in the loop in every decision. It would be unwise to define automated decision making in such a way that these approaches incur the same compliance burdens as a system with no oversight at all.

To be clear, regulation versus diffusion is a false tradeoff, just as is regulation versus innovation.¹¹⁰ None of the above examples are arguments against regulation; they only illustrate the need for nuance and flexibility.

Moreover, regulation has a crucial role to play in enabling diffusion. As a historical example, the ESIGN Act of 2000 in the U.S. was instrumental in promoting digitization and e-commerce: Ensuring that electronic signatures and records are legally valid helped build trust in digital transactions.¹¹¹

In AI, too, there are many opportunities for diffusion-enabling regulation. As one example, the incorporation of journalistic and media content into chatbots and other AI interfaces is limited by media organizations’ justified wariness of AI companies. Many of the AI-meets-journalism deals that have been made thus far are exploitative due to the power asymmetry between AI companies and publishers, and the latter’s inability to bargain collectively. Various models for mandatory negotiation with regulatory oversight are possible.¹¹² (Arguably a more important reason for such regulation is to protect the interests of publishers, which we revisit below).

In areas in which there is legal or regulatory uncertainty, regulation can promote diffusion. The application of liability laws to AI is often unclear. For example, this was the case with small drones until the Federal Aviation Administration regulated the nascent industry in 2016, establishing clear rules and requirements. The resulting clarity spurred adoption and led to a rapid rise in the number of registered drones, certified pilots, and use cases across different industries.¹¹³

Moving beyond the government’s role as a regulator, one powerful strategy for promoting AI diffusion is investing in the complements of automation, which are things that become more valuable or necessary as automation increases. One example is promoting AI literacy as well as workforce training in both the public and the private sectors. Another example is digitization and open data, especially open government data, which can allow AI users to benefit from previously inaccessible datasets. The private sector will be likely to underinvest in these areas as they are public goods that everyone can benefit from. Improvements to energy infrastructure, such as the reliability of the grid, will promote both AI innovation and diffusion since it will help in both AI training and inference.

Governments also have an important role to play in redistributing the benefits of AI to make them more equitable and in compensating those who stand to lose as a result of automation. Strengthening social safety nets will help to decrease the currently high levels of public anxiety about AI in many countries.¹¹⁴ The arts and journalism are vital spheres of life that have been harmed by AI. Governments should consider funding them through taxes on AI companies.

Finally, governments should strike a fine balance in terms of the public sector adoption of AI. Moving too quickly will lead to a loss of trust and legitimacy, as was the case of the New York City chatbot that was evidently inadequately tested and made headlines for telling businesses to break the law.¹¹⁵ The use of AI by the U.S. Department of Government Efficiency (DOGE) includes many dubious applications.¹¹⁶ But moving too slowly might mean that basic government functions are outsourced to the private sector where they are implemented with less accountability.¹¹⁷

For example, the complexity of rules in areas such as taxes and welfare means that people often turn to chatbots for guidance on navigating them, and governments currently lag far behind in providing such services due to understandable caution about the risks involved.¹¹⁸

But the administrative state’s approach to these risks is overly cautious and has been described by Nicholas Bagley as a “procedure fetish,” potentially leading to a “runaway bureaucracy.”¹¹⁹ In addition to losing out on the benefits of AI, Bagley cautioned that incompetent performance will lead to government agencies losing the very legitimacy that they seek to gain through their emphasis on procedure and accountability.

Final Thoughts

AI as normal technology is a worldview that stands in contrast to the worldview of AI as impending superintelligence. Worldviews are constituted by their assumptions, vocabulary, interpretations of evidence, epistemic tools, predictions, and (possibly) values. These factors reinforce each other and form a tight bundle within each worldview.

For example, we assume that, despite the obvious differences between AI and past technologies, they are sufficiently similar that we should expect well-established patterns, such as diffusion theory to apply to AI, in the absence of specific evidence to the contrary.

Vocabulary differences can be pernicious because they may hide underlying assumptions. For example, we reject certain assumptions that are required for the meaningfulness of the concept of superintelligence as it is commonly understood.

Differences about the future of AI are often partly rooted in differing interpretations of evidence about the present. For example, we strongly disagree with the characterization of generative AI adoption as rapid (which reinforces our assumption about the similarity of AI diffusion to past technologies).

In terms of epistemic tools, we deemphasize probability forecasting and emphasize the need for disaggregating what we mean by AI (levels of generality, progress in methods versus application development versus diffusion, etc.) when extrapolating from the past to the future.

We believe that some version of our worldview is widely held. Unfortunately, it has not been articulated explicitly, perhaps because it might seem like the default to someone who holds this view, and articulating it might seem superfluous. Over time, however, the superintelligence view has become dominant in AI discourse, to the extent that someone steeped in it might not recognize that there exists another coherent way to conceptualize the present and future of AI. Thus, it might be hard to recognize the underlying reasons why different people might sincerely have dramatically differing opinions about AI progress, risks, and policy. We hope that this paper can play some small part in enabling greater mutual understanding, even if it does not change any beliefs.

Acknowledgments

We are deeply grateful to Gillian Hadfield, Seth Lazar, and our anonymous peer reviewer for detailed comments on our paper during and after the Knight First Amendment Institute workshop on AI and democratic freedoms. We also thank the participants at the workshop, including Alex Abdo, Borhane Blili-Hamelin, Kevin Feng, Henry Farrell, Katy Glenn-Bass, Atoosa Kasirzadeh, Sydney Levine, Nik Marda, Deirdre Mulligan, and Daniel Susskind. We are fortunate to have received feedback on drafts from many other people, including Shazeda Ahmed, Dean Ball, Nicholas Carlini, Alan Chan, Ajeya Cotra, Justin Curl, Jeffrey Ding, Benjamin Edelman, Jobst Heitzig, Noam Kolt, Mihir Kshirsagar, Timothy B. Lee, Steve Newman, David Robinson, Matthew Salganik, Zachary Siegel, Ollie Stephenson, and Zach Vertin. We are grateful to Shira Minsk and Mandy Soulsby-Bodart for editorial support. Finally, we are grateful for feedback from members of the MINT lab at the Australian National University and from the students in the Limits to Prediction course at Princeton University.

References

[1] Nick Bostrom. 2012. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines 22, 2 (May 2012), 71–85; Nick Bostrom. 2017. Superintelligence: Paths, Dangers, Strategies (reprinted with corrections). Oxford University Press, Oxford, United Kingdom; Sam Altman, Greg Brockman, and Ilya Sutskever. 2023. Governance of Superintelligence (May 2023); Shazeda Ahmed et al. 2023. Building the Epistemic Community of AI Safety. SSRN: Rochester, NY.

[2] This is different from the question of whether it is helpful for an individual user to conceptualize a specific AI system as a tool as opposed to a human-like entity such as an intern, a co-worker, or a tutor.

[3] Daron Acemoglu and Simon Johnson. 2023. Power and Progress: Our Thousand-Year Struggle over Technology and Prosperity. PublicAffairs, New York, NY.

[4] Jeffrey Ding. 2024. Technology and the Rise of Great Powers: How Diffusion Shapes Economic Competition. Princeton University Press, Princeton.

[5] Angelina Wang et al. 2023. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA: ACM, 2023), 626–26.

[6] Casey Ross. 2022. Epic’s Overhaul of a Flawed Algorithm Shows Why AI Oversight Is a Life-or-Death Issue. STAT.

[7] Andrew Wong et al. 2021. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine 181, 8 (August 2021), 1065–70.

[8] Kevin Roose. 2023. A Conversation With Bing’s Chatbot Left Me Deeply Unsettled. The New York Times (February 2023).

[9] Dan Milmo and Alex Hern. 2024. ‘We definitely messed up’: why did Google AI tool make offensive historical images? The Guardian (March 2024).

[10] Jamie Bernardi et al. 2024. Societal adaptation to advanced AI. arXiv: May 2024; Center for Devices and Radiological Health. 2024. Regulatory evaluation of new artificial intelligence (AI) uses for improving and automating medical practices. FDA (June 2024); “Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying down Harmonised Rules on Artificial Intelligence and Amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA Relevance),” June 2024.