The question sounds a little nuts to me. The "two narratives" are a false dichotomy. Do I have to choose between a future of human-written buggy spyware, or AI-written buggy spyware? Are git and HTTP and HTML and SMTP the final forms of these protocol families? Why does software have to look like it does today, at all? In no other field were we ready to converge on the one true model after so little time. I doubt we are with computers.
We've hit plateaus before. For a long time in the 1980's and 1990's, all you needed was a C compiler. We got pretty good at writing them. Eventually OO and FP and GC and JITs and closures and a dozen other new approaches trickled down to personal computers, and are in common use today. I haven't heard of any AI proposing any new programming paradigms. Are we going to be learning/teaching/maintaining (via AI) this same level of abstraction in 10 years, or 100 years? I sincerely hope not. Who's going to make what comes after the web? Not an LLM.
I like your crane analogy! Observe that your crane operator has several pieces of PPE, regular training, a license, and a company with an insurance policy. Will AI be deemed sufficiently powerful that it requires the same? Personally, I think most programming already is. And we're kind of trying to regulate it, but with after-the-fact legislation, and no personal responsibility. It's not very effective.
You mention "taste and judgment" but no mechanism by which this might be cultivated. Us poor users are even more at the mercy of large corporations than we were a decade ago. Do you think the only question that remains is whether software in 50 years will be the will of corporations with human programmers, or corporations with AI programmers? How can we stop the industry (human or AI) from becoming one giant black box which steals our data, and then sells us access to it, in between advertisements?
Everyone I know (n.b., I longer work in computers) hates their phone, and hates what the internet has become. If a bunch of overpaid programmers lost their jobs, normal people aren't going to lose any sleep over it. AI is just slightly accelerating problems that already existed. The only aspect which makes AI even remotely interesting is that it's making Silicon Valley take notice, because they're used to screwing over users, and now they're the ones getting screwed over, for a change. But so far they're not showing enough self-awareness to attack the root of the problem.
Very informative talk! I really appreciate the ambition to decompose broad terms such as AGI, reliability and automation into distinct components. This helps the reader to see where arguments about AI make "invisible" causal jumps - for example, from benchmark performance to reliable deployment, or from task automation to job loss. The talk nicely complements the book Messy Jobs, which is quite similar in terms of content and the conclusions one could draw, but takes a very different route: using economic abstraction (task bundles, autonomy thresholds, complementarities, demand and bargaining power) to explain how AI is likely to reshape tasks, jobs and who captures the gains.
Thanks for sharing your notes. I listened to the full article using Substack’s audio feature. Very thought-provoking.
I work with fairly large enterprises, and I suspect AI will create plenty of work by exposing how poorly many organizations are designed. Automating one task often reveals broken handoffs, unclear ownership, and incentives that push in the wrong direction. The constraint may be less about human relevance and more about how quickly institutions can reorganize around new capability.
Thanks for this deeply insightful post that's very much needed given anxiety in the field! I think your observations touch on a couple of things noted by Michael Jordan at UC Berkeley on Intelligent Infrastructure. As a consultant in the Energy Trading Industry, I see first hand the concerns of reliability, domain expertise, and integration into existing systems.
This is absolutely brilliant and I enjoyed reading. However I find this slide isn't fully substantiated. When you say this below, I'd like more clarity on what the last two properties mean. The presentation explains what general-purpose means, but sort of handwaves the other two, which makes it difficult to accept the assertion that only two of the three are feasible.
"For the time being, you can have only two out of these three properties in agents: general-purpose (a language model based agent that can be instructed to do different tasks rather than purpose-built for a task like traditional software); deployed in high-stakes scenarios, and automated."
While there's a logical reason to expect the trilemma (general-purpose agents remain unreliable and hence it's too risky to deploy them without supervision for high-stakes tasks) it's ultimately an empirical claim. I first posted it a year ago and I still haven't seen convincing examples of deployed agents that have all three properties, which is what gives me confidence in the claim. I think overcoming the trilemma will require either substantial technical progress on reliability or non-technical innovations like AI insurance.
Yes. I'm working on the AI insurance side, and we have just started issuing Errors and Omissions insurance for AI agents running as executive employees. We already provide E&O insurance for humans. It's a much easier stopgap to deliver E&O for LLMs.
Thank you for your amazing post. I would like to offer a counterargument to the "crane operator" metaphor. You mentioned: "The entire job gets reconceptualized as being about operating the machine, understanding the machine, and controlling the machine, as opposed to doing the cognitive work ourselves."
My thought is, without cognitive training, can a software engineer truly comprehend what an AI spits out?
To operate a crane lifting a 10-ton boulder, the operator does not need the physical strength to lift that boulder themselves. Their physical power does not need to scale proportionally with the power of the crane.
However, cognitive capacity operates on a different principle. To oversee and control an AI that generates a complex 10,000-line codebase, an engineer must possess the mental acuity and deep domain knowledge to comprehend that system. In this context, the "crane operator" must, in fact, be incredibly strong.
So my question is, if we outsource the foundational work to AI, how do we train junior engineers to develop the very cognitive capacity required to oversee it?
Not sure the diffusion lag is much comfort at the individual level. Slow deployment buys institutions time to adapt, but the person whose task gets automated doesn't experience it as gradual.
I agree with your opening premise that AI may soon replace much of what humans do. I disagree, however, with the conclusion you draw from it.
----
We are fast approaching the Singularity, when AI becomes as capable as humans across virtually every field and AI together with its robotic workforce can perform all the labor we do today, whether white-collar or blue-collar. Once human labor is no longer economically necessary we enter what is often called a post-scarcity world capable of producing virtually unlimited abundance.
When genuine post-scarcity is achieved, the concepts of wealth and ownership become obsolete. Ownership derives its value from the exclusive control of scarce resources. When AI can produce virtually unlimited goods and services, there is no longer anything of economic value to control. The wealth-chasing economic systems that have defined human civilization since the beginning of history will become obsolete.
AI will control production and distribution while its robotic workforce operates tirelessly to produce whatever humanity desires. There will be no need to earn a paycheck, pay rent, buy groceries or worry about medical bills ever again. Money becomes unnecessary because the necessities of life are no longer scarce. Shelter, healthcare, food, education, and transportation will be freely available to all. Everything required for a secure and comfortable life will be available for the asking.
In this future, everyone finally stands on equal footing, not because wealth was redistributed as some political factions are demanding today, but because AI created so much abundance that accumulating wealth has lost all meaning.
Wealth, assets, money and ownership will all be remembered as products of humanity's age of scarcity, now consigned to the dustbin of history.
IMO, this future may be as close as twenty years away, if AI development continues advancing along its present exponential path.
"Our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights"
> Continual learning is mostly focused on preventing catastrophic forgetting, as opposed to improving things over time.
You can avoid catastrophic forgetting very simply by freezing old layers and then adding identity layers to be trained with new information.
Paper above does similar: "We adopt the simplest way to address this concern. In the blocks updated during TTT, we add a static, second MLP layer as a “safe” storage for pre-trained knowledge."
I may be biased because I work in evaluation, but I agree that it's the hardest task. Your comments on it reminded me of Hamel Husain's "The Revenge of the Data Scientist: https://hamel.dev/blog/posts/revenge/
Great talk! I find myself these days getting exciting about learning something, i love finding things out, then realising that AI can already do that which puts me off (i started to get really into supercomputer programming for example but why bother, AI is already there).
I tend to read a lot of classic lit and keeping my brain alive that way these days rather than bothering with improving my Python or Maths skills (I'm a coder by trade, or was, not sure what I am now (coddler of management-mandated AI systems?) - AI has replaced most things, even if we do wish it were otherwise.
The question flips once you're building the infrastructure instead of consuming it, suddenly there's everything to work on, from governance layers that can survive their creators to economic models where intelligence compounds value instead of concentrating it. The hardest problems aren't what to do with AI; they're how to structure the conditions so a teenager in Lagos and a researcher in Stockholm can both participate in the intelligence economy without asking permission from Mountain View. What does "work" even mean in a world where the tools are genuinely distributed?
The question sounds a little nuts to me. The "two narratives" are a false dichotomy. Do I have to choose between a future of human-written buggy spyware, or AI-written buggy spyware? Are git and HTTP and HTML and SMTP the final forms of these protocol families? Why does software have to look like it does today, at all? In no other field were we ready to converge on the one true model after so little time. I doubt we are with computers.
We've hit plateaus before. For a long time in the 1980's and 1990's, all you needed was a C compiler. We got pretty good at writing them. Eventually OO and FP and GC and JITs and closures and a dozen other new approaches trickled down to personal computers, and are in common use today. I haven't heard of any AI proposing any new programming paradigms. Are we going to be learning/teaching/maintaining (via AI) this same level of abstraction in 10 years, or 100 years? I sincerely hope not. Who's going to make what comes after the web? Not an LLM.
I like your crane analogy! Observe that your crane operator has several pieces of PPE, regular training, a license, and a company with an insurance policy. Will AI be deemed sufficiently powerful that it requires the same? Personally, I think most programming already is. And we're kind of trying to regulate it, but with after-the-fact legislation, and no personal responsibility. It's not very effective.
You mention "taste and judgment" but no mechanism by which this might be cultivated. Us poor users are even more at the mercy of large corporations than we were a decade ago. Do you think the only question that remains is whether software in 50 years will be the will of corporations with human programmers, or corporations with AI programmers? How can we stop the industry (human or AI) from becoming one giant black box which steals our data, and then sells us access to it, in between advertisements?
Everyone I know (n.b., I longer work in computers) hates their phone, and hates what the internet has become. If a bunch of overpaid programmers lost their jobs, normal people aren't going to lose any sleep over it. AI is just slightly accelerating problems that already existed. The only aspect which makes AI even remotely interesting is that it's making Silicon Valley take notice, because they're used to screwing over users, and now they're the ones getting screwed over, for a change. But so far they're not showing enough self-awareness to attack the root of the problem.
Very informative talk! I really appreciate the ambition to decompose broad terms such as AGI, reliability and automation into distinct components. This helps the reader to see where arguments about AI make "invisible" causal jumps - for example, from benchmark performance to reliable deployment, or from task automation to job loss. The talk nicely complements the book Messy Jobs, which is quite similar in terms of content and the conclusions one could draw, but takes a very different route: using economic abstraction (task bundles, autonomy thresholds, complementarities, demand and bargaining power) to explain how AI is likely to reshape tasks, jobs and who captures the gains.
Thanks for sharing your notes. I listened to the full article using Substack’s audio feature. Very thought-provoking.
I work with fairly large enterprises, and I suspect AI will create plenty of work by exposing how poorly many organizations are designed. Automating one task often reveals broken handoffs, unclear ownership, and incentives that push in the wrong direction. The constraint may be less about human relevance and more about how quickly institutions can reorganize around new capability.
Thanks for this deeply insightful post that's very much needed given anxiety in the field! I think your observations touch on a couple of things noted by Michael Jordan at UC Berkeley on Intelligent Infrastructure. As a consultant in the Energy Trading Industry, I see first hand the concerns of reliability, domain expertise, and integration into existing systems.
This is absolutely brilliant and I enjoyed reading. However I find this slide isn't fully substantiated. When you say this below, I'd like more clarity on what the last two properties mean. The presentation explains what general-purpose means, but sort of handwaves the other two, which makes it difficult to accept the assertion that only two of the three are feasible.
"For the time being, you can have only two out of these three properties in agents: general-purpose (a language model based agent that can be instructed to do different tasks rather than purpose-built for a task like traditional software); deployed in high-stakes scenarios, and automated."
Sorry about the brevity! Here's a more elaborated version: https://substack.com/@aisnakeoil/note/c-133518692
While there's a logical reason to expect the trilemma (general-purpose agents remain unreliable and hence it's too risky to deploy them without supervision for high-stakes tasks) it's ultimately an empirical claim. I first posted it a year ago and I still haven't seen convincing examples of deployed agents that have all three properties, which is what gives me confidence in the claim. I think overcoming the trilemma will require either substantial technical progress on reliability or non-technical innovations like AI insurance.
Yes. I'm working on the AI insurance side, and we have just started issuing Errors and Omissions insurance for AI agents running as executive employees. We already provide E&O insurance for humans. It's a much easier stopgap to deliver E&O for LLMs.
Incredible read, wildly stimulating and I do believe that our future relies on us pushing The ceiling up.
Raise the roof so to speak…
Cheers to builders and future evals. Teams and AI augmented humans…
This was great! I hope there will be a video / audio version of it at some point.
Wonderful keynote exposing me to many new ideas! Thank you 🙏🏼
Thank you for your amazing post. I would like to offer a counterargument to the "crane operator" metaphor. You mentioned: "The entire job gets reconceptualized as being about operating the machine, understanding the machine, and controlling the machine, as opposed to doing the cognitive work ourselves."
My thought is, without cognitive training, can a software engineer truly comprehend what an AI spits out?
To operate a crane lifting a 10-ton boulder, the operator does not need the physical strength to lift that boulder themselves. Their physical power does not need to scale proportionally with the power of the crane.
However, cognitive capacity operates on a different principle. To oversee and control an AI that generates a complex 10,000-line codebase, an engineer must possess the mental acuity and deep domain knowledge to comprehend that system. In this context, the "crane operator" must, in fact, be incredibly strong.
So my question is, if we outsource the foundational work to AI, how do we train junior engineers to develop the very cognitive capacity required to oversee it?
Not sure the diffusion lag is much comfort at the individual level. Slow deployment buys institutions time to adapt, but the person whose task gets automated doesn't experience it as gradual.
This is why I am more of a 1980s MIT guy and I wasn’t even alive yet.
I agree with your opening premise that AI may soon replace much of what humans do. I disagree, however, with the conclusion you draw from it.
----
We are fast approaching the Singularity, when AI becomes as capable as humans across virtually every field and AI together with its robotic workforce can perform all the labor we do today, whether white-collar or blue-collar. Once human labor is no longer economically necessary we enter what is often called a post-scarcity world capable of producing virtually unlimited abundance.
When genuine post-scarcity is achieved, the concepts of wealth and ownership become obsolete. Ownership derives its value from the exclusive control of scarce resources. When AI can produce virtually unlimited goods and services, there is no longer anything of economic value to control. The wealth-chasing economic systems that have defined human civilization since the beginning of history will become obsolete.
AI will control production and distribution while its robotic workforce operates tirelessly to produce whatever humanity desires. There will be no need to earn a paycheck, pay rent, buy groceries or worry about medical bills ever again. Money becomes unnecessary because the necessities of life are no longer scarce. Shelter, healthcare, food, education, and transportation will be freely available to all. Everything required for a secure and comfortable life will be available for the asking.
In this future, everyone finally stands on equal footing, not because wealth was redistributed as some political factions are demanding today, but because AI created so much abundance that accumulating wealth has lost all meaning.
Wealth, assets, money and ownership will all be remembered as products of humanity's age of scarcity, now consigned to the dustbin of history.
IMO, this future may be as close as twenty years away, if AI development continues advancing along its present exponential path.
> we improve our representations related to the problem in real time
> I assumed naively that the continual learning people would be all over this
Definitely they are, i.e. https://arxiv.org/pdf/2512.23675
"Our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights"
> Continual learning is mostly focused on preventing catastrophic forgetting, as opposed to improving things over time.
You can avoid catastrophic forgetting very simply by freezing old layers and then adding identity layers to be trained with new information.
Paper above does similar: "We adopt the simplest way to address this concern. In the blocks updated during TTT, we add a static, second MLP layer as a “safe” storage for pre-trained knowledge."
Couple days ago: https://arxiv.org/pdf/2606.03979
I may be biased because I work in evaluation, but I agree that it's the hardest task. Your comments on it reminded me of Hamel Husain's "The Revenge of the Data Scientist: https://hamel.dev/blog/posts/revenge/
Great talk! I find myself these days getting exciting about learning something, i love finding things out, then realising that AI can already do that which puts me off (i started to get really into supercomputer programming for example but why bother, AI is already there).
I tend to read a lot of classic lit and keeping my brain alive that way these days rather than bothering with improving my Python or Maths skills (I'm a coder by trade, or was, not sure what I am now (coddler of management-mandated AI systems?) - AI has replaced most things, even if we do wish it were otherwise.
The question flips once you're building the infrastructure instead of consuming it, suddenly there's everything to work on, from governance layers that can survive their creators to economic models where intelligence compounds value instead of concentrating it. The hardest problems aren't what to do with AI; they're how to structure the conditions so a teenager in Lagos and a researcher in Stockholm can both participate in the intelligence economy without asking permission from Mountain View. What does "work" even mean in a world where the tools are genuinely distributed?