Thanks for sharing your notes. I listened to the full article using Substack’s audio feature. Very thought-provoking.
I work with fairly large enterprises, and I suspect AI will create plenty of work by exposing how poorly many organizations are designed. Automating one task often reveals broken handoffs, unclear ownership, and incentives that push in the wrong direction. The constraint may be less about human relevance and more about how quickly institutions can reorganize around new capability.
Thanks for this deeply insightful post that's very much needed given anxiety in the field! I think your observations touch on a couple of things noted by Michael Jordan at UC Berkeley on Intelligent Infrastructure. As a consultant in the Energy Trading Industry, I see first hand the concerns of reliability, domain expertise, and integration into existing systems.
This is absolutely brilliant and I enjoyed reading. However I find this slide isn't fully substantiated. When you say this below, I'd like more clarity on what the last two properties mean. The presentation explains what general-purpose means, but sort of handwaves the other two, which makes it difficult to accept the assertion that only two of the three are feasible.
"For the time being, you can have only two out of these three properties in agents: general-purpose (a language model based agent that can be instructed to do different tasks rather than purpose-built for a task like traditional software); deployed in high-stakes scenarios, and automated."
While there's a logical reason to expect the trilemma (general-purpose agents remain unreliable and hence it's too risky to deploy them without supervision for high-stakes tasks) it's ultimately an empirical claim. I first posted it a year ago and I still haven't seen convincing examples of deployed agents that have all three properties, which is what gives me confidence in the claim. I think overcoming the trilemma will require either substantial technical progress on reliability or non-technical innovations like AI insurance.
Very informative talk! I really appreciate the ambition to decompose broad terms such as AGI, reliability and automation into distinct components. This helps the reader to see where arguments about AI make "invisible" causal jumps - for example, from benchmark performance to reliable deployment, or from task automation to job loss. The talk nicely complements the book Messy Jobs, which is quite similar in terms of content and the conclusions one could draw, but takes a very different route: using economic abstraction (task bundles, autonomy thresholds, complementarities, demand and bargaining power) to explain how AI is likely to reshape tasks, jobs and who captures the gains.
The question sounds a little nuts to me. The "two narratives" are a false dichotomy. Do I have to choose between a future of human-written buggy spyware, or AI-written buggy spyware? Are git and HTTP and HTML and SMTP the final forms of these protocol families? Why does software have to look like it does today, at all? In no other field were we ready to converge on the one true model after so little time. I doubt we are with computers.
We've hit plateaus before. For a long time in the 1980's and 1990's, all you needed was a C compiler. We got pretty good at writing them. Eventually OO and FP and GC and JITs and closures and a dozen other new approaches trickled down to personal computers, and are in common use today. I haven't heard of any AI proposing any new programming paradigms. Are we going to be learning/teaching/maintaining (via AI) this same level of abstraction in 10 years, or 100 years? I sincerely hope not. Who's going to make what comes after the web? Not an LLM.
I like your crane analogy! Observe that your crane operator has several pieces of PPE, regular training, a license, and a company with an insurance policy. Will AI be deemed sufficiently powerful that it requires the same? Personally, I think most programming already is. And we're kind of trying to regulate it, but with after-the-fact legislation, and no personal responsibility. It's not very effective.
You mention "taste and judgment" but no mechanism by which this might be cultivated. Us poor users are even more at the mercy of large corporations than we were a decade ago. Do you think the only question that remains is whether software in 50 years will be the will of corporations with human programmers, or corporations with AI programmers? How can we stop the industry (human or AI) from becoming one giant black box which steals our data, and then sells us access to it, in between advertisements?
Everyone I know (n.b., I longer work in computers) hates their phone, and hates what the internet has become. If a bunch of overpaid programmers lost their jobs, normal people aren't going to lose any sleep over it. AI is just slightly accelerating problems that already existed. The only aspect which makes AI even remotely interesting is that it's making Silicon Valley take notice, because they're used to screwing over users, and now they're the ones getting screwed over, for a change. But so far they're not showing enough self-awareness to attack the root of the problem.
I may be biased because I work in evaluation, but I agree that it's the hardest task. Your comments on it reminded me of Hamel Husain's "The Revenge of the Data Scientist: https://hamel.dev/blog/posts/revenge/
Great talk! I find myself these days getting exciting about learning something, i love finding things out, then realising that AI can already do that which puts me off (i started to get really into supercomputer programming for example but why bother, AI is already there).
I tend to read a lot of classic lit and keeping my brain alive that way these days rather than bothering with improving my Python or Maths skills (I'm a coder by trade, or was, not sure what I am now (coddler of management-mandated AI systems?) - AI has replaced most things, even if we do wish it were otherwise.
The question flips once you're building the infrastructure instead of consuming it, suddenly there's everything to work on, from governance layers that can survive their creators to economic models where intelligence compounds value instead of concentrating it. The hardest problems aren't what to do with AI; they're how to structure the conditions so a teenager in Lagos and a researcher in Stockholm can both participate in the intelligence economy without asking permission from Mountain View. What does "work" even mean in a world where the tools are genuinely distributed?
Thank you, Arvind, for a refreshing take on AI. There are so many fantasists in this business, and not enough real-world deep thinkers. I especially like the notion that ASI will lead to HSI -- Human superintelligence. Bring it on!
“Collaboration technology” sounds harmless until it turns up in a meeting about headcount. The same tool can remove drudge work, justify layoffs, turn into Big Brother, or just increase the amount of work expected from the poor souls left. The model’s capability matters, but whoever controls the rollout controls who benefits.
The backlash warning lands harder with the gap already measured. Pew's 2025 survey: 56% of AI experts expect AI to be net-positive for the US over the next 20 years, versus 17% of the public, the exact expert-vs-everyone split your 'don't just roll over' point turns on. And Reuters/Ipsos has 71% of Americans already worried about permanent AI job loss. The community isn't out ahead of a backlash so much as on the wrong side of a 39-point perception gap.
Thank you for sharing this talk! Your mentioning your evaluation group got me wondering - do you focus by sector when thinking of evaluation targets or is it more general than that? Do you know of groups operating with your or similar evaluation frameworks on specific sectors?
Thanks for sharing your notes. I listened to the full article using Substack’s audio feature. Very thought-provoking.
I work with fairly large enterprises, and I suspect AI will create plenty of work by exposing how poorly many organizations are designed. Automating one task often reveals broken handoffs, unclear ownership, and incentives that push in the wrong direction. The constraint may be less about human relevance and more about how quickly institutions can reorganize around new capability.
Thanks for this deeply insightful post that's very much needed given anxiety in the field! I think your observations touch on a couple of things noted by Michael Jordan at UC Berkeley on Intelligent Infrastructure. As a consultant in the Energy Trading Industry, I see first hand the concerns of reliability, domain expertise, and integration into existing systems.
This is absolutely brilliant and I enjoyed reading. However I find this slide isn't fully substantiated. When you say this below, I'd like more clarity on what the last two properties mean. The presentation explains what general-purpose means, but sort of handwaves the other two, which makes it difficult to accept the assertion that only two of the three are feasible.
"For the time being, you can have only two out of these three properties in agents: general-purpose (a language model based agent that can be instructed to do different tasks rather than purpose-built for a task like traditional software); deployed in high-stakes scenarios, and automated."
Sorry about the brevity! Here's a more elaborated version: https://substack.com/@aisnakeoil/note/c-133518692
While there's a logical reason to expect the trilemma (general-purpose agents remain unreliable and hence it's too risky to deploy them without supervision for high-stakes tasks) it's ultimately an empirical claim. I first posted it a year ago and I still haven't seen convincing examples of deployed agents that have all three properties, which is what gives me confidence in the claim. I think overcoming the trilemma will require either substantial technical progress on reliability or non-technical innovations like AI insurance.
Very informative talk! I really appreciate the ambition to decompose broad terms such as AGI, reliability and automation into distinct components. This helps the reader to see where arguments about AI make "invisible" causal jumps - for example, from benchmark performance to reliable deployment, or from task automation to job loss. The talk nicely complements the book Messy Jobs, which is quite similar in terms of content and the conclusions one could draw, but takes a very different route: using economic abstraction (task bundles, autonomy thresholds, complementarities, demand and bargaining power) to explain how AI is likely to reshape tasks, jobs and who captures the gains.
This was great! I hope there will be a video / audio version of it at some point.
Wonderful keynote exposing me to many new ideas! Thank you 🙏🏼
The question sounds a little nuts to me. The "two narratives" are a false dichotomy. Do I have to choose between a future of human-written buggy spyware, or AI-written buggy spyware? Are git and HTTP and HTML and SMTP the final forms of these protocol families? Why does software have to look like it does today, at all? In no other field were we ready to converge on the one true model after so little time. I doubt we are with computers.
We've hit plateaus before. For a long time in the 1980's and 1990's, all you needed was a C compiler. We got pretty good at writing them. Eventually OO and FP and GC and JITs and closures and a dozen other new approaches trickled down to personal computers, and are in common use today. I haven't heard of any AI proposing any new programming paradigms. Are we going to be learning/teaching/maintaining (via AI) this same level of abstraction in 10 years, or 100 years? I sincerely hope not. Who's going to make what comes after the web? Not an LLM.
I like your crane analogy! Observe that your crane operator has several pieces of PPE, regular training, a license, and a company with an insurance policy. Will AI be deemed sufficiently powerful that it requires the same? Personally, I think most programming already is. And we're kind of trying to regulate it, but with after-the-fact legislation, and no personal responsibility. It's not very effective.
You mention "taste and judgment" but no mechanism by which this might be cultivated. Us poor users are even more at the mercy of large corporations than we were a decade ago. Do you think the only question that remains is whether software in 50 years will be the will of corporations with human programmers, or corporations with AI programmers? How can we stop the industry (human or AI) from becoming one giant black box which steals our data, and then sells us access to it, in between advertisements?
Everyone I know (n.b., I longer work in computers) hates their phone, and hates what the internet has become. If a bunch of overpaid programmers lost their jobs, normal people aren't going to lose any sleep over it. AI is just slightly accelerating problems that already existed. The only aspect which makes AI even remotely interesting is that it's making Silicon Valley take notice, because they're used to screwing over users, and now they're the ones getting screwed over, for a change. But so far they're not showing enough self-awareness to attack the root of the problem.
I may be biased because I work in evaluation, but I agree that it's the hardest task. Your comments on it reminded me of Hamel Husain's "The Revenge of the Data Scientist: https://hamel.dev/blog/posts/revenge/
Great talk! I find myself these days getting exciting about learning something, i love finding things out, then realising that AI can already do that which puts me off (i started to get really into supercomputer programming for example but why bother, AI is already there).
I tend to read a lot of classic lit and keeping my brain alive that way these days rather than bothering with improving my Python or Maths skills (I'm a coder by trade, or was, not sure what I am now (coddler of management-mandated AI systems?) - AI has replaced most things, even if we do wish it were otherwise.
The question flips once you're building the infrastructure instead of consuming it, suddenly there's everything to work on, from governance layers that can survive their creators to economic models where intelligence compounds value instead of concentrating it. The hardest problems aren't what to do with AI; they're how to structure the conditions so a teenager in Lagos and a researcher in Stockholm can both participate in the intelligence economy without asking permission from Mountain View. What does "work" even mean in a world where the tools are genuinely distributed?
Thank you, Arvind, for a refreshing take on AI. There are so many fantasists in this business, and not enough real-world deep thinkers. I especially like the notion that ASI will lead to HSI -- Human superintelligence. Bring it on!
I love the floor/ceiling image, and will start using it in my work. Thanks!
“Collaboration technology” sounds harmless until it turns up in a meeting about headcount. The same tool can remove drudge work, justify layoffs, turn into Big Brother, or just increase the amount of work expected from the poor souls left. The model’s capability matters, but whoever controls the rollout controls who benefits.
The backlash warning lands harder with the gap already measured. Pew's 2025 survey: 56% of AI experts expect AI to be net-positive for the US over the next 20 years, versus 17% of the public, the exact expert-vs-everyone split your 'don't just roll over' point turns on. And Reuters/Ipsos has 71% of Americans already worried about permanent AI job loss. The community isn't out ahead of a backlash so much as on the wrong side of a 39-point perception gap.
Thank you for sharing this talk! Your mentioning your evaluation group got me wondering - do you focus by sector when thinking of evaluation targets or is it more general than that? Do you know of groups operating with your or similar evaluation frameworks on specific sectors?