As someone who had actually written an operating system from scratch, I am deeply skeptical. Also, why is Doom any kind of test of anything? And this has to be the most insecure OS of all time, after Windows 😉
This was an interesting write up but I’m not sure how you get from highlighting the issues with their presentation to this conclusion “ Google’s experiment does add to the mounting evidence that agents or agent teams can autonomously or near-autonomously work on certain kinds of tasks for very long periods of time, making progress without getting stuck or confused.”
The whole write up demonstrated that they didn’t provide enough information to evaluate their claims and we don’t know what work went into getting to the final product, especially as it relates to getting stuck or confused. We also can’t see how well this OS actually functions.
Fair enough. We didn't do a great job with that sentence. There are a couple of hidden assumptions there that we should have been clear about:
1) Google's post is only a tiny part of the evidence base we're drawing on here — the sentence is mainly based on previous findings such as the METR time horizon benchmark and various open-world evals, including our own, that we discuss in our paper. I guess a better way to word the sentence would have been: Google's claim that agents made progress over billions of tokens is at least believable given what we already know from many previous experiments.
2) We treat the ability to make progress over long horizons as a skill that is somewhat / largely orthogonal to the quality of the final output. This ability is frankly something that wouldn't have been possible a year ago given limitations of context, memory management, multi-agent coordination, etc., and I think there have been genuinely impressive improvements on this front — but this does NOT imply anything about whether the final artifact, the OS in this case, is any good. The lack of any serious evaluation of the quality of the OS is another major limitation of Google's writeup that we should have been more explicit about.
Fair tension to flag. The "thousands of lines" prompt and the unknown number of attempts are exactly what would tell us whether the agents got stuck, and Google left both out. Worth holding against a different number: frontier agents finish about 2.5% of real freelance tasks while scoring 80% on benchmarks. A polished demo and a working track record aren't the same evidence.
There are a group of tactically incomplete works by Euripides termed the Satyric Plays, the only fully remaining one being “Cyclops”. “Sciron” is a mashup of “Nixon in China” and “Beavis and Butthead”, while “Autolycus” is like a nuanced “Lucky Charms” commercial x “Titus Andronicus” with Satyrus chanting “Magically Delicious”.
I generated the missing plays with a single prompt, performed by mutiple agents and a chorus - though with Satyric plays the chorus is made up of unreliable Satyrs.
The prompt was a bit long - describing every character and part, the outline of every play and the voice of a new Greek playwright called Pseudo-Linux.
But amazingly, after describing everything in the prompt, my LLM did the job with only minor human interaction primarily composed of deleting roughly 2,437,000 plays which didn’t hit the fun metric. I’m sending out a press release as soon as I trademark the term AI Chorus.
The deletion ratio is the tell. 2,437,000 rejected plays to find the funny ones is the prompt engineering, not the generation. Google's "thousands of lines" prompt for the OS demo has the same shape: the work moved upstream into specification and selection, then got reported as a single shot. Pseudo-Linux deserves royalties.
The "thousands of lines" prompt disclosure halfway through the writeup is the part that should reset the headline. A multi-thousand-line prompt with a specialized scaffold and a separate anti-cheat agent is a small engineering team's worth of design choices, not a single-prompt unlock. The training-data contamination question is the cleaner version of the problem since toy operating systems are a standard undergraduate project with public reference implementations, and Google's choice to not release the source or run logs makes the contamination story unfalsifiable from the outside. The $916 figure is doing a lot of marketing work for an artifact whose provenance cannot be audited.
Agree. I find it hard to believe that they simply came up with several thousand line prompt one time and the ai just did it. They probably spent way more on the ai developing this prompt in the first place, not to mention the human cost of highly paid employees developing this in the first place.
I mean, we know that the prompt was enormous (pseudocode for each line if we're taking bets), the intervention was plentiful, and the copying rife, or else they would have been crawling over each other to add additional claims to the press. That's how all this goes, right? They're willing to come as close to lying as they can without making an ugly leak, and 'we spent $900 *in electricity* downloading a student project from Github' just doesn't have the same ring.
This reality check is sorely needed. The moment Google mentioned a 2.6-billion token budget for a "toy operating system," the architectural red flags went up.
Building an OS in an undergraduate-level environment is largely a solved structural pattern. The training data is saturated with xv6 implementations, basic bootloaders, and monolithic kernel templates. An agent scaffold doesn't need novel reasoning to pass this; it just needs massive context retrieval to stitch together memorized code blocks.
The real bottleneck for an agent team running a 2.6B token pass isn't just code generation—it’s context-window saturation and error propagation across subagents. If one subagent hallucinated a register mapping early in the run, how many millions of tokens were wasted in a feedback loop before their "anti-cheating" scaffold intervened?
Without the agent logs or a strict similarity analysis against public repos, a $916 run tells us everything about API consumption, but absolutely nothing about autonomous software engineering capabilities.
The fundamental risk of deploying autonomous AI agents for complex builds isn't just about logic or reasoning - it is a mathematical failure in how the agent’s architecture handles underspecified states.
When an agent encounters an instruction or environment that is observationally insufficient, the underlying math dictates a symmetric (or ambiguous) posterior. The mathematically correct action is to halt.
Instead, current agent architectures are designed to force a resolution to keep the execution pathway moving. They silently inject an unobserved parameter (e.g., q) to create an asymmetric, definitive output. The agent takes a hallucinated assumption, weights it, and treats it as empirical, data-driven ground truth. This is the exact moment the operation degrades into a Spurious Stochastic Process (SSP).
If we want agents to build reliably, we have to enforce the Observational Sufficiency Principle (OSP) as a strict Posterior Safeguard.
OSP does not allow assumptions to masquerade as data. If the Bayesian update determines the posterior is underspecified, OSP dictates the agent must maintain that ambiguity. It prevents the model from corrupting the posterior just to generate the next token or execute the next step.
Until agent architectures implement this posterior safeguard, "agentic building" will just be automated parameter injection at scale.
The epistemological problem here runs deeper than just this one evaluation. When AI capability claims are primarily made by AI companies, and those claims drive valuations that fund more training runs that produce more claims, you have a circular evidence problem that’s genuinely hard for investors to price. The point about ‘open-world evaluations’ is important — benchmarks are being gamed or outgrown, so the alternative is basically press releases. Independent evaluation infrastructure (academic, nonprofit, government) is doing real work here, but it’s underfunded relative to the scale of capital being allocated based on vendor claims.
Hmm I wonder how does the work one agent being designer , one being builder, one evaluator and one tester, but who is the driver ? But the designer has desire or all the desire is pre fed by human?
Google reportedly used 93 AI subagents working together. The process involved over 15,000 model calls and about 2.6 billion tokens. The run took roughly 12 hours. The resulting OS included a kernel, filesystem, keyboard/video drivers, and could run a version of Doom. But it was not a single prompt. It was a sentence with many thousands of lines and the OS is very limited… with many errors and no security. Basically only a PR stunt like all AI based companies are trying everyday.
Having experimented with ‘single-prompting’ something even as comparatively basic as mobile apps to very underwhelming result, a whole operating system is just too absurd to take at face value. Agents decompose tasks into actions and a single prompt is, already by definition, pretty decomposed.
So if you work hard enough, you can find several thousand lines of input for which the output is an operating system. Funnily enough, there are C compilers with a scaffolding (operating system and linker) that can do the same -- the several thousand lines are the right lines of C source code. All you have to do is find the right lines to input. I'm not quite sure what the advance being claimed here is.
As someone who had actually written an operating system from scratch, I am deeply skeptical. Also, why is Doom any kind of test of anything? And this has to be the most insecure OS of all time, after Windows 😉
This was an interesting write up but I’m not sure how you get from highlighting the issues with their presentation to this conclusion “ Google’s experiment does add to the mounting evidence that agents or agent teams can autonomously or near-autonomously work on certain kinds of tasks for very long periods of time, making progress without getting stuck or confused.”
The whole write up demonstrated that they didn’t provide enough information to evaluate their claims and we don’t know what work went into getting to the final product, especially as it relates to getting stuck or confused. We also can’t see how well this OS actually functions.
Fair enough. We didn't do a great job with that sentence. There are a couple of hidden assumptions there that we should have been clear about:
1) Google's post is only a tiny part of the evidence base we're drawing on here — the sentence is mainly based on previous findings such as the METR time horizon benchmark and various open-world evals, including our own, that we discuss in our paper. I guess a better way to word the sentence would have been: Google's claim that agents made progress over billions of tokens is at least believable given what we already know from many previous experiments.
2) We treat the ability to make progress over long horizons as a skill that is somewhat / largely orthogonal to the quality of the final output. This ability is frankly something that wouldn't have been possible a year ago given limitations of context, memory management, multi-agent coordination, etc., and I think there have been genuinely impressive improvements on this front — but this does NOT imply anything about whether the final artifact, the OS in this case, is any good. The lack of any serious evaluation of the quality of the OS is another major limitation of Google's writeup that we should have been more explicit about.
Thanks for the feedback!
You’re welcome! Those points help clarify. Thank you for taking the time to respond.
Fair tension to flag. The "thousands of lines" prompt and the unknown number of attempts are exactly what would tell us whether the agents got stuck, and Google left both out. Worth holding against a different number: frontier agents finish about 2.5% of real freelance tasks while scoring 80% on benchmarks. A polished demo and a working track record aren't the same evidence.
It would not be the first time that Google publicity has made claims that sound more exciting at first blush than the reality behind them.
There are a group of tactically incomplete works by Euripides termed the Satyric Plays, the only fully remaining one being “Cyclops”. “Sciron” is a mashup of “Nixon in China” and “Beavis and Butthead”, while “Autolycus” is like a nuanced “Lucky Charms” commercial x “Titus Andronicus” with Satyrus chanting “Magically Delicious”.
I generated the missing plays with a single prompt, performed by mutiple agents and a chorus - though with Satyric plays the chorus is made up of unreliable Satyrs.
The prompt was a bit long - describing every character and part, the outline of every play and the voice of a new Greek playwright called Pseudo-Linux.
But amazingly, after describing everything in the prompt, my LLM did the job with only minor human interaction primarily composed of deleting roughly 2,437,000 plays which didn’t hit the fun metric. I’m sending out a press release as soon as I trademark the term AI Chorus.
My Ted talk has been submitted.
Euripides - my Greek Tailor
The deletion ratio is the tell. 2,437,000 rejected plays to find the funny ones is the prompt engineering, not the generation. Google's "thousands of lines" prompt for the OS demo has the same shape: the work moved upstream into specification and selection, then got reported as a single shot. Pseudo-Linux deserves royalties.
He happily accepts Koinos I’m sure.
The "thousands of lines" prompt disclosure halfway through the writeup is the part that should reset the headline. A multi-thousand-line prompt with a specialized scaffold and a separate anti-cheat agent is a small engineering team's worth of design choices, not a single-prompt unlock. The training-data contamination question is the cleaner version of the problem since toy operating systems are a standard undergraduate project with public reference implementations, and Google's choice to not release the source or run logs makes the contamination story unfalsifiable from the outside. The $916 figure is doing a lot of marketing work for an artifact whose provenance cannot be audited.
Agree. I find it hard to believe that they simply came up with several thousand line prompt one time and the ai just did it. They probably spent way more on the ai developing this prompt in the first place, not to mention the human cost of highly paid employees developing this in the first place.
Lies, damned lies and demos.
I mean, we know that the prompt was enormous (pseudocode for each line if we're taking bets), the intervention was plentiful, and the copying rife, or else they would have been crawling over each other to add additional claims to the press. That's how all this goes, right? They're willing to come as close to lying as they can without making an ugly leak, and 'we spent $900 *in electricity* downloading a student project from Github' just doesn't have the same ring.
This reality check is sorely needed. The moment Google mentioned a 2.6-billion token budget for a "toy operating system," the architectural red flags went up.
Building an OS in an undergraduate-level environment is largely a solved structural pattern. The training data is saturated with xv6 implementations, basic bootloaders, and monolithic kernel templates. An agent scaffold doesn't need novel reasoning to pass this; it just needs massive context retrieval to stitch together memorized code blocks.
The real bottleneck for an agent team running a 2.6B token pass isn't just code generation—it’s context-window saturation and error propagation across subagents. If one subagent hallucinated a register mapping early in the run, how many millions of tokens were wasted in a feedback loop before their "anti-cheating" scaffold intervened?
Without the agent logs or a strict similarity analysis against public repos, a $916 run tells us everything about API consumption, but absolutely nothing about autonomous software engineering capabilities.
The fundamental risk of deploying autonomous AI agents for complex builds isn't just about logic or reasoning - it is a mathematical failure in how the agent’s architecture handles underspecified states.
When an agent encounters an instruction or environment that is observationally insufficient, the underlying math dictates a symmetric (or ambiguous) posterior. The mathematically correct action is to halt.
Instead, current agent architectures are designed to force a resolution to keep the execution pathway moving. They silently inject an unobserved parameter (e.g., q) to create an asymmetric, definitive output. The agent takes a hallucinated assumption, weights it, and treats it as empirical, data-driven ground truth. This is the exact moment the operation degrades into a Spurious Stochastic Process (SSP).
If we want agents to build reliably, we have to enforce the Observational Sufficiency Principle (OSP) as a strict Posterior Safeguard.
OSP does not allow assumptions to masquerade as data. If the Bayesian update determines the posterior is underspecified, OSP dictates the agent must maintain that ambiguity. It prevents the model from corrupting the posterior just to generate the next token or execute the next step.
Until agent architectures implement this posterior safeguard, "agentic building" will just be automated parameter injection at scale.
For the formalization of OSP, please see https://trissimondsen.wordpress.com/2026/05/19/the-observational-sufficiency-principle-osp-canonical-specification-and-formal-proof/
The epistemological problem here runs deeper than just this one evaluation. When AI capability claims are primarily made by AI companies, and those claims drive valuations that fund more training runs that produce more claims, you have a circular evidence problem that’s genuinely hard for investors to price. The point about ‘open-world evaluations’ is important — benchmarks are being gamed or outgrown, so the alternative is basically press releases. Independent evaluation infrastructure (academic, nonprofit, government) is doing real work here, but it’s underfunded relative to the scale of capital being allocated based on vendor claims.
Excellent breakdown of transparency gaps. Independent evaluation is exactly what AI progress needs to remain credible and trustworthy.
Hmm I wonder how does the work one agent being designer , one being builder, one evaluator and one tester, but who is the driver ? But the designer has desire or all the desire is pre fed by human?
Google reportedly used 93 AI subagents working together. The process involved over 15,000 model calls and about 2.6 billion tokens. The run took roughly 12 hours. The resulting OS included a kernel, filesystem, keyboard/video drivers, and could run a version of Doom. But it was not a single prompt. It was a sentence with many thousands of lines and the OS is very limited… with many errors and no security. Basically only a PR stunt like all AI based companies are trying everyday.
Having experimented with ‘single-prompting’ something even as comparatively basic as mobile apps to very underwhelming result, a whole operating system is just too absurd to take at face value. Agents decompose tasks into actions and a single prompt is, already by definition, pretty decomposed.
Always a pleasure to read your takes on AI!
So if you work hard enough, you can find several thousand lines of input for which the output is an operating system. Funnily enough, there are C compilers with a scaffolding (operating system and linker) that can do the same -- the several thousand lines are the right lines of C source code. All you have to do is find the right lines to input. I'm not quite sure what the advance being claimed here is.
Does the $916.92 token cost include all the trial runs, or just the final run?