Discussion about this post

User's avatar
Christine Mendoza's avatar

the automate vs augment part is the question i try to start with when i build for clients. not "can the agent do this task" but what should the person still own once it's in. and checking the whole trajectory and not just the output is the part nobody budgets for. a workflow that gets the right answer the wrong way is going to break the day i'm not watching.

Rohit Yadav's avatar

In the METR trial, agent tools slowed developers. That result should worry deployers most.

Trajectory review is the right standard for evaluators, but it is expensive, and an agent judging another agent inherits the same blind spots. A deployer cannot afford that on every run.

So what is the cheap signal that tells a deployer the benchmark gap has shown up in their own workflow? I would watch how often the people using the agent change or reverse its output.

Do you see override data becoming part of the evaluation ecosystem, or does it stay an internal metric?

Happy to connect, let's talk more about it.

29 more comments...

Ready for more?