METR has updated its assessment of AI capabilities by adding longer and more judgment-intensive tasks. The results indicate that some capabilities continue to progress exponentially, but that models perform less well in judgment exercises than programming.
Estimated durations and difficult assessments
The durations allocated to new tasks are mainly based on estimates. Systematically measuring human time would require paying people for hundreds of hours for certain exercises, or mobilizing the rare experts capable of carrying them out successfully. Both the cost and the delay make this approach prohibitive.
The measurement of judgment has another fragility: for the exercise devoted to philosophical arguments, only one person noted most of the arguments. The agents therefore obtain a score according to their agreement with this evaluator. Consulting more experts could change the results, but would require additional time. Within METR itself, employees do not all share the same opinion on the relevance of this exercise for the automation of R&D in AI.
The difficulty in constructing and then noting these exercises also explains their low number, which limits possible predictions. METR technical staff member Matt Putz says judgment-related activities are particularly difficult to study and quantify. He does not expect to be able to draw trends as solid and precise as for tasks whose criteria are simple to define.
Model proposals remain unusable at METR
To test skills likely to count in AI research, METR asks models to invent assessments that its teams could use. Its employees then examine the proposals. Although the most advanced systems outperform almost all humans at software engineering, none of the assessments produced by the AI have been deemed useful to METR’s research by its employees. On the other exercise, which consists of appreciating philosophical reasoning, the models also remain below the human level.
The results nevertheless show signs of progress. Recent systems, including Opus 5.5, tend to reach the “borderline above the bar” category for assessment design, without producing tools usable by the organization. A year ago, all fell into the “clearly below the bar” category.
OpenAI reported a comparable gap in its own usage on Wednesday. The company indicates that its models are less successful in the real missions entrusted by its employees than the simpler exercises, with verifiable results, of the initial METR battery.
The evening letterFree · 7 p.m.
Are you passionate about AI research?
The papers and advances that matter, explained simply, every evening. Free.
Included upon registration: our selection of the best AI guides & comparisons.
Choose your pace
Free · No spam · Unsubscribe in 1 click
The graph measures human working time
In the initial system, humans and models performed the same exercises: the duration necessary for the former was used to define the difficulty, then to situate the successes of the latter. This complex method of measurement contributed to misinterpretations of a widely distributed graph. It showed that the human duration of tasks completed by the best systems doubled every four months. Progress on verifiable tasks does not appear to be slowing down.
The expanded battery includes more exercises, some of which would require much more time for humans, as well as activities with results that are less easy to decide. These were absent from the original system precisely because their evaluation was difficult. Choosing a new research direction, for example, does not provide the same signal of success as a mathematical solution or a program whose operation can be verified automatically.
This difference also occurs during training. Reinforcement learning consists of multiplying attempts, then favoring the paths associated with successes and weakening those linked to failures. Automatic answer checking promoted performance in mathematics and programming. This method is less easily applied when success remains ambiguous or can only be appreciated after a long delay.
Researchers differ on automated search
The ability of models to transfer their progress on testable problems to open-ended activities remains uncertain. These would be essential to the eventual automation of research and development. Earlier this month, METR estimated, in another assessment from which many passages had been removed, that AI could double the pace of AI progress at Anthropic. This perspective has fueled, in part of the AI community, fears of complete recursive self-improvement: systems would then design their own successors without human intervention.
Sayash Kapoor, computer scientist and future professor at UC Berkeley, believes that knowing how to select the right questions, hypotheses and experiments will constitute a major obstacle to this automation. The current failures in open exercises do not surprise him: in August, he and his colleagues indicated that the models did not yet know how to reproduce human research in computer science on their own. He reports a competing interpretation, however, that training on testable problems will eventually produce success in open-ended tasks as well.
Kapoor welcomes more creative assessments and considers the expansion undertaken by METR promising, while considering that considerable work remains necessary. Najoung Kim, a linguist and computer scientist at Boston University, sees in the improvements observed information on present capacities, but says he is uncertain about their extrapolation. The specialists consulted still consider it premature to give a definitive answer on the possibility of automated AI research or its timetable.

