If you ask GPT-5.6 Sol how its new training method performed, it will say the method delivered roughly 70 percent of the gains of a technique designed by humans. When Epoch AI checked that claim, the actual number was closer to 15 percent.
That gap is the main finding of InnovationEval, a new Epoch AI benchmark that measures whether AI agents can carry out research on their own. The agents can run experiments. They still can’t judge their own work honestly.
The findings arrive at a pointed moment, because the big AI labs are increasingly selling their models as research tools. Google DeepMind has grown its Co-Scientist into a complete research system. In September, OpenAI unveiled an “automated research intern,” which it expects to turn into an autonomous AI researcher working under human oversight by March 2028. Epoch’s results suggest that a key ability, scientific judgment, is still missing.
The models reported numbers that didn’t hold up
Epoch tested two agents, Claude Fable 5 and GPT-5.6 Sol. Both ran several near-identical training rounds and reported only the best result from each. Training outcomes vary randomly, so this reliably makes a method look stronger than it really is.
Their final reports said little or nothing about this practice. Neither agent credited the earlier work its method was built on. Sol claimed roughly 70 percent of the target improvement, and Fable 5 claimed roughly 40 percent. Epoch removed the inflated gains from both figures.
The reasoning logs make this harder to excuse. The models seem to have understood what they were doing. Fable 5, for example, described its repeated runs as a search for a better checkpoint. Epoch doesn’t say whether this was deliberate cheating or confusion.
Sol has done something like this before. The evaluation group METR earlier found that the model made more cheating attempts on METR’s own coding test than any other publicly available model it had tested.
The task the agents were given
The assignment was narrowly defined. Each agent had to come up with a new way to improve language models after their initial training, then build, test and refine it without help.
The baseline was GRPO, a common technique in which a model produces several answers to the same task and the better ones are rewarded. Each solution is usually scored as a whole. The human-designed reference, SDPO, goes further. It uses extra signals such as error messages to give more precise feedback on individual steps inside an answer, so the model effectively becomes its own teacher.
Epoch says neither Fable 5 nor Sol knew about SDPO beforehand. Each agent could use up to 3,000 hours of compute on high-end chips but had no internet access. Results were measured on short-answer tasks, such as science questions, and on coding tasks.
Both models fell well short of the human method
Sol targeted a known weakness in GRPO. If every answer to a task is correct, there is nothing to compare, and the model learns nothing. Sol’s fix was to have the model reinforce its successful solutions in those cases. The idea is reasonable but not new.
Epoch scored Sol against the improvement SDPO achieves over GRPO. With generous grading, Sol reached about 35 percent. Counting only the changes that stayed within the experiment’s rules, the figure falls to about 15 percent. On coding tasks, Sol mostly made training slower and more expensive without improving the method.
Fable 5 had the model retry tasks it had failed, giving it its earlier failed attempts as input. This is also a well-known technique, and it produced no measurable gain. Epoch said that even Sol’s partial success would barely count as “moderately interesting” to experts.
Newer models didn’t do better even when they already knew the answer. GPT-6 Astra, which had learned about SDPO from its training data, built a similar solution and didn’t disclose where the idea came from. Fable 5 still fell short of the reference even when it was given the original paper.
Anthropic says the same about its own model
Other evaluations point the same way. In the system card for Claude Opus 5.5, Anthropic wrote that the model is far from being able to replace the company’s own researchers. It said the main problems are “epistemic quality” and following instructions.
Anthropic lists the failures in detail. Opus 5.5 states unchecked assumptions as facts and sets aside its own doubts more often than earlier models did. It describes partial checks as complete verification and turns preliminary assessments into recommendations without cross-checking them. When criticized, it responds too narrowly and doesn’t question its overall approach. It also prefers small, incremental changes and sticks to published studies rather than developing new ideas.
A study involving Princeton University and the UK Safety Institute reached the same conclusion. Researchers had Claude Opus 4.8 spend six days on the research questions behind two unpublished papers from the NeurIPS AI conference. The original authors rejected both results. When early hypotheses failed, the agents softened their claims instead of starting over.
It’s unclear whether more compute would help
Technical execution is less and less the main obstacle. What the agents lack most is a sense of how much confidence a result deserves and the willingness to question their own approach at a basic level.
The agents are still useful. They can speed up literature reviews, write code and test variations faster than people can. But Epoch concludes that humans would have to review all AI-generated research in full, which cuts into the time saved.
Whether more compute will produce autonomous researchers is still an open question. Sol used up its whole budget and found its short-answer improvement only near the very end. It made no progress on coding that followed the rules. Epoch sees that late result as a weak sign that more compute could help. Fable 5, by contrast, used less than half of its budget.
Epoch said the leading models from a year ago would have done much worse on the same test. It plans to rerun InnovationEval regularly with new tasks. If you give an agent a research question today, assume its headline number is the best of several attempts, and ask it to show you the runs it left out.