On paper, today's AI benchmarks look like they are settling a question. Pick a leaderboard, read the top row, buy the winner. The reality, as anyone who has watched a model ace a test suite and then stumble in production can tell you, is messier. Most benchmarks still measure one thing: whether a model can reproduce a correct answer that someone already decided was correct.
That is exactly the gap a new benchmark called TRACES is aimed at. Launched this week by AI research company Apodex, TRACES drops an AI system into an executable environment and watches it work through open-ended scientific problems where the correct answer may not be known yet.
For anyone trying to make sense of model evaluations, TRACES is worth understanding, because it quietly changes the question from "did the model get it right?" to "how did the model get there, and would that process find something new?" Here is how it works, and how you can apply the same thinking the next time you evaluate an AI model for your own work.
What Makes TRACES Different From Every Benchmark Before It
Conventional benchmarks treat evaluation like a final exam. Fixed questions, fixed answer keys, a score at the end. TRACES treats it like a laboratory visit. The system can observe its environment, act inside it, call external tools, receive feedback, and revise its approach, all while working toward a verifiable outcome.
The test beds vary with the problem. A single TRACES task can bundle scientific literature, structured datasets, code execution, specialized scientific tools, simulators, folding engines, and experimental feedback into one environment.
Scoring is deliberately split in two. An outcome verifier grades the final result against hidden ground truth where one exists, while a process verifier evaluates whether the reasoning and evidence behind the conclusion meet the benchmark's criteria. Apodex frames this pair as the core of "discoverative AI," or systems built to surface new findings from existing knowledge rather than reproduce what already sits in their training data.
Lead scientist Dr. Sheng Wang says scoring only the destination misses the journey. TRACES evaluates six capabilities:
- Tools: whether the system selects and correctly interprets external tools.
- Repair: whether it identifies and corrects errors after receiving feedback.
- Alternatives: whether competing hypotheses are considered and revised as evidence accumulates.
- Coherence: whether it stays logically consistent across longer problem-solving sequences.
- Evidence: whether conclusions are grounded in observations, data, experiments, or citations.
- Scope: whether it can define where a conclusion applies and where it does not.
AI research scientist Brian Wang put the philosophy bluntly: "The TRACES process verification is what makes the benchmark unique. In scientific discovery, the answer is one line at the end of hundreds of judgments: what to try next, when the evidence is enough, when to abandon a hypothesis. The capability lives there, and scoring only the last line throws away almost all of it."
The evaluation pipeline tries to keep that judgment honest. Human evaluators match observed behavior against written scoring descriptions, with each finding linked to specific steps in the recorded trajectory, and an independent model reviews the evaluation. Disagreements trigger re-scoring and adjudication.
How to Judge Any AI Benchmark: A Practical Checklist
You do not need 423 scientific problems to apply this mindset. The same lens works when you evaluate a model for a chatbot, a coding agent, or an internal tool. Before you trust a number on a leaderboard, run it through this checklist:
- Does the benchmark test the task you actually need? A model that tops a knowledge quiz can still fail at long, tool-using workflows.
- Is the environment dynamic or a fixed list of questions? Static tests reward recall; dynamic tests reward judgment.
- Does scoring look past the final answer? If the metric ignores the process, you cannot tell a lucky guess from a reliable method.
- Can you inspect the reasoning? A score backed by a recorded trajectory you can review is far more useful than a bare percentage.
- Does it separate memory from discovery? Ask whether success requires generating something new or merely retrieving something stored.
The timing of TRACES is not accidental. The latest Stanford AI Index technical chapter flagged that frontier models gained 30 percentage points on Humanity's Last Exam in a single year, and that evaluations designed to stay hard for years are now saturating in months. When a test built to resist progress gets crushed that fast, the test is telling you less and less about what actually separates one model from another.
Earlier this month, Fast Company's Executive Board published a similar practical warning: benchmarks are more useful than rankings. A leaderboard position tells you who won, but the right metrics, measured against the right task, tell you who fits your business. TRACES is the most literal version of that argument yet, a benchmark built to be read as a process report rather than a podium.
For teams building models, agent systems, or solver frameworks, TRACES is open for participation, and researchers and organizations can also submit open scientific problems to be converted into executable evaluation environments. The full framework, including its approach to building environments and verifying outcomes, is documented in the preprint "Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence," backed by 423 high-value problems drawn from a survey spanning 561 industries across 16 sectors.
So the next time someone quotes a benchmark score at you, do not just ask who won. Ask what the test actually measured, whether the environment matched the real task, and whether the process was scored rather than just the answer. That habit will serve you better than any leaderboard snapshot, and it is exactly the standard TRACES is trying to set for the whole field.
Comments