Benchmarking a web search API is like watching a student who memorized the answer key. A search agent has a fetch tool, and if the gold labels sit in a public dataset, it can quietly download them mid-evaluation and skip retrieval entirely. The result is a near-perfect score that proves nothing about whether the search layer actually works. Keenable AI thinks it has a cleaner way: a live benchmark called NEEDLE that regenerates its questions every hour from fresh public sources, so there is no fixed answer key to game.
NEEDLE stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation, and it is now open source. Rather than freezing one static query set, the harness rebuilds its queries continuously. News is regenerated hourly from roughly 124 curated RSS feeds together with Google Trends. Finance, scholar, legal, and rare-entity queries refresh each day from sources like SEC XBRL filings, arXiv, Europe PMC, CourtListener, and public agent logs.
Why a static benchmark keeps failing
The contamination problem is not hypothetical. If test questions already sit in the training data, a model can regurgitate the answer without ever performing the task. In the search world the equivalent is sharper: the agent can fetch the dataset mid-run. A benchmark built on a fixed, downloadable answer set cannot tell you whether the engine retrieved anything at all.
NEEDLE attacks that on two fronts. First, it never locks the queries in place, so there is nothing stable to memorize. Second, it feeds the judge and the pool from the same running stream, and every engine sees identical query text, scored under one protocol. Nothing is re-ranked after the fact, pages are never fetched, and the evidence is clipped to 2,000 characters for everyone.
How the scorecard works
Fifteen search APIs run against the same query text, and their results are measured against an oracle called "ultimate," which pools everything every engine retrieved and orders it by relevance. That pooled ceiling is a useful yardstick: it shows the best result the whole field managed to find, so the gap between any engine and the ceiling is an upper bound on agentic search quality today.
Scoring splits by task type. News and deep-tail have no single correct answer, so an LLM judge rates each result from zero to four and reports nDCG@5 with a duplicate-URL penalty. Finance is an answer-recall task at five results. Scholar and legal are known-item tasks graded by identifier match.
- Finance is nearly solved: Exa scores 0.910, Keenable 0.872, Perplexity 0.871, Google 0.847, against an ultimate of 0.965.
- Scholar spreads the field: Keenable 0.774 down to Tavily 0.310, against a 0.869 ceiling.
- Deep-tail is hardest and closest to real agent traffic: Exa leads at 0.557 of ultimate, Keenable follows at 0.470, Bing sits at 0.199.
Latency tells a separate story. Agents call search many times per task, so speed matters. In the same window Keenable-realtime answered at 193 ms p50 and 284 ms p95, while Exa ran at 1,876 ms and 2,955 ms, and Bing at 2,767 ms and 9,381 ms. Across the board, the fastest engine also stayed competitive on relevance.
What the gap to the oracle really means
The most interesting number is not any single vendor score but the widening gap as queries get harder. Finance sits close to its ceiling, which suggests the field has roughly solved structured-fact retrieval. Scholar spreads out because title queries are answerable from metadata while body queries are not. Deep-tail, the category closest to how agents actually search, shows the biggest gap, meaning better results existed that no engine surfaced.
That distinction is the real insight. A large gap to ultimate means good results were out there but every engine failed to rank them well, a ranking problem. A weak ultimate score means something else: even after pooling every provider, the benchmark found little strong evidence, a retrieval problem shared by the whole market. Being able to tell those two apart is what makes a live harness more than a leaderboard.
- A fixed benchmark rewards memorization, so treat static scores with suspicion.
- The fastest engine can still lead on relevance, so judge latency and quality together.
- If finance and deep-tail gaps are that different, apply vendor results only to the task they actually resemble.
NEEDLE runs as a Python CLI using uv sync, with generate and run subcommands, an OpenRouter key for judging, and one API key per engine tested. It fits on a laptop or in CI, and Keenable publishes a live dashboard with the 7-day means. For teams auditing agentic search, a self-refreshing, pooled benchmark is a genuinely useful yardstick, and the open-source release means the method can be inspected and reused.
The takeaway is simple. When the benchmark can cheat, nobody can trust the score. When the benchmark rebuilds its questions every hour, the score starts to mean something again.
Comments