A search benchmark that rebuilds its own questions every hour, so nothing can memorise it

Keenable open-sourced NEEDLE, a live search-and-retrieval benchmark that never freezes a query set. News queries are regenerated hourly from RSS feeds and Google Trends; finance, scholar, legal and rare-entity queries are rebuilt daily from SEC filings, arXiv, Europe PMC and CourtListener. It scores Google, Bing, Brave, Tavily, Parallel, Exa and Keenable itself, using an LLM relevance judge for news and rare-word queries and deterministic answer matching for finance, and archives every run to a public dashboard. The code is MIT-licensed and reproducible from the repository.

What it means If you are choosing a search backend for an agent, this is a contamination-resistant comparison you can re-run yourself — but the benchmark’s author is also one of the ranked engines.