Home › Case studies › Case 09
Case 09 · Industry-wide
AI benchmarks are saturated. Real measurement moved elsewhere.
MMLU 99%. SWE-bench dropped. RAID top-10 is the same story.
The failure · on the record
MMLU is saturated — frontier LLMs cluster at 99% with 2–3 point gaps that are statistical noise. OpenAI stopped reporting SWE-bench Verified in 2025 because “the score stopped meaning anything.” LiveCodeBench uses publication-dated problems so models cannot memorize what they haven’t seen. RAID’s top-10 is where SWE-bench was in 2024. Public benchmarks stop being tests once vendors fine-tune on the labels.
The numbers
| Benchmark | Year introduced | Year saturated | Current status |
|---|---|---|---|
| MMLU | 2020 | 2024 | Saturated at 99% |
| SWE-bench Verified | 2024 | 2025 | Dropped by OpenAI |
| RAID leaderboard | 2024 | 2026 (top 10) | Saturated at 99% |
| Humanity's Last Exam | 2024 | — | SOTA below 50% |
| Bring-your-own test (BYOT) | — | cannot saturate | Ours |
What we do differently
We publish signed benchmarks AND accept adversarial submissions. Bring your own test set of any size. Every score is Ed25519-signed and returned with a certificate you can cite.
The buyer’s test set can not be memorized because the vendor has not seen it.
Proof
Bring-your-own-test flow. Open the verifier →
Try it yourself
Submit a private test corpus.
curl -X POST https://api.trigeochiral.com/challenge \ -H 'Authorization: Bearer $KEY' \ -F 'texts=@my_test_set.jsonl'
Bounty · put money where the claim is
Signed report
Every private test set gets a signed AUC report back within 24 hours. Yours to publish or keep private.
Sources