The Open ASR Leaderboard adds private tests to stop gaming

The Open ASR Leaderboard, a public ranking of speech recognition models, has added a set of private test sets to make it harder to game. The problem is the familiar one Goodhart described: once a public benchmark becomes the target, a model can climb it by training on the test data, finding data that looks like it, or tuning for the narrow case the benchmark covers, often American scripted English. None of that means the model is better in the real world.

The fix is eleven private splits supplied by two data vendors, Appen and DataoceanAI. They span American, British, Australian, Canadian, and Indian accents, both scripted and conversational speech, with a roughly even gender split. The leaderboard reports aggregate numbers only, such as average word error rate and separate scripted, conversational, US, and non-US averages, and deliberately publishes no per-split scores so nobody can optimize against a specific set. The private data is off by default and shown as a "Rank Δ" you can toggle on, which tells you how much a model's standing depends on the public sets it may have seen. The Hugging Face writeup is candid about the weak spot. A researcher pointed out that the same vendors sell training data to ASR companies, so the providers committed not to sell this exact evaluation data, and community datasets can be added through the project's GitHub.

Why it matters

If you pick a speech model off a leaderboard, the gap between its public score and its private score is the real signal. A large Rank Δ means the ranking was measuring test familiarity, not transcription quality.