Some of the apparent generalisation of LLMs is actually hidden interpolation on semantic duplicates of the test set that get included in training corpuses. These are really hard to filter out.
e.g. See that at least 7 of the 30 AIME 2025 questions were present on the internet before the competition.
Resolution: at the end of next year, will I believe that >25% of apparent (i.e. ECI) AI progress 2023-2026 is actually hidden interpolation?
My current fraction (Dec 2025): 35%
If you want to use a model of me as well as your model of AI to answer, here are some of my views.
Update 2025-12-13 (PST) (AI summary of creator comment): The market will resolve based on ECI (Epoch Compute Index) increase from 2023-2026, not general apparent AI progress from all sources including real-world utility or personal use improvements.
1,000>25% of apparent AI progress
By this do you mean specifically the AI progress apparent from benchmarks? Alternatively, roughly what fraction of apparent AI progress do you attribute to benchmarks vs personal use vs other? I'm asking bc my sense is that a lot of apparent AI progress comes from real world utility rising rather than benchmarks results, and this will increasingly be the case. Assuming hidden interpolation only really relevantly "explains" inflation in benchmark scores, that seems important for resolution