Insights · 6 min read
Never-Published Data and Benchmark Contamination, Explained
If a model has already seen the answer, the test proves nothing. Data that was never public cannot have been seen.
A student who has seen the exam beforehand scores well and learns little. AI models have the same problem. When a test, or the material it is built from, sits in a model’s training data, the score stops measuring ability. This is called benchmark contamination, and it is one reason AI labs pay attention to data that was never published.
How contamination shows up
A 2025 study examined SWE-Bench Verified, a popular test built from public GitHub repositories. Given only an issue description, with no view of the repository, leading models could name the correct files to edit up to 76 percent of the time on those tasks, versus up to 53 percent on repositories outside the benchmark. When asked to reproduce the exact fixed function, they matched up to 35 percent of consecutive five-token sequences on the benchmark, versus up to 18 percent on other coding benchmarks. The authors say the gains may partly come from memorization rather than reasoning.
In February 2026, press coverage reported OpenAI saying that SWE-bench Verified had lost its value as a meaningful measure of coding ability. It reported that many tasks and solutions had leaked into models’ training data and that several leading models could reproduce some original fixes from memory. OpenAI recommended a harder successor benchmark and said it was building non-public tests.
What private data does differently
The successor benchmark is useful because of where its data comes from. Its public and held-out sets use repositories under copyleft licenses, which the authors say makes them unlikely to appear in proprietary training sets. For its commercial set, they contacted startups and purchased their private engineering repositories. The results differ sharply: one leading model solved 41.8 percent of public tasks and 15.7 percent of commercial ones, and the authors note that the best models score under 20 percent on the commercial set.
Another route to clean evaluation is time. One benchmark collects new contest problems continuously, so problems that postdate a model’s training can be selected. Private business records offer a third route: they never entered anyone’s training set in the first place.
How a private test set gets built
Private data does not become a clean test by existing. In the benchmark above, engineers built a runnable environment for each repository, with system packages, build tools and dependencies. They ran the reference tests repeatedly to confirm that they pass consistently and dropped problems that failed the check. People then reviewed each test that must fail before the fix and pass after, and dropped any that were irrelevant to the task description or too broad. In other words, the value of private code as a test comes from this verification work as much as from privacy.
Three kinds of data
| Kind | May models have seen it? | Fit for clean testing? |
|---|---|---|
| Public repositories and posts | Likely | Poor; scores may reflect memory |
| New public material after a cutoff | Not yet | Good, but only for a while |
| Private work records, never published | No, unless leaked | Strong, if checkable outcomes exist |
What “never published” has to mean
Value comes from the claim being true. Before you make it, check:
- Public mirrors. Forks, mirrors and open-source extracts of your repositories count as published.
- Pasted snippets. Code or logs shared on public forums, even in a question, can be scraped.
- Former staff. Material copied to personal public accounts is out of your control; note it.
- Open-sourced parts. If you released a component, treat that part as public.
- Leaks through earlier sales. Data licensed once may have been used to train a model, which affects its later use as a test set.
Training and testing are different uses
Contamination matters in two ways. For testing, it makes scores meaningless. For training, it means a model that has already seen public material gains little from seeing it again, while private material is genuinely new. Labs therefore have two reasons to want records that never left your systems: they add knowledge the model lacks, and they can be held back as honest exams.
Contamination is a spectrum
Exposure is rarely all or nothing. A repository that was public for one month, a few files pasted into a forum or a library that was open-sourced years ago leaves a partial footprint. Partial exposure lowers value less than full exposure, but it can still decide whether a set works as a clean test, because even a small leak of answers can inflate scores. Mapping exactly what was exposed, and when, is more useful than a flat yes or no.
What to tell a buyer
If you claim that data was never published, be ready to support it with facts: where the repositories were hosted and whether they were ever public, whether any component was open-sourced, which mirrors or forks exist, and when. A careful statement with limits is worth more than a broad promise that turns out to be false. Buyers who find one false claim tend to question the rest.
Why this matters for price
Private data that can be turned into checkable tasks serves two buyers at once: teams that train agents and teams that need honest tests. Epoch’s interviews with practitioners say labs favor tasks that are valuable and quantifiable, and that exclusive terms can cost four to five times more. A clean test set is a strong candidate for exclusivity, because a test that many labs train on stops being clean.
Honesty point: never published does not mean valuable. A private archive of routine, easy tasks is still routine. The data also needs the properties covered in /insights/what-makes-data-valuable-for-ai-training: real problems, checkable results and clean rights.
The free estimate at /estimate treats code with history and linked sources as the core of value. If your records were never public, that is a point in your favor. Compare it with the scraped alternative in /insights/scraped-data-vs-proprietary-data-value.
Questions we get
- What is benchmark contamination?
- It happens when a test, or the material it is built from, appears in a model’s training data. The model can then score well by remembering instead of reasoning, so the benchmark overstates ability.
- Why is private data better for testing AI?
- Because models cannot have seen it. In one benchmark, models solved far fewer tasks from purchased private repositories than from public ones, which suggests public scores partly reflect memory.
- Does never-published data always sell for more?
- No. It needs real, checkable tasks and clear rights too. Being private only removes the contamination problem.
- Does licensing my data to one AI lab contaminate it?
- It can. Once a lab has trained on it, the lab can no longer use it as a clean test. That is one reason exclusive terms are priced higher.
What is your data worth?
Sources
- Liang, Garg and Zilouchian Moghaddam, “The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason” (2025)
- The Decoder, report on OpenAI retiring the SWE-bench Verified benchmark (February 23, 2026)
- Deng et al., “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” (2025)
- Jain et al., “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code” (2024)
- Epoch AI, “An FAQ on reinforcement learning environments” (2026)