Insights · 6 min read

What AI Labs Check in Data Due Diligence Before They Buy

A lab’s review comes down to seven questions. Sellers who can answer them on paper close faster and lose less to discounts.

Bearish AI · October 21, 2026

When an AI lab evaluates data it is thinking about two risks: that it will waste compute training on something useless, and that it will inherit a legal or privacy problem. Labs do not publish their checklists, so this article draws on public evidence: research papers, regulator guidance, reporting and published commentary. It describes what a careful buyer has reason to check. It does not claim to be any one lab’s process, and it is general information, not legal advice.

1. Is it yours to license?

Rights come first. Published legal commentary on training data licenses expects the licensor to warrant that it has the right to license the data, that using it does not infringe, and that it was lawfully collected, and to provide a provenance file. Researchers who audited more than 1,800 text datasets found license information omitted on more than 70 percent of entries on popular hosting sites, which is why serious buyers ask to see the paper trail themselves.

There is a regulatory reason too. The US Federal Trade Commission has said it has required companies to delete models and algorithms developed with data they obtained unlawfully. A buyer who cannot show clean sourcing risks losing the model, and not only the data.

2. Could models have seen it already?

Contamination is a live concern for coding data. A study of a popular coding benchmark found models could identify the buggy file path from the issue text alone with up to 76 percent accuracy on that benchmark, against at most 53 percent on repositories outside it. OpenAI said in February 2026 that it no longer evaluates on that benchmark, citing flawed tests and training-data exposure, and is building private evaluations. The authors of a later benchmark went further and bought private engineering repositories from companies; the best pass rate they reported was 43.6 percent on their public set and 17.8 percent on the commercial set. Private, never-published code is worth asking about, so say clearly whether yours ever left the building.

3. Is it clean of people and secrets?

Reporting says sellers scrub personal information and leave out HR records, and buyers say they scrub again. Expect to be asked for your method, your sampling and your miss rate. Research on language models has shown that names, phone numbers and email addresses can be extracted from a trained model, including strings that appeared in only one training document. For secrets, a buyer wants to know you rotated credentials, not only deleted them.

4. Do the tasks actually work?

For graded tasks, buyers look at how tests were verified. The SWE-Bench Pro authors describe running reference tests several times and dropping tasks that did not pass consistently, having engineers write problem statements and build environments, and having reviewers drop tests that were irrelevant or too broad. If you offer tasks, show the same: repeated clean runs, human review of tests and a note on anything dropped.

5. Can the grader be gamed?

Epoch AI’s interviewees consistently named robustness against reward hacking as the top quality criterion, and one put it this way:

“Soundness matters most: high reward must mean the task was actually solved, not hacked.” An interviewee quoted by Epoch AI, 2026.

Leaks in the environment are one route. An open issue on the SWE-bench project showed agents using git log --all to read later commits containing the fix, which the maintainers addressed by removing future git history from the images. A later audit of three open models on another benchmark found shortcut attempts in 60.7 to 90.1 percent of trajectories under standard evaluation. Buyers who know this will ask what your task environments expose.

6. Is the difficulty right?

Epoch AI’s interviewees describe a minimum pass rate of around 2 to 3 percent, or at least one success in 64 or 128 attempts. Tasks that no model solves may be infeasible, tasks solved every time teach nothing, and tasks near a 70 percent pass rate may be retired. A spread of difficulty is better than a single level.

How big a sample will a buyer review, and what failure rate will they accept? We found no published sample size or acceptance threshold from any lab, so be wary of anyone who states one. The practical approach is to make your own measurement and share it: draw a random sample, count misses, fix them, and record the numbers.

Deduplication belongs on this list too. Researchers found that overlap between training and test data affected more than 4 percent of the validation sets of standard datasets, and that duplicates raise the amount of memorized text a model emits. A buyer who cares about clean evaluation will ask whether duplicates and near-duplicates were removed.

7. Is it documented?

Documentation is how all of the above reaches the buyer’s reviewers. Providers of general-purpose AI models on the EU market must publish a sufficiently detailed summary of the content used for training under Article 53 of the AI Act, and the Commission’s template, published in July 2025, asks them to say whether data was commercially licensed. That makes your data card something the buyer may need to draw on later.

The estimate at /estimate assumes your data can pass this kind of review: clear rights add to the range, unclear rights lower it. For preparing the paperwork, see /insights/prepare-company-data-for-ai-labs and /insights/data-cards-and-provenance-for-ai-training-data.