Insights · 6 min read
How to choose an AI training data provider: nine questions to ask
The questions that separate a dataset you can trust from one that quietly wastes compute.
Training an agent on bad tasks is expensive twice. Once for the data, and again for the compute burned learning the wrong thing. Epoch AI cites an industry estimate of roughly $2,400 in compute spent per task during reinforcement learning training. Here is what to ask any provider, including us, before you sign.
1. Where did this come from?
Ask for provenance down to the file. Who made it, when, and in what system? A good provider can show a checksum trail. “Our experts wrote it” is a start, not an answer.
2. Could the model have seen it already?
Public code and public benchmarks leak into training sets. Researchers at Purdue and Microsoft found models could name the buggy file for up to 76% of SWE-Bench Verified issues from the issue text alone, against under 53% on repositories outside the benchmark. Ask whether the source was ever public.
3. How is it graded?
Every task needs a check that says pass or fail without a human. For code, that means real tests plus hidden tests. For agent tasks, it means checking the final state of the system, not the agent’s explanation.
4. Can the grader be gamed?
Robustness against reward hacking is the top quality criterion industry interviewees name. Ask what stops an agent from editing the tests, peeking at a future commit or faking the output. Ask how many iterations the graders went through.
5. Is it deterministic?
A task that passes on one run and fails on the next teaches noise. Ask whether every task is checked on repeated clean runs before release.
6. Who is in the data, and how are they protected?
Real data involves real people. Ask how names, secrets and customer details are removed, and whether a human reviews the result.
7. Is the difficulty calibrated?
Tasks that every model fails, or every model passes, teach little. Ask for difficulty tiers and how they were set.
8. Was it written in my language, or translated?
Translated tasks carry the source language’s phrasing and assumptions. If you need Arabic, French or German, or Gulf and Egyptian dialects, ask who wrote it first, and whether they know how business runs in that market.
9. What exactly do I get, and on what terms?
Formats, harnesses, environment images, data cards, exclusivity and refresh cadence. Get them in writing.
How we answer
We built Bearish AI around these nine questions. Our approach page and synthetic samples show our answers. For the real thing, request the sample pack at data@bearishfh.com.
See the format for yourself.