Insights · 6 min read

Scraped Data vs Proprietary Data: Which One Is Worth More?

Scraped data is abundant, contested and cheap per unit. Proprietary data is scarce. Scarcity alone does not make it valuable.

Bearish AI · October 21, 2026

AI labs built their first generation of models on data they could collect from the open web. The next phase will depend more on data they have to ask for. For an owner of proprietary records, that shift is the opportunity. It also comes with caveats, and this article covers both.

The supply of scraped data

Epoch AI’s estimate puts the effective stock of quality-adjusted human public text at about 300 trillion tokens (a 90 percent range of 100 trillion to 1,000 trillion), fully used somewhere between 2026 and 2032. When a resource is large and available to everyone, its price per unit falls, and no single holder has leverage.

The supply is also getting smaller in practice. A large audit of 14,000 web domains behind common training corpora found AI-specific restrictions rising sharply between 2023 and 2024. In one widely used corpus, about 5 percent of all tokens and 28 percent or more of the most actively maintained, critical sources were fully restricted by sites’ terms or crawling instructions. The authors argue this could bias the diversity and freshness of future models.

What the big licensing deals show

Some content owners now get paid. Reddit’s public listing filing disclosed data licensing arrangements with an aggregate contract value of $203 million and terms of two to three years. News Corp’s deal with OpenAI was valued by the Wall Street Journal at about $250 million, covering use of its content and archives for answering queries and for training.

Be careful what you take from these. Those owners hold huge, fresh, public-facing archives with large audiences. Their deals are not price lists for a company’s private engineering or support records. They show that rights holders have bargaining power when they can prove ownership. They do not show what any one archive is worth.

Side by side

Why proprietary data can win

• Scarce by nature. Epoch notes that non-public data, such as instant messages, is fragmented and hard to use at scale, which also makes it rare.
• Human interaction. Researchers on model collapse argue that data reflecting genuine human interaction becomes more valuable as generated text spreads.
• Clean testing. Private material can serve as a fair test; see /insights/never-published-data-and-benchmark-contamination.
• Provenance. You know where it came from. Scraped data rarely comes with that record.

A three-question test for your own data

• Could a model have seen it? If it was ever public, assume yes and discount it.
• Could someone else produce it? If a competitor or a scraper could rebuild it, it is not scarce.
• Can you prove you may license it? If not, it behaves like scraped data with the same legal doubt.

Records that answer no, no and yes are the ones that stand apart from the open web. Most archives will answer some of these differently for different parts, which is another reason to look at them piece by piece.

Where proprietary data falls short

Proprietary does not mean valuable. Reports on wind-down data sales describe payouts of $10,000 to $100,000 per company. Duplicated, generic or hard-to-verify material is worth little whether it is private or not, as laid out in /insights/what-makes-data-worth-less-to-ai-labs. It also needs work that scraped data does not: scrubbing secrets and personal details, linking sources and clearing rights. That preparation is a cost, and it is also where much of the value gets created.

Why human records may gain value

Researchers who documented model collapse argue that as more online text is produced by models, records of genuine human interaction with systems become scarcer and more valuable. Public web data is increasingly mixed with generated material. A private archive of people solving real problems has a clear origin. That is a structural advantage, though it does not guarantee a price on its own, and it only counts when the records are prepared and the rights are clear.

How proprietary data usually gets sold

Three shapes appear in public reporting. Wind-down archives are sold as a lot, at one-time prices in the range of tens of thousands of dollars per company. Graded tasks are sold per task, mostly at $200 to $2,000 each according to Epoch’s interviews, with contracts that often reach six or seven figures per quarter for large buyers. And content owners license on multi-year terms, as in the two-to-three-year Reddit arrangements. Which shape fits your data depends on what you hold and how it is prepared. A raw archive fits the first. A prepared and graded set can reach the second.

Rights cut both ways

Scraped data’s legal uncertainty is part of its price. A $1.5 billion class settlement over books taken from pirate libraries gave the market a number for unclear provenance. Your records are different only if you can show that you hold the rights, which is the subject of /insights/data-ownership-rights-before-selling-data-to-ai-companies. This is general information, not legal advice.

A free, conservative way to see where your own archive might sit is the estimate at /estimate. It will not promise a figure the market has not paid, and it will say if what you have looks small.