Approach
How a lesson gets made.
Raw history is noise. So is a question with no answer key. A lesson is real work, linked into one story, cleaned of anything private, and tied to a check that can say pass or fail without a human in the loop.
The pipeline
Six steps, every time.
- 1.
Collect
We gather every system a piece of work touched: specs, chat, design files, code, reviews, CI runs and deploys. Each file is checksummed at the source so its history can be traced later.
- 2.
Link
Records are joined into one episode using names, file paths, pull request references and timing. Every link carries a confidence score and the evidence behind it. Weak links are reviewed by hand or dropped.
- 3.
Protect
People become stable role pseudonyms. Secrets, credentials and customer details are removed in two passes: automated pattern scanning, then human review. Rights, confidentiality and licensing terms are reviewed separately for every release. Anything we cannot clear, we do not ship.
- 4.
Reproduce
For tasks, we rebuild the failing and fixed states straight from version control and run the project’s own build and tests. A task ships only if the broken state fails and the fix passes, every time.
- 5.
Grade
Code tasks are graded by visible and hidden tests. Workshop tasks are graded on the final state of the system, never on what the agent says. Language tasks use exact-match fields plus a weighted rubric.
- 6.
Document
Each release ships with a data card: what is in it, how it was made, what was removed, and the provenance of every file.
Guild scenarios clear the same bar. Each one is written by a professional who does that work in the region, built on the documents and rules of that market, from Saudi e-invoicing to Gulf payroll, and graded on the final state of the books, the order or the inbox.
Difficulty
Three tiers, graded three ways.
| Tier | What it asks | How it is graded | Example |
|---|---|---|---|
| Tier 1 | A single change, often in one file. The hard part is the diagnosis. | Visible and hidden tests | A recurring event that drifts across daylight saving time |
| Tier 2 | A bug that has to be traced across files from a report or a failing build. | Tests plus a scope check on what changed | A refund that double-counts tax on mixed-rate invoices |
| Tier 3 | A multi-step business workflow across apps, documents or languages. | State checks on the final environment | Match a German supplier’s invoice to a Riyadh buyer’s order and receipt, pay what is owed, and write back in German |
See one of each, fully worked, on the samples page.
Why history matters
Same feature, three architectures.
Bearish OS, an internationally deployed enterprise platform, has grown through three generations, from a first-generation Express and MongoDB stack to a NestJS and Postgres platform with a multi-agent layer. Many features exist in all three versions, each with its own decisions, bugs and fixes. That lets a model learn why designs change, not just what they look like at the end. And because the Workshop runs that same system, an agent can study how a feature was built, fix what broke in it, and then operate it live.
Our bar
What every release has to clear.
Never public
Our source work was never on the open web, so models can’t have memorized the answers.
Deterministic
Every task is checked on repeated clean runs before release. Flaky tasks are cut.
Sealed graders
Hidden tests and reference fixes never reach the agent, in training or in evaluation.
Documented provenance
Every file has a known origin and a documented path to delivery. Rights and licensing are reviewed for each release.