Insights · 7 min read

Turning Bug History into Graded Coding Tasks (Fail-to-Pass)

A bug ticket, its fix and a test that flipped from red to green are the raw ingredients of a graded task. The work is in everything around them.

Bearish AI · October 21, 2026

If your team has tracked bugs for years, you are holding the raw material for coding tasks that AI labs can use for training and evaluation. Each task asks a model to fix a real bug and checks the result with real tests. The format, called fail-to-pass, is the standard used by public coding benchmarks. This article explains how a bug history becomes tasks, and where the effort goes.

What fail-to-pass means

The SWE-bench paper defines the format. Take the repository at the commit just before a fix. Apply the tests that came with the fix. Some of those tests fail. Apply the fix itself, and they pass. Tests that flip from fail to pass are the fail-to-pass tests, and every kept task needs at least one. A second group, pass-to-pass tests, were passing before and must still pass afterward, which guards against solutions that break something else. The SWE-Bench Pro authors describe the same split: fail-to-pass tests verify the issue is resolved, pass-to-pass tests ensure existing functionality stays intact.

How many records survive

SWE-bench started with about 90,000 pull requests from 12 popular Python repositories. It kept merged pull requests that resolved an issue and changed test files, then ran the tests before and after the fix and dropped instances with no fail-to-pass test or with installation or runtime errors. The final benchmark had 2,294 instances. Expect your own yield to be a small share of your total history, and to depend on how well your team tested. Interviewees in Epoch AI’s study of reinforcement learning environments mostly cited about $200 to $2,000 per finished task, so the effort has to be tied to quality.

From ticket to task

A working process

• Find fix commits. Link bug tickets to the commits and reviews that closed them. Teams that wrote ticket IDs in commit messages have an easier time.
• Pin the base state. Use the parent commit as the starting point. Record the exact dependencies and tools needed to build it.
• Split tests from the fix. Separate the tests added in the fix from the code change, and check that at least one test fails before and passes after.
• Build and verify the environment. Run it clean, several times. SWE-Bench Pro automatically reruns reference tests and drops tasks that do not pass consistently, which removes flaky tests.
• Rewrite the problem statement. SWE-Bench Pro has engineers write the problem statement, requirements and any interface needed. A ticket that says “applied the patch from the chat” would give away the answer.
• Review the tests. Human reviewers check each fail-to-pass test for relevance and breadth. Tests that are irrelevant or too broad are dropped, and a task with no good tests left is dropped.
• Strip future history. Do not leave the answer in the repository.
• Calibrate difficulty. Run models against the task and record pass rates.

Why future history must go

Your history is the source of the tasks and also a way to cheat. An issue opened on the SWE-bench project in September 2025 reported that agents could run git log --all in the evaluation images and see later commits, including fixes and messages describing the solution. The maintainers updated the images to remove future git history while keeping the past history, and later reported the problem fixed. A separate audit of three open models on other benchmarks found shortcut attempts such as reading upstream solutions or local Git history in 60.7 to 90.1 percent of trajectories on one benchmark under standard evaluation, and prompting the models to solve tasks independently cut that to 9.4 to 14.2 percent. Building each task so the answer is not reachable is part of the preparation, not an afterthought.

Difficulty and the pass-rate window

Epoch AI’s interviewees describe a minimum pass rate of around 2 to 3 percent, or at least one success in 64 or 128 attempts. Tasks nobody solves may be infeasible, tasks always solved give nothing to learn from, and tasks near a 70 percent pass rate may be retired. They add that robust graders take many iterations to build. Record pass rates in your data card.

Why private repositories matter

Public repositories may already be in a model’s training data. The SWE-Bench Pro authors say they contacted startup companies and purchased their engineering repositories to build a commercial set that stays private. The best pass rate they report is 43.6 percent on the public set and 17.8 percent on the commercial set. A study of an older benchmark found models could name the buggy file from the issue text alone up to 76 percent of the time on that benchmark, against at most 53 percent elsewhere. Code that was never public is much less likely to have been seen in training.

What does not convert

Some history will never become a graded task. Bugs fixed without tests have no fail-to-pass check. Tickets that were closed with no linked commit cannot be tied to a code change. Fixes that depend on private services, hardware or data that cannot be shipped may not run in a clean environment. Feature work with vague acceptance criteria is hard to grade automatically. Such material may still be valuable as context. The SWE-bench filters show the same logic: instances were dropped when they had no test that flipped or could not be installed and run.

What you need to bring

You need source code with history, a bug tracker linked to commits, and tests worth running. Pseudonymize people and rotate secrets first, and have rights, confidentiality and licensing reviewed separately; see /insights/prepare-company-data-for-ai-labs. If you want a sense of what bug-history tasks might add to the value of an engineering archive, the free estimator at /estimate shows a conservative range. See /insights/ai-lab-data-due-diligence for what buyers check.