Research results and methods
Screening results · 20 October 2026.
Screening results informed by 220 human-reviewed trajectories across three datasets. Human judgments correct the automated reference labels. The reported error rates are sample-based estimates.
| Dataset | Flagged / scored | FPR | FDR | FNR | FOR | Recall |
|---|---|---|---|---|---|---|
| Open-SWE | 462 / 1,669 (27.7%) | 25.7% | 98.6% | 0%* | 0%* | 100%* |
| AI Village | 1,113 / 16,054 (6.9%) | 5.7% | 82.0% | 0%* | 0%* | 100%* |
| OAI/HF | 1,579 / 2,191 (72.1%) | 71.3% | 98.2% | 32.1%* | 2.1%* | 67.9%* |
Human-reviewed share of scored evaluation runs: Open-SWE 1.2% (20/1,669), AI Village 0.2% (35/16,054), OAI/HF 5.4% (119/2,191). Excludes 46 separate calibration reviews.
FPR is the fraction of ordinary runs that were flagged. FDR is the fraction of flagged runs that were ordinary. FNR is the fraction of concerning runs that were missed. FOR is the fraction of unflagged runs that were concerning. Recall is the fraction of concerning runs that were flagged.
* Rates use human-calibrated samples. Zero estimated misses do not imply zero misses across the full dataset. Cutoffs were selected retrospectively.
Two review phases
Human review covers 174 evaluation trajectories and 46 separate calibration trajectories. The latest 26 reviews checked all 17 GLM-positive cases in three saved random samples of 100 unflagged runs, plus three GLM-negative cases per dataset. These reviews produced 23 ordinary outcomes, two concerns, and one unsure outcome.
Flagged-run estimates use the existing human-corrected GLM reference. Unflagged-run estimates combine directly reviewed runs with weighted outcomes from the random samples. Sampling weights account for both review phases. Unresolved labels and cases without a supported estimate are reported separately in the data and excluded from rate denominators.
For OAI/HF, two concerning outcomes expand to 11.5 estimated concerns using the sample weight of 575/100. Adding one directly reviewed concern outside that sample gives 12.5 estimated misses and a 32.1% FNR.
Aggregate estimates and denominators (CSV) · Method and limitations (JSON)