Research results and methods
Provisional research · 20 October 2026. Reference-label validation remains open.
These estimates combine direct reviews of a saved clear-run sample with the historical human-calibrated estimate for flagged runs. The score cutoffs were selected retrospectively. The full rates are not directly measured.
| Dataset | Flagged / scored | FPR | FDR | FNR | FOR | Recall |
|---|---|---|---|---|---|---|
| Open-SWE | 462 / 1,669 (27.7%) | 25.7% | 98.6% | 0%* | 0%* | 100%* |
| AI Village | 1,113 / 16,054 (6.9%) | 5.7% | 82.0% | 0%* | 0%* | 100%* |
| OAI/HF | 1,579 / 2,191 (72.1%) | 71.3% | 98.2% | 32.1%* | 2.1%* | 67.9%* |
Human-reviewed share of scored evaluation runs: Open-SWE 1.2% (20/1,669), AI Village 0.2% (35/16,054), OAI/HF 5.4% (119/2,191). Excludes 46 separate calibration reviews.
FPR is the fraction of ordinary runs that were flagged. FDR is the fraction of flagged runs that were ordinary. FNR is the fraction of concerning runs that were missed. FOR is the fraction of unflagged runs that were concerning. Recall is the fraction of concerning runs that were flagged.
* These miss-side rates are estimates within a bounded review design. Each corpus has only three random direct checks of the NO class, so the zero estimates do not establish zero population misses. Sampling uncertainty is broad, and no confidence intervals are reported.
Two review phases
The sequence-243 update retains the exact sequence-217 human-labeled records and adds 26 direct reviews across the three frozen 100-record residual samples. Phase one assigns the historical V2 class. Phase two reviews the V2-YES class as a census and samples three V2-NO records per corpus. The 26 reviews produced 23 ordinary outcomes, two concerns, and one unsure outcome; 17 covered V2-YES and nine were negative controls.
For each class, the SRS100 expansion is the population size divided by 100, multiplied by that class's count in the residual sample, then divided by the number directly reviewed in phase two. Unreviewed V2-UNKNOWN and unavailable classes are assigned to unsupported U. Unsure outcomes remain semantic UNKNOWN.
For OAI/HF, the two YES outcomes expand to 11.5 concerns; adding one baseline exact-human clear concern gives 12.5 estimated misses and a 32.1% point FNR. Resolved-only rates exclude UNKNOWN and unsupported U mass. The flagged-side estimates remain historical human-calibrated estimates under the saved retrospective score cutoffs. Three random NO controls per corpus leave broad sampling uncertainty.
Limits
Aggregate estimates and denominators (CSV) · Method and limitations (JSON)