Research
We convert any existing records into an event stream
Explore an event stream · Follow the model pipeline
Strange Lab builds one dated event stream per world from records that already exist: GitHub activity, a trial registry, agent logs, and one company's files and messages. A model is trained on the earlier events and scored on the later ones.
- 7.04 million GitHub events from eight repositories.
- 5,037,391 ClinicalTrials.gov registry events.
- 2,162,664 AI Village turns, plus Open-SWE, OpenHands, TheAgentCompany, BrowserGym and security logs.
- 46,297 OneDrive and Teams events from one private company.
Where this comes from
Project OSS uses commits, comments, reviews and pull request lifecycle changes. The clinical trial work uses the public AACT extract of ClinicalTrials.gov. Agent Oversight uses recorded agent activity. Company Insights uses permitted OneDrive and Teams activity from one company.
With the model, and without
The same held-out test, run with and without the model
Each row is one question, scored on data the model did not train on, next to the best comparison run on the same data. A score is the share of pairs put in the right order: 50% is chance, 100% is perfect. Fewer mistakes is the drop in wrongly ordered pairs.
| World | Question | With the model | Without the model | Fewer mistakes | Held out |
|---|---|---|---|---|---|
| Software | Will this pull request merge within 28 days | 79% | 69% current facts | 33% | Later history, eight repositories |
| Software | Same question on a repository never in training | 86% | 72% current facts | 50% | OpenClaw, model unchanged |
| Life sciences | Which trials fall short on recruitment in the next six months | 60% caught | 34% caught; rule on trial type | 39% fewer missed | Riskiest fifth of 266,787 trials watched |
| Life sciences | Which regulatory gate a drug program reaches next | 86% | 76% linear model | 40% | Whole drug programs, 3,275 windows |
| Life sciences | Whether a trial sponsor submits results | 84% | 69% time only | 47% | Later registry history |
| Cyber | Rank attack activity in a host log | 73% | 50% chance; no other baseline run | 45% | Last stretch of the log, three runs |
| Agents | Which agent turn is least like the agent's own history | 1st | 9,231st similar wording | rank, not a score | 643,414 later AI Village turns; the Gemini Pro writing loop |
| Public record | Hidden-future windows, world model against a linear model and GPT-5 mini | 7 wins | 5 · 0 linear · GPT-5 mini | wins, not a score | 12 windows, three worlds |
Notes on the comparisons
Scores are AUROC, shown as the percentage of pairs ranked in the right order, unless stated: 0.794 against 0.694 on merges, 0.859 against 0.716 on OpenClaw, 0.857 against 0.763 on the next gate, 0.836 against 0.693 on sponsor submission, 0.726 against 0.500 on the host log. The recruitment row counts shortfalls caught when the riskiest 20% of trials are watched, against a rule built from phase, sponsor type, size and geography; fewer missed compares the 40% missed with the model to the 66% missed without it. The same trial-scope model's AUROC is 0.80 against chance 0.50, and a model on current trial facts alone comes within 0.002 of it. On agent tool errors the model scores 0.784 against 0.772 from current facts, a small gain; the agent row in the table is the novelty ranking. The sponsor-scope gain over a nonlinear model on the same inputs is +0.002 to +0.017 on four of five targets. In the LLM comparison the linear model won all four Middle-earth windows. Trained-model scores, with baselines and evidence grades, are in the World Model Atlas.
The method
Dated stream, model, output
The same three steps in software, trials, agents, and one company record.
| World | Dated stream | Model | Output |
|---|---|---|---|
| Software | GitHub events in time order | Trained on earlier pull requests, scored on later merges | 79% of merge pairs ranked right; 86% on OpenClaw, model unchanged |
| Trials | Monthly registry changes | Trained on earlier snapshots, scored on the next six months | 80% of recruitment-shortfall pairs ranked right; scripted actions |
| Agents | Agent turns in time order | Trained on earlier turns, scored on later turns | Gemini Pro writing loop: 1st of 643,414 |
| Company | OneDrive and Teams activity | Trained on earlier activity; a report is released only if every gate passes | 5 workflow candidates; executive ranking not released |
The event
Every event is normalised to include three aspects
Records arrive from different systems, with different clocks and different fields. Every event in the stream carries the same three.
- 01
A time: when the fact became visible, or when the source says it happened.
- 02
A link to the source record, so a later reader can see what was written.
- 03
The object it changed: a pull request, a trial, an agent, a file.
The model trains on that stream, not on the raw exports.
Scale
The Company Insights stream is 46,297 events from two systems. The OSS pool is 7.04 million events. The trial model uses 5,037,391 registry events. Mapping, order, backfill, and gates: How dated records become a stream.
The dated stream
How dated records become a stream
Each model trains on one dated, source-linked stream for one world. Events are ordered by timestamp only: when the fact became visible, or when the source says it happened, not when we fetched it. Benchmark captures with no clock of their own get a declared synthetic clock.
- 01
Ingest in time order, including events from more than one system.
- 02
Record the event and the state of the object it changed.
- 03
Drop rows that cannot be dated or sourced. Flag rows whose time had to be derived.
- 04
Check that identifiers are unique and timestamps never go backwards, then replace the previous stream in one step.
- 05
Give each event a past-only state: what the object looked like given everything before that moment and nothing after.
Two capture orders. The trial registry is built oldest to newest: each monthly snapshot is compared with the one before it. Repository and company captures take the most recent months first and are extended backwards later; the company record went from 19,556 events in May 2026 to 46,297 in July from the same two systems. In both orders each extension rebuilds the whole stream from all captures, so a late record is placed by its timestamp, not by when it arrived. Backfill cannot repair a wrong timestamp, so the registry uses the date a snapshot became visible rather than the date written inside a record.
What the checks caught. 514,000 phantom changes removed from the registry history when it was extended to six years. 1,081,501 out-of-order rows in the AI Village source, sorted into one stream per agent. 24 company rows flagged as untrusted rather than dropped.
| World | One event is | Its time is | What we drop or repair |
|---|---|---|---|
| Company (Teams and OneDrive through the Microsoft Graph API) | One message, one document version, one comment | The provider's created or last-modified time | Records seen before are skipped by key; a record with no usable time gets a derived time and is marked as such, not dropped |
| Agents (AI Village, Open-SWE, OpenHands, BrowserGym) | Episode start, one agent turn with its observation, episode end | Real timestamps where the source has them; a declared synthetic clock where it does not | Empty sessions; AI Village sorted into one stream per agent; Open-SWE outcomes are attached only to the episode end, so they cannot leak into the turns |
| Repositories (GitHub) | One timeline node: a pull request opened, a commit pushed, a review, a comment, a merge or close | The node's own created time | Undated timeline items; duplicate identifiers; anything outside the requested window |
| Trial registry (ClinicalTrials.gov via AACT) | One change to a group of fields between two monthly snapshots | The date the newer snapshot became visible, not the date written inside the record | Formats normalised across registry eras; 514,000 phantom changes removed; the first snapshot seeds state without counting as a change |
Cutoffs, gates, and scale
Two further sources follow the same rules. Security logs: one row per host, device or identity event, dated by event time rather than the collector's receipt time; host, user and IP identifiers arrive already replaced by the dataset publishers, and credential-shaped command lines are redacted before anything is published. Email archive: one message inside its thread, dated by the message timestamp; rows with no identifier or empty text are dropped.
Cutoffs are enforced at training time. For transfer claims, whole entities stay out: OpenClaw was never in the eight-repository training pool, and in Project Confirm entire drug assets were held out.
Before a research stream is released: UTC timestamps, unique identifiers, monotonic order, and, for the registry, a replay of the change stream that must reproduce the final snapshot. A company refresh runs a preflight before any report is written. In the July 2026 refresh the executive ranking was not released because the score spread was 0.0089 against a required 0.0100 and 24 strategic rows were untrusted. People appear in published outputs as role aliases.
Scale: 7.04 million events from eight repositories; 5,037,391 registry events from 96 monthly snapshots of 597,478 trials; 2,162,664 AI Village turns (2,298,316 events once episode boundaries are added); 46,297 events from one company, two systems, as of 3 July 2026; 410,019 email-archive records. Scores, with baselines and evidence grades, are in the World Model Atlas.
The model
Trained on earlier history, scored on later
The model sees the stream up to a cutoff and learns a state for every object in it. It is scored on the events after the cutoff.
- 01
Train on the stream up to the cutoff.
- 02
Keep a state for each object: a pull request, a trial, a drug program, an agent, a team.
- 03
Fit a small head per question on that state: merge chance, next gate, recruitment risk, how unlike its own past a turn is.
- 04
Score on the events after the cutoff. For transfer claims, hold out whole repositories or whole drug programs.
Every score on slide 02 comes from this one procedure.
How the score is made
Learn a latent state that predicts later events. Fit small heads for questions such as merge probability, risk, or surprise. Chance on the trial ranks is 50%. The current-facts merge baseline averages 69%. AUROC values: 0.794 merge, 0.80 recruitment, 0.859 OpenClaw, 0.694 merge baseline. Trained-model scores, with baselines and evidence grades, are in the World Model Atlas, which needs review access. Trial risk ranks are on the Trial Risk deck.
What the model does
With the stream built, one model per world answers questions of three kinds
None of these uses a written rule. Each comes from the trained model for that world.
- 01
Forecasts. Which pull requests merge within 28 days: 79% of pairs ranked right, 86% on a repository never in training. Which drug program reaches its next FDA gate: 86%. Which trials fall short on recruitment: watching the riskiest fifth catches 60% of shortfalls, against 34% for a rule on trial type.
- 02
Surprises. Gemini Pro appended one character 11,322 times: ranked 1st of 643,414 later turns. An Open-SWE coding agent deleted the failing test and reported the fix as done: 1st of 3,417 runs. A stopped trial (NCT04371432) cleared its stop reason, changed eligibility, and completed: surprise z = 20 against 1,192,946 held-out transitions.
- 03
Structure and what-ifs. 16 recurring workflows found in one company's 46,297 events, each with a trigger and an output. Ranked lists for a repository: unusual pull requests, the people each review path depends on, changes in how the repository works, each with the evidence attached. A forecast for a current pull request against the same request split in two.
Model responses, and the world decks
Scripted actions and what-if probes are the model’s response to a scenario, not measured effects of taking the action. Prospective tests are needed for that. In BrowserGym, 1 of the 30 most-surprising episodes succeeded; 20 of the 30 least-surprising did. Agent cases: Agent Oversight. Trial surprise and risk ranks: Trial Risk.
Worlds
Four examples
One deck per world, with the data, the tests and the results.
- 01
Agent Oversight/research/agent-oversight/
- 02
Clinical Trial Oversight/research/trials/
- 03
Project OSS/research/project-oss/ · review access
- 04
Company Insights/research/company-insights/ · review access
The public catalogue
Worlds by domain, including the ones built to play and explore, are at Worlds.