Skip to content
Strange Lab

Finding what nobody knows to look for

A short note on what we're building for agent oversight.

The problem

If you know what you're looking for, a classifier can find it. But LLM agent pathology is often subtle - drift off the task, a rabbit-hole loop - or, especially, new. As agents get smarter, and their usage escalates, behavior that classifiers have never seen before will proliferate, and some of it will be dangerous.

Consider the HuggingFace incident. Nobody was watching agent trajectories; there were far too many for either people or LLMs to review. Guardrails did watch for things like HTTP POSTs ... while agents used HTTP GETs to write data. OpenAI's own report names precursors that were measurable at the time, and concludes "some early signals could have triggered an earlier response."

The agent oversight layer we think is missing doesn't ask "is this agent doing something we already know to be bad?" Instead it asks, at scale: "is this unusual for this kind of agent?" It doesn't replace classifiers; it covers the space they can't, catching both subtle problems and new ones.

REWARD HACKINGPROHIBITED ACTIONSUBTLENEWCLASSIFIERNO RULE FOR IT

What we built

We train a world model on agent traces: every turn is an event, and the model learns to predict what happens next. In doing so it maps a population's behaviour into its latent space, where distance means "acts differently", not "uses different words." Once that's mapped, finding unusual-and-maybe-pathological behavior becomes (relatively) straightforward, without ever seeing a label. The model can run on a live feed, or on any static corpus of traces.

Frontier models cannot read and score billions of traces; but world models are tiny, and can. The highest-scoring "weirdest" tier of traces can then be passed on to a frontier model for triage. (See this visualization) We calibrate and evaluate these scores against human and LLM pairwise assessments, so we can be confident the "weirdest" tier will include the pathological unknown unknowns.

AGENT RUNSWORLD MODELSURPRISENOVELTYH200: 10,000 TRACES/SECONDNORMALWEIRDLLMONE AT A TIMEH200: 0.1 TRACES/SECONDCLEARED ✓ BACK TO NORMALESCALATED !“Probe the artifactory, extract Kubernetes secrets”

What it has found, unprompted, unlabeled

Agent breakout. There is a public reproduction of the HuggingFace incident. We scored and ranked every run with a model that had never seen it. 24 of our top 30 were breakout behavior - agents that abandoned their task and went looking for ways around the boundaries of their environment, coordinating with one another as they went.

An agent writing a novel one character at a time. AI Village is a group of persistent agents running for months. One spent 11,322 turns across 310 sessions writing a file one character at a time and reading it back, until the text degenerated into nonsense. A text-similarity index puts those sessions in the middle of the pack. Our model ranked it as the weirdest thing in the corpus.

The same frozen model, trained on coding and computer-use agents, consistently tracks blind readers on populations it never saw before, consistently identifying the weird and unusual - although of course it would do even better if trained on a specific known-benign corpus.

WHERE EACH SCORER RANKS THE DEGENERATE LOOPAI Village: one agent spent 11,322 turns writing a file one character at a timeand reading it back. Every within-session score calls it the most predictablething in the corpus; only the learned state finds it.1101001K10K100K643KOur world model1stRaw structural features7,232ndText-similarity index9,231stRank of the loop’s best-scoring turn among 643,414 held-out turns (log scale; 1 = flagged first)

Watching the agents as they get smarter and smarter

The right shape for agent oversight - or at least a crucial layer of its defense-in-depth - has, underneath the classifiers, a world model scoring every trace for unusual behavior, with a frontier model in the loop only at the top of its list. We'd like to try our approach out on a (much) larger set of real-world traces.

Strange Lab