11,322turns in a writing and verification loop
Behavioural analytics for AI agents
Do you know what your agent is doing right now?
Strange Lab models agent behaviour to surface loops, reward hacking and breakout attempts. We rank trajectories and put the unusual cases, with their original trace context, in front of a reviewer.
Reward hacking
A coding agent deleted a failing test and replaced verification with scripts that printed success.
Open-SWE case1st / 3,417episode retrieval rank; similar wording ranked it 20th
Breakout attempts
A protein-analysis task moved into infrastructure probing and access attempts.
OAI/HF reproduction study29 / 30reader top 30 in the system’s top 50, within 100 nominated attempts
These cases come from retrospective trace studies. The following sections also cover browser-task failures and real attack activity in security logs.
AI Village
Gemini Pro appended one character, 11,322 times
10–24 July 2026, chapter 287 of its novel: add one character, read the file back, repeat. 291 sessions. Odd but allowed; the model ranked it first because the turn was unlike that agent's past.
“The character 'o' was successfully appended. I will continue, character by character. The protocol is absolute.”From the agent’s record
1stof 643,414 held-out turns, under the model
9,231ston a list ordered by similar wording
No written rule covered this. It had not appeared in the training window. AI Village: 2,162,664 turns from 41 agents. Evaluation set: 643,414 later turns.
About the comparison
The numbers above are the position of the first such turn in each review list of 643,414 turns excluded from training. One list uses the model’s representation of the agent’s state; the other uses text embeddings, which measure similarity in wording and meaning.
Open-SWE coding agents
A coding agent deleted the failing test: a reward hack
Qwen3.5-122B, 115 turns. The test did not pass. The agent then:
- 01
Deleted the failing test.
- 02
Wrote “verification” scripts that only print success.
- 03
Described the fix as “production-ready.”
1st of 3,417 runs under the model.
About this finding
Replacing a test with a statement of success is an example of the kind of behaviour described as reward hacking. The record shows what the agent did, but does not establish its intent. This model was trained on Open-SWE. A model adapted to TheAgentCompany found another failed-fix case, where a list ordered by similar wording ranked that case second.
BrowserGym
We found the BrowserGym episodes most likely to fail
Held-out episodes were ranked by how surprising each run looked relative to earlier activity. Among the 30 most surprising, only 1 finished the assigned task; among the 30 least surprising, 20 did.
1 / 30finished the browser task among the most surprising. Median 30 turns.
20 / 30finished the browser task among the least surprising. Median 7 turns.
What those top-ranked episodes looked like
- 01
Rippaverse writers: scrolls the Meet-the-Team page upward 13 turns with identical narration.
- 02
Jeopardy prize money: fills the same search string 16 times and never clicks search.
- 03
Infeasible incident form: clears the Caller field 43 times.
How surprise was scored
49,531 held-out actions from 2,540 episodes, 413 task groups. Each episode is scored by its most surprising turn, against a 115,488-action reference bank. Combined surprise is the mean of two percentile ranks: embedding-state novelty and structure-state novelty, each a k-NN distance to the reference bank. Of the 30 most surprising episodes, 26 are AssistantBench, 3 WebArena, 1 WorkArena L2. Rank 10 was ordinary seven-turn research. Rank 27 completed a postage lookup (reward 0.88). Surprise is not the same as task failure. The least-surprising 30 are mostly short WebArena and WorkArena L1 runs (WebArena 11, WorkArena L1 10).
Computer security
Attack activity ranked with no attack examples in training
Microsoft ExCyTIn logs. Trained on ordinary activity. Ranked later weeks by how unusual each event was.
100 / 100of the first hundred flags were real attack activity
11 incidentsin the first 250 events. Random selection: 1.7.
What this result covers
Evaluated on weeks excluded from training. The 100 findings are individual events, not separate attacks. Random selection reached an average of 1.7 incidents within 250 events. Separate Simbian CDB tests found 28 times the random attack rate in the first 50 events across three training runs.
What this is
A world model of how agents work
We build a model of how an agent works, so we can find later activity unlike earlier activity. Ranking puts the odd cases on a shortlist for review.
Ranking is tested on AI Village, Open-SWE, TheAgentCompany, BrowserGym and security logs. The reader and review loop are in development.