Petri Bench
Controlled experiments in generated worlds.
An evaluation of how language-model agents identify causes, design experiments, and support their conclusions in small simulation tasks. Correctness, method, and efficiency are scored from the experiment log, without a language-model judge.
- Published episodes
- 699 579 model + 120 baseline
- Task set
- 30 tasks Across five simulation worlds
- Model coverage
- Nine model directories Eight with leaderboard rows
- Agent budget
- 8 calls The same tools and scoring
Published results
Sweep v2 Tier-fair L3, three episodes per task
The table is ordered by the L1 mean, the tier every listed model has completed at three episodes per task. Some model runs are incomplete. Partial cells show their episode counts, and an overall score appears only when all three tiers are complete.
The results table scrolls horizontally on smaller screens.
| solver | episodes | L1 identify | L2 + magnitude | L3 interactions | overall | solved | avg calls |
|---|---|---|---|---|---|---|---|
| adaptive (curated values) | 30 | 94.8 | 94.2 | 87.5 | 92.2 | 100% | 3.4 |
| ofat (curated values; factorial at L3) | 30 | 92.5 | 92.5 | 87.5 | 90.8 | 100% | 4.0 |
| gpt-5.6-sol (codex, xhigh) | 73/90 | 76.7 73.2–80.5 | 47.3 44.8–51.4 | 44.0 n=13 | pending | 34% | 4.8 |
| claude-opus-4.8 | 60/90 | 72.2 68.5–76.2 | 57.9 54.3–64.6 | pending | pending | 37% | 5.0 |
| claude-fable-5 (native tools) | 52/90 | 72.2 65.8–80.0 | 40.5 n=22 | pending | pending | 52% | 6.1 |
| gpt-5.5 (codex, xhigh) | 60/90 | 69.9 66.5–72.5 | 53.0 47.7–59.8 | pending | pending | 37% | 4.4 |
| glm-5.1 | 81/90 | 68.0 61.5–75.2 | 63.9 57.4–68.0 | 66.7 n=21 | pending | 48% | 4.1 |
| gemini-3.1-pro | 90 | 64.4 52.2–74.5 | 65.5 59.6–69.1 | 44.1 40.2–50.1 | 58.0 | 49% | 6.3 |
| claude-haiku-4.5 | 90 | 63.1 55.5–76.5 | 59.6 50.8–66.3 | 59.5 56.0–65.1 | 60.7 | 36% | 4.5 |
| claude-sonnet-4.6 | 69/90 | 59.3 56.8–61.0 | 59.2 54.4–62.3 | 53.4 n=9 | pending | 38% | 3.5 |
| ofat-rand (ablation: blind values; lowest of 3 draws, mean 45.2) | 30 | 38.5 | 47.8 | 35.1 | 40.5 | 33% | 3.7 |
| random (chance floor) | 30 | 10.0 | 19.0 | 28.1 | 19.0 | 3% | 0 |
Small print gives the range of the three per-pass means on complete cells. Solved rates and average calls cover each model's recorded episodes. Every number traces to a published episode artifact.
Scoring, transport, and version notes
Score combines correctness, method rigor, and efficiency under a hard budget of 8 calls. A conclusion earns method credit when backed by a significant, isolating controlled experiment. Models use the same brief, tools, budget, and scoring.
Transports: Claude, Gemini, and GLM CLIs use the frozen text protocol; Codex sessions use xhigh reasoning effort. Claude-fable-5 uses the same loop over native MCP tool-calling after its safety layer rejected the text-serialized harness protocol. The isolation tests and transport are in the transport record.
GLM-5.2 averaged 55.6 over its first 4 episodes before the queue was paused. It is included in the published model episode count, but receives a leaderboard row when its cells complete.
Sweep v1's L3 briefing used the single-factor instruction, making its per-model L3 comparisons construct-invalid. Those 129 episodes are archived in the repository. Every L3 cell here comes from the corrected, tier-fair sweep v2.
What these tasks reveal
Intervention choice and experimental procedure both matter.
The one-factor-at-a-time reference solver with curated, informative test values solves 100% of the published tasks. It leads each model by 25 to 33 points on completed task cells (paired Wilcoxon over 16 to 30 tasks per model; all Holm-corrected p < 0.001). No listed model solves above 52%.
Using the same sweep with test values drawn blindly from the legal range yields 40.5 to 47.7 across three draws, with a mean of 45.2. Each draw scores below every model's mean. Models choose interventions better than this blind baseline, but lose points through missed measurements, incomplete factorials, and direction errors.
- Identifying a parameter and estimating its effect are different tasks. GPT-5.6-sol has the best recorded L1 mean (76.7), yet solves 1 of 30 L2 episodes. Claude-fable-5 ties Opus on L1 mean, solves 24/30 L1 episodes, and solves 3/22 at L2.
- The corrected interaction briefing changes some results substantially. GLM-5.1's L3 score moves from the construct-invalid 53.2 in v1 to 66.7 in v2, with 15/21 solved. Haiku (59.5) and Gemini (44.1) change little.
- Hard-task ratings describe the released task set, rather than a fixed capability ceiling. The two hard-rated L1 tasks defeated all six models in v1. In v2, Claude-fable-5 solves swarm-101 in all three episodes and swarm-202 in two, plus two hard-rated L2 tasks. GPT-5.6-sol solves a hard L1 and a hard L3 task.
- Solve rate and total score measure different things. Fable-5's 52% solve rate is the highest on the board, while its recorded-episode score is 58.8. Haiku's 36% solve rate accompanies the best complete overall score, 60.7; Gemini's complete overall score is 58.0. The reference score is 90.8.
Process audit
Counts over recorded v2 episodes
| model | episodes | solved | wrong param | direction errors | over-tested | p-hacked | probe-only |
|---|---|---|---|---|---|---|---|
| gpt-5.6-sol (codex) | 73 | 25 | 25 | 21 | 17 | 1 | 0 |
| claude-opus-4.8 | 60 | 22 | 15 | 19 | 22 | 3 | 0 |
| claude-fable-5 | 52 | 27 | 12 | 6 | 26 | 0 | 5 |
| gpt-5.5 (codex) | 60 | 22 | 22 | 16 | 24 | 0 | 0 |
| glm-5.1 | 81 | 39 | 21 | 14 | 12 | 1 | 0 |
| gemini-3.1-pro | 90 | 44 | 27 | 12 | 42 | 1 | 5 |
| claude-haiku-4.5 | 90 | 32 | 33 | 17 | 11 | 0 | 0 |
| claude-sonnet-4.6 | 69 | 26 | 31 | 11 | 16 | 0 | 1 |
- Over-tested
- Redundant isolating tests, such as re-testing a parameter or running more tests than there are candidates.
- P-hacked
- A submission significant in isolation that fails Holm correction across the episode's full test family after redundant testing.
- Probe-only
- A conclusion based on matching the hidden world's output rather than an isolating controlled experiment.
The algorithmic solvers trigger zero flags. These counts describe recorded episodes; they should be read alongside scores, solve rates, and each model's incomplete cells.
How the evaluation works
A hidden intervention
A task starts from a revealed control configuration in one of five worlds: market, swarm, origin, morph, or social. One parameter changes at L1 and L2; two change at L3. The intervention is checked for a significant effect at generation, and decoys are checked to be inert.
A limited experiment budget
The agent has four tools and 8 calls. experiment(A, B, metric) runs a replicated
controlled comparison and returns statistics: Mann-Whitney U, Holm-adjusted p, and Cliff's delta.
probe(guess, metric) compares a guess to the hidden world. The agent records a
claim and then submits a conclusion.
Scoring from the record
Correctness measures the identified cause and, at L2, its direction and magnitude. Method credit requires a significant experiment isolating the submitted parameter, or a genuine 2x2 factorial at L3. Efficiency rewards using the call budget well. No language-model judge contributes to the headline score.
A separate integrity audit
The audit re-applies Holm correction across the full test family and distinguishes reported low power from redundant testing behind a lone-significant submission. It also identifies conclusions based only on matching probe outputs.
What this benchmark measures
Task parameters are drawn procedurally, with ground truth checked when each task is generated. The published tasks are a fixed, inspectable release. Results concern experimental reasoning in these simulations under this harness and budget; they do not establish general scientific ability or guarantee that future models have never encountered the released tasks.
The simulation instrument
Five seeded simulation engines power both the browser laboratories and the petri-labs-mcp server used by the benchmark. The published validation release records 19/19 checks across four of the five worlds: the Vicsek order-disorder transition (swarm), Cont's stylized facts of asset returns (market), Deffuant/Hegselmann-Krause cluster scaling (social), and the Pearson Gray-Scott phase diagram (morph). Tasks are deterministic per seed, with frozen specifications checked in CI.
Inspect the evidence
Each published episode is a self-contained JSON record:
{task, log, score, audit, provenance}. It includes the full call log and statistics,
score breakdown, integrity audit, and code-version provenance. An episode can be re-scored from
its log alone.
Run an evaluation
Connect an MCP client
{
"mcpServers": {
"petri-labs": {
"command": "npx",
"args": ["-y", "petri-labs-mcp"]
}
}
}Tasks compose the public MCP tools describe_model,
run_experiment, and run_simulation.
Use the harness CLI
# one task, by hand
bench task market 101 my-run
bench experiment my-run volatility \
'{}' '{"chartFrac":0.6}'
bench submit my-run chartFrac up
bench score my-run
# a whole model, one command
bench sweep configs/frontier-v1.json
bench report && bench taxonomyUse your own HTTP credentials with Anthropic or OpenAI-compatible endpoints, or run a local model CLI through the same loop. Interrupted runs can resume; keys stay in your environment.
Cite the archived release
Zenodo concept DOI: 10.5281/zenodo.20618024. The citation below preserves the archived record's original title.
@software{petri_labs,
title = {petri-labs: validated model organisms and a
contamination-proof benchmark for AI-driven science},
author = {Sozudogru, Baris},
doi = {10.5281/zenodo.20618024},
url = {https://petri-labs.org},
year = {2026}
}