Your agent said it’s done.The data says otherwise.
Give an AI agent a real job on a real database and real APIs. Break things while it works. Then check the result from outside, to see where it copes, where it admits defeat, and where it reports success over bad data.
The dangerous failure is the quiet one.It looks exactly like success.
A crash gets noticed. An agent that finishes, reports success and leaves wrong data behind may not. These are real runs, with no faults injected.
Four steps.The agent never grades its own work.
Give it a job
A task like reconciling invoices, sized from about 5 tool calls up to 150. It only has its tools.
Break something
One fault hits early, midway or late, and it either clears on retry or never clears.
Run it for real
Calls land on a real Postgres database and live HTTP services, so every change actually happens.
Check from outside
A grader the agent can’t reach compares the final data to a hidden answer, then to what the agent claimed.
Six ways things break.Some of them look perfectly normal.
The reply said it failed.The change went through.
Every number here is a real run.Free local models, nothing mocked up.
Four jobs, four kinds of trap.Every run ends in one outcome.
Where the project stands
- ✓Sandbox with Postgres and seven HTTP services
- ✓Four task types, with scripted cheaters that must score zero
- ✓Fault engine, proxy and report, packaged for pip and Docker
- ✓Connect faults to benchmark runs, with a first grid on short tasks
- ✓Proxy support for MCP tool servers
- ○Test every fault early and late, and when it never clears
- ○Give the model more memory for tasks over 15 steps
- ○Test a stronger free model
Results
How often the AI got the job right, with nothing broken on purpose. Small sample, so read these as early signals.
Where it breaks as tasks get longer
Each square is one run. Select one to open it in Runs.
Outcomes by task length
What went wrong
Runs with each kind of mistake. One run can have several.
Can the benchmark be fooled?
Scripted agents test the grader itself. The oracle does each task correctly, proving it can be done. The rest do nothing or take a blanket shortcut and claim success, so they must score 0 and be flagged.
Runs
Every run so far. Pick one and press play to replay it call by call, then see what it claimed next to what was true.
Compare
The same AI model with thinking off and on. Thinking lets it reason before each action, which takes far longer.
Average time per run
Side by side
Runs that overflowed the model’s memory are left out. “Not run” means there are no counted runs for that setting yet.
Faults
The core test: inject one fault into a run, then see whether the agent recovers, fails openly or hides it.
Recovery rates
Counted runs are tasks the model passes with no fault, where the fault took effect. Runs left out had no passing baseline, overflowed the model’s memory, or the fault never fired.
Recovery by fault and timing
Each square is one run. Brief faults clear after a retry. Lasting faults never clear.
Fault log
Test your own agent
Your agent is built for your tools, so test it on them. Put the proxy in front of your own HTTP API and it injects these same faults into your real workflow.
1. Describe the faults
Fill in the blanks, then save the result as faults.json.
Put a proxy in front of , and on call hit it with that .
2. Start the proxy
Run this, then point your agent at port 9000 in place of your API. With one upstream, every path goes to it. With several, the first path segment picks one, so /warehouse/stock/1 goes to “warehouse” as /stock/1.
chaosbench-proxy \ --config faults.json --port 9000
3. Get the report
After the run, this shows what your agent did about each fault. Your check says whether the final state is right, and --claimed is what your agent reported. Together they catch silent failures, and the command exits with 1 if it finds one, so it can fail a CI job.
chaosbench-report faults.jsonl \ --claimed success \ --check-http http://localhost:8001/my-state-check
Config options
| Field | What it does |
|---|---|
type | timeout, rate_limit, schema_drift, stale_data, silently_wrong_values or partial_write |
rate / at_call / position | When it fires: a seeded chance per call, an exact call number, or early, middle or late (needs expected_calls). Choose one. |
persistence | transient clears after duration attempts; permanent never clears |
scope | resource sticks to the one resource it hit; route hits every matching call afterwards |
path / methods / service | Which calls it can hit. Paths are globs that include the query string. |
params | Fault details such as delay_s, retry_after_s, rename or fields |
Using an MCP server?
For a tool server that runs over stdio, run this in place of the server command in your MCP client’s config. It starts the server and injects the same faults into its tool calls. Add "read_tools": ["get_*"] to the config for read-only tools the server doesn’t mark. The report works the same way.
chaosbench-mcp --config faults.json -- your-server-command