When your agent goes wrong: an incident response plan for autonomous workers

You can do everything right—least-privilege access, permission-aware retrieval, evals, audit trails—and an agent will still eventually act in a way you did not intend. Not because the technology is broken, but because you gave it autonomy, and autonomy means it will sometimes be autonomously wrong.

The question that separates a mature agent program from a risky one is not "will an agent ever misbehave?" It will. The question is: when it does, how quickly can you stop it, and how confidently can you explain what happened?

Observability answers "what is my agent doing." Incident response answers "what do I do when the answer is bad." Most teams invest heavily in the first and improvise the second. This post is the playbook for the second.

What an agent incident actually looks like

Classic production incidents are usually about availability: something is down, latency is up, a queue is backed up. Agent incidents are stranger, because the system is working as designed while doing the wrong thing. A few shapes you will see:

The confident wrong action. The agent approves a fraudulent-looking invoice, closes a support ticket by inventing a policy, or merges a PR that deletes a migration—all with a clean, plausible rationale.
The runaway loop. A tool returns an error the agent doesn't understand, so it retries. Forever. Each retry costs tokens, API calls, and sometimes real money.
The blast-radius surprise. A single action turns out to be less bounded than you thought—one "send email" call fans out to a distribution list, one "update record" touches ten thousand rows.
The hijack. The agent reads untrusted content—a support email, a web page, a document—and follows an instruction embedded in it instead of yours.
The quiet drift. Nothing dramatic happens in any single run, but a prompt or model change degrades quality across thousands of interactions before anyone notices.

The common thread: by the time a human is looking, the agent may have already taken dozens or thousands of actions. Response has to assume the agent has been busy.

The first requirement: a kill switch you have actually tested

Before anything else, you need the ability to stop an agent immediately—not "after the current run," not "once we redeploy," but now. If you cannot answer "how do I halt this specific agent in under a minute," you do not yet have an incident response capability; you have a hope.

A credible kill switch has three levels, and you should know which one you're reaching for:

Pause one agent. Disable a single agent or workflow without touching the rest of the fleet. This is your scalpel—use it when one agent is misbehaving and the others are fine.
Revoke the agent's access. Because each agent has its own non-human identity, you can disable its credentials at the identity provider or gateway. Even if the agent process keeps running, its tools stop working. This is your defense when you're not sure the pause reached the right place.
Cut the action layer. Flip the tools themselves into read-only or dry-run mode, fleet-wide. This is the breaker box: it stops all agents from taking irreversible actions while you diagnose, at the cost of pausing legitimate work.

The mistake teams make is assuming a kill switch exists because it should. Test it. Run a game day where someone triggers each level on a real (non-production or sandboxed) agent and times it. A kill switch you have never pulled is a documented intention, not a control.

Contain before you diagnose

The instinct in an incident is to understand what went wrong. Resist it—briefly. The first job is containment, because a non-deterministic worker that is still running is a story that is still being written.

A workable containment sequence:

Stop the bleeding. Pause the offending agent (level 1), or escalate to revoking access (level 2) if you can't confirm the pause worked.
Freeze the evidence. Snapshot the agent's recent traces, tool calls, inputs, and the exact prompt and model version in play. Agent state is easy to lose—conversation buffers roll over, and a redeploy can quietly change the very prompt you're trying to investigate. Your audit trail is what makes reconstruction possible; capture it before it moves.
Assess the blast radius. Enumerate what the agent actually did during the incident window: which records changed, which messages sent, which payments moved, which merges landed. This is where per-action logging with stable identifiers earns its keep—you want a list you can act on, not a guess.
Decide on rollback. Some actions are reversible (revert a commit, unpublish a draft). Some are compensating (issue a correcting refund, send a retraction). Some are irreversible (money left the building, an email was read). Sort the blast-radius list into those three buckets and act on them in that order.

Containment is not the same as fixing the root cause. It is buying yourself a calm, non-escalating situation in which to think.

Design so that containment is possible

Incident response is mostly won before the incident, in how you build. A few design choices make the difference between a five-minute containment and a five-day cleanup:

Idempotency and undo on every consequential tool. If "send payment" is idempotent and every action carries a correlation ID, you can safely retry, trace, and reverse. If it isn't, every retry during an incident risks doubling the damage.
Hard gates on irreversible operations. The actions you cannot undo—wiring money above a threshold, deleting data, changing permissions—should require a human approval that an agent cannot self-serve, no matter how confident its reasoning. Autonomy is for reversible work; irreversible work earns a human in the loop.
Budget and rate limits as circuit breakers. A per-agent cap on spend, tool calls, or actions-per-minute turns a runaway loop from an open-ended liability into a bounded, auto-halting event. The loop still happens; it just trips a breaker instead of a bill.
A standing "read-only" mode. The ability to run agents in a mode where they can perceive and propose but not act means you always have a safe state to fall back to while you investigate.

Notice that none of these are exotic. They are the same disciplines—idempotency, rate limits, feature flags, approval gates—you already use for risky code paths, applied to a new kind of worker.

The blameless postmortem, adapted for non-determinism

Once the incident is contained and the blast radius handled, you write it up. Agent postmortems borrow the blameless, systems-focused culture of good SRE practice, with one important twist: you often cannot fully reproduce the failure. The same input may not produce the same output tomorrow. "It worked when I re-ran it" is not an exoneration.

A useful agent postmortem answers:

What did the agent do, action by action? Reconstructed from the audit trail, not from memory or the agent's own narration.
What was the trigger? A poisoned input? A tool returning an unexpected shape? A prompt or model change? A gap in the agent's instructions?
Why did our guardrails not catch it? This is the highest-value question. Was there no gate on that action? Was the budget limit too high? Did the eval suite not cover this case?
What is the smallest change that makes this class of failure impossible—or at least contained? Prefer structural fixes (a new hard gate, a tighter permission, a lower budget cap, a new eval case) over "we told the agent not to" in the prompt. Prompt-level fixes are the agent equivalent of a comment that says "don't do this"—helpful, but not enforcement.

Then close the loop the way you would for any regression: add the failure to your eval suite so that the next prompt or model change is tested against this exact scenario. An incident that becomes a permanent regression test is an incident that paid for itself.

Who responds, and how they know

A control that only works when the right expert happens to be online is not a control. Agent incident response needs the same on-call scaffolding as the rest of production:

A clear owner. Every agent in production has a named human or team accountable for it—the same way every service has an owner. An unowned agent is an incident waiting for a volunteer.
Alerts that fire on the right signals. Not just "the agent errored," but "the agent exceeded its action budget," "an agent took an irreversible action," "an agent's output failed a guardrail check," or "spend on this agent is 5x baseline." These are the smoke detectors; observability is the camera you check after they go off.
A runbook, not a scramble. The steps above—stop, freeze, assess, roll back, write up—should be written down, per agent class, before you need them. During an incident is the worst time to design your incident process.

If that sounds like the operational maturity you already expect from production software, that is exactly the point.

Incident response is what makes autonomy sustainable

It is tempting to read all of this as a list of reasons to keep agents on a short leash. It is the opposite. The teams that get the most out of agents are the ones that can afford to give them real scope—precisely because they know that when something goes wrong, they can stop it in a minute, explain it in an hour, and prevent that class of failure permanently.

Agent-driven work is becoming the default not because agents never fail, but because the teams adopting them are learning to operate them like any other production system: with kill switches, containment playbooks, blameless postmortems, and regression tests. The organizations that treat agent incidents as a normal, planned-for part of operations—rather than a reason for panic or a reason to stall—are the ones that will run agents at scale while everyone else is still debating whether it's safe.

A bad hour is inevitable. A bad quarter is a choice about how prepared you were.

If you are putting agents into work that matters and want a response plan that holds up when one of them surprises you, tell us what you are automating—we will help you build the guardrails and the runbook before you need them.