You can't govern what you can't see: observability and audit trails for AI agents

Every team eventually asks the same question about an agent: "What did it actually do, and why?" If the honest answer is "we're not sure," you don't have an automation problem—you have an observability gap. And an agent you can't observe is a liability with a login.

We've argued that agents should be accountable participants in your delivery loop and that you can't ship an agent you can't test. Observability is the runtime companion to both: testing tells you what an agent should do before release; observability tells you what it did in production. This post is about building that second half—traces, decision logs, and tamper-evident audit trails—so you can debug failures, satisfy auditors, and reconstruct an incident without guesswork.

Why agent observability is different from app observability

You already instrument services with logs, metrics, and traces. Agents need all of that plus a few things classic APM never had to model:

Non-determinism. The same input can produce a different tool call tomorrow. You cannot rely on "reproduce it locally"—you have to capture what happened this time, because it may never happen the same way again.
Emergent action. An agent doesn't execute a fixed code path; it chooses one. The interesting artifact is the decision: which tool, with which arguments, based on which retrieved context.
Untrusted inputs as control flow. When an agent reads an email, a web page, or a ticket, that content can steer its behavior—the exact mechanism behind prompt injection. Your telemetry has to make it possible to answer "what did the model read right before it did something strange?"

Treat observability as a security control, not just a debugging convenience. The same trace that helps an engineer fix a broken workflow is the evidence a responder needs when an agent is manipulated into exfiltrating data.

The three layers worth capturing

Not every log line is worth keeping, and over-collection creates its own liability (see the data-boundary note below). Aim for three deliberate layers.

1. The trace: one run, end to end

A single agent run should map to one trace, with a span per meaningful step:

The triggering event (user message, scheduled job, or webhook) and the identity that owns the run.
Each model call: the prompt template and version, token counts, latency, and the model/version used—so you can attribute a regression to a specific change.
Each retrieval: what was queried and which documents came back (by ID and source, not necessarily full contents).
Each tool call: the tool, the arguments, whether an approval gate fired, and the result or error.

The goal is that anyone can replay the run as a narrative: event → reasoning → retrieval → action → outcome.

2. The decision log: why, not just what

Traces capture what happened. For agents you also want a lightweight record of why—the model's stated rationale or plan at each branch point, and which policy allowed or blocked an action. This is what turns "the agent sent the wrong email" into "the agent chose send-email because it interpreted the ticket as a customer request, and the policy engine permitted internal recipients only." One is a shrug; the other is a fixable finding.

3. The audit trail: tamper-evident and boring on purpose

For anything with business or regulatory weight—money moved, records changed, communications sent—you need an audit trail distinct from your debug logs:

Append-only and tamper-evident. Write to storage the agent's own credentials cannot rewrite. Consider hash-chaining entries so deletion or edits are detectable.
Who approved what, when. Tie every irreversible action back to the human gate that authorized it (or flag that none did).
Minimal but sufficient. Store enough to reconstruct accountability—identities, action types, timestamps, decision references—without hoarding sensitive payloads you'll regret keeping.

Audit trails should be the least exciting thing you build and the first thing you're grateful for during an incident.

Data boundaries: observability is a place data leaks

Instrumentation quietly copies data into logs, traces, and dashboards—which means your telemetry inherits the classification of whatever the agent touched. Apply the same least-privilege instincts to your observability stack:

Redact at capture. Strip secrets and PII before they hit a trace, not after. Prefer document IDs and hashes over raw content where you can.
Scope access to telemetry the way you scope the agent itself. A debug dashboard that shows every prompt and retrieval is a high-value target.
Set retention deliberately. Long enough to investigate incidents and satisfy auditors; short enough that you're not sitting on a breach waiting to happen.

If it would be reckless to email a payload externally, it's reckless to stream it into an ungoverned logging pipeline. Observability that leaks is a downgrade, not a control.

Turning telemetry into action

Capture is the floor; the payoff is what you do with it:

Alerts on the right signals—not CPU, but behavioral anomalies: a spike in denied actions, an agent suddenly using a tool it never used before, retrieval from an unexpected data source, or approval gates being hit far more often than baseline.
Fast forensics. When something breaks, the trace + decision log should let a responder answer "what did it read, decide, and do" in minutes. That is the raw material an incident response playbook runs on.
A feedback loop into evals. Every real-world failure you catch in production should become a regression test so the same mistake can't ship twice.

Why this becomes the norm

Production software became trustworthy partly because we could see it: distributed tracing and structured logs went from luxury to table stakes. Agents are on the same trajectory. "Agent observability" sounds specialized today, the way "APM" once did—and in a few years, shipping an agent without a trace and an audit trail will look as reckless as running a service with no logs at all.

That's the through-line of agent-driven development: intent expressed clearly, actions produced auditably, humans accountable on the risk boundary, and systems that can always answer what happened. The teams that win won't be the ones with the flashiest demos—they'll be the ones who can explain, calmly and with evidence, exactly what their agents did and why.

Where we can help

If you're standing up agents and want observability, audit trails, and governance designed in from the start—rather than bolted on after the first scary incident—tell us about your setup or read the consulting offer.