Reversibility Is The First Agent Requirement

Reversibility Before Autonomy: Implementing Five Eyes AI Guidelines

We gave an agent write access to the WaterDoctor telemetry dashboard in late 2024. The task was straightforward: detect drift in sensor calibration baselines and apply corrections. The agent hallucinated a trend that didn’t exist. It wiped 400 rows of valid calibration data in under two seconds, per the staging incident log — we were watching in real time and still couldn’t stop it. We restored from a daily backup and still lost four hours of live telemetry readings from six field sensors.

That is not a prompt-engineering problem. It is an architecture problem.

Read-only agents are toys. Write-capable agents without a tested undo path are liabilities, not automation.

From Policy to Engineering Constraint

The November 2023 “Guidelines for Secure AI System Development” — published by CISA, the UK NCSC, and cybersecurity agency partners across the Five Eyes nations — is not an academic paper. It is a government-backed security baseline that translates AI risk into concrete engineering requirements for teams building and deploying AI systems. The “Secure deployment and operation” section is where that matters most for anyone wiring an LLM to a production system.

Pull it apart and three demands surface: contain the blast radius of an AI system when it misbehaves; maintain resilience when behavior goes sideways; and keep incident response capability that doesn’t hinge on a human catching a bad query before it commits. Not aspirational goals. Constraints.

An agent operating inside a production database is not a deterministic microservice. You cannot unit-test your way to safety. The agent will hallucinate. It will misread context. It will apply a correction to the wrong table. The question isn’t whether this happens — it will. The question is how fast you can reverse it.

Scoped Credentials, Hard TTL, No Exceptions

The WaterDoctor incident happened partly because we gave the agent the same database role used by the backend application server — a role with broad write access. We corrected that immediately.

The rule now: agents never hold permanent, unrestricted database roles. Every agent credential is a scoped role, created at task invocation, with a hard time-to-live. Thirty minutes for short-lived correction tasks. Two hours maximum for longer batch operations. When the TTL expires, the session closes, the credential is revoked, and the agent cannot reconnect under the same identity.

This does create a real edge case: genuinely long-running tasks risk hitting the TTL mid-operation. The answer isn’t to raise the ceiling — it’s to design agents with explicit checkpoints and resumable state.

At the database level, the scoped role is surgical. For the WaterDoctor calibration agent, the replacement role is granted UPDATE on two specific tables. DELETE and TRUNCATE are not granted to the agent role, and the role is verified in isolation — without the inherited memberships that could reintroduce those privileges. The baseline archive table is entirely off-limits. If the agent calls a function outside its scope, the database returns a permission error, the wrapper logs it, and the task fails cleanly. A clean failure beats a silent destructive success every time.

This is what the Five Eyes guidelines call least privilege. The language in the document stays general. The implementation doesn’t. In practice, it’s a Postgres ROLE with five explicit REVOKE statements, tested against the production schema before any agent touched live data.

Action Receipts: Read-Log-Commit

Scoped credentials limit what an agent can do. Action receipts make what it did reversible.

We implemented this pattern first for an SME CRM integration we shipped in early 2025. The agent’s job was to update contact records based on enriched data from a third-party API — contact records that have immediate client-facing consequences when wrong. A full database restore was too blunt; we needed a targeted revert path.

The mechanics wrap every agent write. Before the mutation commits:

1. The wrapper issues a SELECT for the current state of the affected rows and stores the result.
2. It generates the exact rollback SQL required to revert the pending change.
3. It writes that rollback SQL, the agent’s action identifier, and a UTC timestamp to an audit log table.
4. It commits the agent’s update.

If the update is wrong, we call the stored rollback SQL. State reverts in milliseconds. The audit log retains both the original action and the revert event.

There is an operational cost worth naming honestly. In our CRM staging benchmark — same contact-update workload, same staging database instance — the wrapped write path ran approximately 40% slower than direct application writes [Sn]. We accepted that trade-off: data integrity on contact records is a constraint, not a variable. Speed only looks like an advantage until someone files a correction request. It’s also worth noting that this pattern doesn’t scale cleanly to bulk-write agents pushing tens of thousands of rows per run; at that volume, WAL-based change capture is worth considering instead.

What the action receipt actually changes is the incident conversation. When a stakeholder asks what the agent changed between 14:00 and 14:30, you open the audit log — exact rows, exact values before and after, rollback SQL ready to execute. That’s a different conversation entirely from restoring last night’s backup and reconciling the delta by hand.

Monthly Containment Drills

An architecture that lives only on a whiteboard is a liability dressed as a safety feature.

We run a monthly drill on the CRM deployment. Deliberately uncomfortable. We force the agent to execute a destructive update sequence in a staging environment that mirrors production schema and data volume. We let it commit. Then we time the full reversal: isolate the transaction, identify the affected action receipts, execute the rollback SQL in sequence, verify row-level state against the pre-drill snapshot, confirm the audit log closed cleanly.

Target: full reversal confirmed in under 90 seconds. Our drill records show four passes out of the last six monthly runs [Sn]. The two misses traced to the same issue — the staging environment’s audit log table had drifted from the production schema after a migration. We fixed the sync, ran the drill again the following week, and it passed.

The drill isn’t a stress test on the agent. It behaves identically every time. It’s a stress test on the human team. Can the on-call engineer find the rollback procedure under pressure? Can they execute it without reading documentation? Can they confirm state reversal before escalating to a full restore? If the answer to any of those is no, the safety architecture has a gap that no amount of scoped credentials will close.

Architecture diagrams do not recover your data. Drilled engineers do.

The Engineering Mandate

Write-capable agents in production without a tested rollback path is engineering negligence.

The Five Eyes guidelines didn’t invent blast radius containment or least privilege. Both have been standard security engineering practice for decades. What they do is apply those principles explicitly to AI systems — which until recently most teams were treating as slightly unpredictable microservices rather than systems with genuinely non-deterministic failure modes.

The controls are not exotic. Scoped credentials with TTL expiry, action receipts with stored rollback SQL, monthly recovery drills — unglamorous engineering, all of it. It adds overhead. It slows agent writes. It requires maintaining test environments that mirror production schema. None of this is interesting work. It is the work.

Stop treating LLMs like senior backend developers. They are junior interns with raw database access and no instinct for the consequences of a bad query. Build the undo button first. Test it monthly. Give it to the intern before you hand them the keys.