On-Prem Agents Need Data-Gravity Runbooks

The Appliance Illusion

A server in your rack makes your compute resident. It does not make your data resident. Procurement teams that treat the two as the same will find the difference expensive.

Vendors are trying to move coding agents closer to enterprise data. That’s the right direction, because many organisations cannot send source code to a shared cloud. But the purchase order is the easy part. The runbook is where it gets hard, and many enterprises don’t have one for this class of system.

Here’s the gap. Enterprise architects know how to track source code. They can tell you where the repository lives, who can clone it and which backups hold it. Ask where an autonomous agent puts its vector cache, where it writes its reasoning trace, or which endpoint receives its diagnostics, and many can’t answer yet. An agent isn’t a static binary. It keeps generating new data, and every stream of data ends up somewhere.

Then there’s the hybrid tether. “Local compute” often arrives with remote dependencies: analytics, a reasoning fallback to a larger hosted model, model telemetry. Some of these are disclosed. Others sit in a default configuration nobody reads. I’m not claiming any specific vendor does this. My point is that you have to ask in writing, before the box is racked.

Hard Subnet Boundaries and Legacy Telemetry

Subnet isolation isn’t new. AI is just another workload, if a telemetry-hungry one, and we’ve solved similar problems before.

In 2019 we built a dashboard for a local government deployment under strict data residency rules. Remote log aggregation was prohibited, which ruled out cloud-based observability entirely. That meant no hosted logging, no SaaS metrics and no vendor-side debugging. So we built local telemetry pipelines. Logs were collected, stored, rotated and queried inside the perimeter. It was more work, and it was the design that passed review.

WaterDoctor is the second example. Its edge deployments handle sensor telemetry that couldn’t leave the local subnet. We didn’t stream raw readings to a cloud for analysis. We ran local rule engines at the edge, so decisions were made where the data was produced.

The lesson carries over to LLMs with few changes. A system that can’t phone home has to fit its whole operational footprint inside the local perimeter: logs, metrics, debug traces and the update mechanism. Many on-prem AI agents will struggle with this. They aren’t malicious. They were designed cloud-first and later ported to a rack, and cloud-first software tends to assume someone else is holding the logs. Test each product against that assumption instead of taking it on trust.

Mapping the New Cognitive Artifacts

Traditional data residency focused on database rows and user uploads. That perimeter was drawn for systems that store what users hand them. A generative agent also stores what it derives from what users hand it, so the perimeter has to be redrawn.

During normal execution, an agent creates at least four classes of cognitive artifact. The enterprise has to account for each one.

• Prompts and system instructions. This is the operational logic. It encodes how your organisation wants the agent to behave, including internal conventions, naming rules and security policies. A tuned prompt library is intellectual property.
• Intermediate reasoning logs and scratchpads. This is the execution trace. It records what the agent looked at, what it considered and why it acted. It can contain verbatim fragments of source code, configuration and any credentials the agent stumbled across.
• Embeddings and vector indices. This is the retrieval cache. It’s a derived copy of your corpus, transformed but not anonymised. Embeddings are not anonymisation: Morris et al. showed that text can be partially recovered from dense embeddings, so treat vector indices as sensitive data.
• Generated outputs and code snippets. This is the final artifact: a patch, a script or a summary. It’s the most visible class and usually the best governed.

In our experience, residency reviews tend to cover only that last class. The first three are where the risk piles up.

The practical test is simple. Say the hardware runs the model locally but ships reasoning logs to a vendor cloud for “product improvement”. The residency boundary is broken. It doesn’t matter that the model weights never left your rack, because your code, as seen through the agent’s trace, did.

The Vendor Lock-In of RAG Caches

Residency has a second face, which is ownership. Knowing where the artifacts live isn’t enough. You have to be able to take them with you.

A recent SME client tried to move from a cloud-based AI agent to a local setup. They had put compute at the top of the project plan. Then they found the real problem. Their tuned prompt library and RAG index effectively belonged to the vendor. There was no supported export path. The prompts existed only inside the vendor’s workspace, and the index existed only in the vendor’s format.

Moving the compute means little if you lose the memory.

Generating embeddings costs time and compute. Tuning prompts costs engineering hours. It also costs institutional learning, since every correction your team made to the agent’s behaviour is baked into that library, and that’s the more expensive part. Suppose an appliance obfuscates its vector database, or uses a proprietary embedding format that only its own runtime can read. Then you’re tied to that hardware. The lock-in sits in the cache, not the licence.

Appliances have lifecycles. Hardware gets refreshed, vendors change strategy and workspaces get reorganised. Without a contractually mandated export, an enterprise at the end of that lifecycle has to rebuild its cognitive state from raw data. It re-embeds the corpus, re-tunes the prompts and relearns the corrections. That cost rarely shows up in a budget, because nobody wrote it into the runbook.

Architecting the Exit Plan

Procurement needs an architecture checklist before any hybrid or on-prem AI appliance touches the corporate network. A security questionnaire written for SaaS won’t do. The checklist has to be written for a system that creates memory.

1. Mandate explicit storage locations. The vendor must document, per artifact class, where code, logs, prompts, caches and embeddings are written. That means paths, volumes, retention periods and encryption keys. “Stored securely” is not a location.

2. Block outbound telemetry by default. Put the appliance in a subnet with no egress and open holes only for destinations you’ve reviewed. If the product breaks, the vendor must provide a local diagnostic interface. A system that works only when it can phone home is a cloud service in a metal case. Our 2019 dashboard and the WaterDoctor edge work point the same way: a system that must run isolated must also be operable isolated. The trade-off is slower vendor support and manual patching, so budget for both.

3. Separate local execution from remote reasoning. Ask which operations run on the box and which call out. Building and testing code locally while sending reasoning requests to a hosted model can be a legitimate architecture. It just isn’t an on-prem one, and the contract shouldn’t describe it as such.

4. Require an export API. Vector indices, prompt libraries and agent memories must be exportable in documented, standard formats, with the embedding model and version recorded alongside. Test the export during the proof of concept, not at termination. Even a clean export may not be portable if the target system uses a different embedding model, so confirm you can re-embed from raw data.

5. Write the exit clause first. Define what happens to every artifact class when the contract ends, including verified deletion on the vendor side.

None of this is exotic. It’s the discipline we already apply to databases and backups, extended to a new category of data.

Data residency now includes cognitive artifacts. The enterprises that run on-prem agents well will plan for the day the agent leaves or the appliance is unplugged. Own the cache. Own the embeddings. Everything else is a purchase order.