Unexpected Architecture for AI Agent-Ready Systems

When a solutions architect hears “design a system that includes AI agents,” the conversation usually goes straight to infrastructure. Where will the models run? How will we manage context windows? What’s the memory strategy? Which vector database? How do we expose tools to the agent? These are real questions, and they have real answers, and there’s no shortage of articles, vendor pitches, and reference architectures ready to help you think through them.

This is not one of those articles.

The infrastructure questions matter, but they are not the architectural decisions that determine whether the resulting system actually works. The decisions that matter most are upstream of all of that, and they tend to get skipped because they don’t look like AI problems. They look like ordinary systems design problems, applied to a kind of user the field hasn’t quite figured out how to think about yet. That’s what this article is about: the unexpected architectural concerns that determine whether a system is genuinely ready for AI agents, as opposed to just technically capable of running them.

Agents are users (and that’s where it gets interesting)

Start here, because everything else follows from it. An AI agent acting inside your system is, structurally, a user. It has an identity (or it should). It performs actions. Those actions read and modify state. They have consequences. They need to be authorized, logged, and constrained. None of this is new. We have been governing users in systems for fifty years. We know how to do this.

The interesting part is that most organizations deploying agents today are not treating them as users. The agent is running under a service account, or a developer’s credentials, or some shared API key, and it inherits whatever permissions that underlying connection happens to have. Nobody designed a permission model for the agent specifically. Nobody asked what the agent should and should not be able to do. The agent ended up with a set of capabilities that were originally provisioned for a backend service or a human developer, and those capabilities are almost always far broader than the agent actually needs.

This is not a new failure mode. It’s the same failure mode that produces every “junior employee accidentally drops the production database” incident you have ever heard about. Somebody had write access to something they didn’t need write access to, and the system trusted them because the system had no way to know any better. When it happens with a human, we call it a permissions problem. When it happens with an agent, we call it an AI safety problem, but it’s the same problem, with the same root cause, and the same fix: design the permission model deliberately, scope it tightly to the actual job, and don’t let any principal (human or agent) accumulate access by default.

So the first unexpected architectural decision is the one that should be the most obvious: agents need their own identities, with their own permission scopes, designed for the specific work they are doing, and nothing more. If you can’t articulate what an agent in your system is allowed to do (in terms of specific resources and specific actions), you don’t have a permission model. You have a hope.

Agents are users, but faster and more gullible

Here’s where the analogy starts to need some adjustment. An agent is a user, but it’s a user with some unusual properties that change how the controls have to work.

A human user can do maybe one stupid thing per minute, on a bad day. An agent can do a thousand stupid things per minute, each one cryptographically signed by its identity and looking exactly like legitimate work. Rate limits, which are a polite suggestion for human users, become a hard architectural requirement for agents. If an agent has the ability to issue refunds and you haven’t capped how many refunds it can issue per hour, you have a system that can drain a bank account in the time it takes someone to notice. The blast radius of any over-permission scales with how fast the principal can act, and agents act fast.

A human user, when something feels off, will usually pause. They’ll ask a colleague, or check a policy, or just hesitate. An agent has no such reflex. Each invocation is fresh; there is no continuity of judgment across calls. Controls that depend on the user catching their own mistakes (warning dialogs, confirmation prompts, “are you sure?” patterns) don’t work, because there’s nobody on the other end to be made uncertain by them. The control has to be in the system itself, not in the user’s reflection.

A human user can be tricked, but it usually takes effort. Phishing campaigns have to be designed; social engineering takes time; the attacker has to find a vulnerable employee and work on them. An agent can be tricked by a single sentence buried in a document it happens to read. Prompt injection is, structurally, a privilege escalation attack carried out through the user, except the user is infinitely gullible and processes thousands of inputs an hour. You would never give a human employee production access if that employee would do whatever any stranger wrote on a sticky note. We give agents production access and act surprised when something goes wrong.

The architectural consequence is that the input the agent processes has to be treated with the same suspicion as the agent’s output. Most systems are built on the assumption that internal data is trustworthy because it came from inside the system. For agents, that assumption is broken. A document uploaded by a customer, an email forwarded by a partner, a row in a database populated by an upstream integration: any of these can contain instructions that the agent will treat as authoritative if you let it. The validation perimeter for an agent-ready system has to extend inward, not just outward.

Agents don’t fit on the org chart

This is the one that actually breaks the user analogy, and it’s worth sitting with because the implications run deep.

Human users have roles. Roles correspond to functions. Functions are bounded. A finance clerk does finance things; an HR analyst does HR things; a customer service rep does customer service things. We deliberately design jobs this way, because concentrating cross-functional access in a single person is a known control failure. The whole concept of segregation of duties (the person who initiates a payment isn’t the person who approves it; the person who creates a vendor isn’t the person who pays the vendor) exists specifically to prevent any one principal from doing too much of a sensitive process on their own. These aren’t bureaucratic frictions. They’re load-bearing controls, and they prevent fraud, error, and runaway blast radius.

Agents shred this assumption. A single agent, in a single task, might read a customer email, query the CRM, check inventory, draft a refund authorization, update a ticket, and send a follow-up message. That’s five departments worth of work, performed by one principal, in one reasoning chain, with no separation between the step that proposes an action and the step that approves it. There is no human role that corresponds to this scope of work, because no human job is structured this way, because we already know that’s a bad idea.

And then we deploy an agent that does exactly that, under a single identity, and we call it productivity.

The architectural fix here is the one that probably matters most, and it’s also the one most teams resist because it feels like it’s making things harder. Instead of building one agent that does the whole task, you build several agents, each with a narrow scope of work, and you connect them with a deterministic workflow that controls the handoffs. The agent that drafts the refund is a different principal from the agent that approves the refund. The agent that reads the customer email is a different principal from the agent that updates the customer record. Each one has its own identity, its own permissions, its own audit trail, and the workflow between them is plain code (not LLM reasoning) that enforces the order of operations and the conditions for moving forward.

This sounds like more work, and in the short term it is. But it’s the only way to preserve the structural controls that the rest of your system already depends on. Without it, you’ve quietly eliminated segregation of duties for any business process that involves an agent, and your auditors (whether internal or external) are eventually going to have something to say about that. More importantly, you’ve eliminated the inspection points that you yourself need to debug the system when it misbehaves. A long agent reasoning chain with no internal boundaries is not just an audit problem; it’s an incident response nightmare. When something goes wrong, you have a transcript. You don’t have a control surface.

Agents propose, systems decide

Here’s another unexpected one, and it’s the architectural pattern that ties the previous points together.

The instinct, when wiring up an agent, is to give it the tools it needs and let it use them. The agent calls an API, the API does the thing, the result comes back to the agent, the agent decides what to do next. This works in demos, and it scales until it doesn’t, at which point it scales catastrophically.

The more reliable pattern is to treat agents as proposers, not actors. The agent doesn’t issue the refund; the agent produces a structured proposal that says “I think we should issue a refund of this amount to this customer for these reasons.” That proposal is a typed object: known fields, known shape, known constraints. Deterministic code (not the agent) inspects the proposal, validates it against business rules, checks it against the agent’s permissions, and either enacts it or rejects it. The agent never touches state directly. The agent produces artifacts that the system either commits or doesn’t.

This is the same pattern a database uses when accepting a transaction. The client proposes a write; the engine validates it and either commits or rolls back. We don’t let clients write directly to disk because we figured out a long time ago that this ends badly. The same logic applies to agents and system state, for the same reasons.

The benefits compound quickly. Validation logic lives in deterministic code, which means you can test it, version it, and reason about it. The agent’s output becomes inspectable before it has any effect, which gives you a natural checkpoint for human review on high-stakes actions. Model upgrades and prompt changes become safer because the validation layer catches anything that drifts outside expected bounds. Audit becomes meaningful, because every state change is linked to a specific proposal, a specific validation result, and a specific authorization. You’re no longer logging chat transcripts and calling it accountability; you’re logging actual events with actual attribution.

The cost is that you have to do the design work upfront. You have to decide what proposals look like, what validates them, and what enacts them. You can’t just hand the agent a tool and walk away. But this is exactly the work that makes the system work, and skipping it is what produces the incident reports that show up six months later.

Build for the bad day, not the demo

Every agent system works beautifully in the happy path. The interesting question is what happens when the agent is wrong, because the agent will be wrong, regularly, by design. These are probabilistic systems. They produce incorrect outputs some percentage of the time, and that percentage is never zero. An architecture that only works when the agent is right isn’t a system; it’s a demonstration.

So the architectural question is not “how do we make the agent always right” (we can’t) but “how does the system behave when the agent is wrong.” Some of this is covered by the proposal-and-validation pattern: bad proposals get caught at the validation layer and never become state changes. Some of it is covered by tight permissions and rate limits: the blast radius of any single bad action is bounded by what the agent’s identity is allowed to do. But some of it requires explicit thinking about reversibility.

Every state change driven by an agent should fall into one of three categories. Reversible actions can be undone if it turns out the agent shouldn’t have taken them; these are the safest and need the lightest gates. Compensable actions can’t be literally undone but can be offset by a corresponding action (a refund offsets a charge, a retraction offsets a sent message); these need slightly heavier gates. Irreversible actions cannot be undone or offset (sending an email to a customer, transferring money to an external account, deleting data, calling a third-party API with side effects); these need the heaviest gates, often including human approval, and should be designed out of the agent’s path entirely wherever possible.

Most agent incidents that make the news involve irreversible actions taken without sufficient gating. The system worked exactly as designed; it’s just that the design didn’t account for the bad day. Designing for the bad day means asking, for every action the agent can take, what happens if this turns out to be wrong, and making sure the answer is something other than “we deal with the consequences.”

The decisions that matter happen before the first agent ships

Here’s the pattern that emerges if you look at all of this together. The architectural decisions that determine whether an agent-ready system actually works are decisions about identity, permission scope, principal separation, input validation, proposal patterns, validation gates, rate limits, reversibility, and audit structure. These are not AI decisions. They’re systems design decisions, made with ordinary systems design vocabulary, applied to a principal class that the field is still learning to recognize.

And critically, these decisions all have to be made early. Once a system is built around a single overpowered agent identity with direct access to state, you cannot retrofit principal separation without rewriting most of it. Once an agent is producing free-form actions that get enacted directly, you cannot retrofit a proposal-and-validation layer without breaking everything downstream. Once permissions have accumulated and nobody quite knows which ones are actually used, you cannot tighten them without an archaeology project. The properties that make an agent-ready system work are properties of the architecture, and architectures are easier to design well than to fix later.

This is the part that should land hardest for solutions architects: the reliability of the agent system you ship is determined mostly by decisions you make before you write the first line of agent code. By the time you’re tuning prompts or choosing models, the structural shape of the system is already mostly fixed. If you didn’t design checkpoints in, you don’t have them. If you didn’t separate principals, you have a single overpowered actor pretending to be a workflow. If you didn’t define what a proposal looks like, the agent is mutating state directly and you’re hoping for the best. None of these are problems you can solve with a better model or a smarter prompt. They’re problems you solve, or fail to solve, at the architecture phase.

The infrastructure questions that everyone starts with (where the model runs, how memory works, which vector store) are real, but they’re solved problems with vendors lining up to help you. The architectural questions in this article are the ones that don’t have vendors yet, because they’re not products. They’re decisions, and they belong to whoever is designing the system.

If you’re designing a system that will include AI agents, the unexpected work is not figuring out the AI parts. It’s recognizing that agents are a new class of principal in your system, and applying the same architectural discipline to them that you would apply to any other principal that could read and modify state at high speed with imperfect judgment. The vocabulary you already have (identity, permissions, validation, audit, blast radius, segregation of duties, reversibility) is exactly the vocabulary you need. The only thing that’s new is recognizing that it applies here too, and that it has to be applied early, before the agent ships, while the architecture is still soft enough to shape.

That’s the unexpected part. Not the AI. The architecture.