Skip to Content

Insights Blog

AI Agent Architecture: The 5 Decisions That Matter

Copy LinkPrintEmailFacebookX
A software architect sketching a system diagram of connected boxes and arrows on a whiteboard in a Chicago office

Most writing on AI agent architecture lists components: a model, some tools, memory, an orchestrator. That is an inventory, not an architecture. An architecture is the handful of decisions you cannot cheaply reverse once the system is in production. After twenty-plus years shipping production software at LoadSys, and the recent stretch building agent systems with Claude Code, Cursor and the Anthropic APIs, I count five of those decisions. Get them right and the rest is plumbing. Get them wrong and you will find out from an agent that confidently reports success on work it never finished.

What AI agent architecture actually means

An AI agent is a program in which a language model decides, at runtime, what to do next. That single sentence separates agents from the LLM features most teams already ship. A summarizer calls a model once and returns. An agent calls a model, reads the result, calls a tool, reads that result, and keeps going until some condition says stop. The loop is the agent. Everything else is support structure around the loop.

Anthropic’s engineering team draws the line in a way I find useful: workflows are systems where the model and tools follow code paths you wrote in advance, while agents are systems where the model directs its own process and tool usage. The same guide lists the common building blocks (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) and recommends starting with the simplest one that works. I agree, and the reason simplicity matters is not elegance: every point where the model chooses the next step is a point where you must later prove it chose correctly.

So the architectural question is never “which components do I need.” The question is where the model gets to decide and where your code decides for it. The five decisions below are where that line gets drawn.

Decision 1: Who holds the control flow

The original ReAct pattern, from the 2022 paper by Yao and colleagues, interleaves reasoning and action in a single loop and lets the model drive. It is flexible and it is the default in most frameworks. It is also the hardest shape to make predictable, because the step count, the cost and the failure modes are all set by the model at runtime.

The alternative is a planner-executor split: the model produces a plan up front, your code (or a separate model call) executes the steps, and re-planning only happens at checkpoints you define. Costs become bounded. Traces become readable. The trade is that a bad initial plan fails late instead of adapting early.

My rule: if a human expert could write down the steps before starting, the control flow belongs in code and the model fills in the steps. If the expert would say “it depends on what I find,” the model holds the loop, with a hard iteration cap and a budget. On client work we almost always end up with a hybrid, where a deterministic outer workflow owns the stages and a bounded agent loop runs inside each stage. I covered the planning half of that in detail in our piece on plan, export, verify workflows for coding agents.

Decision 2: Where state lives and what survives a crash

A demo agent keeps everything in process memory. A production agent runs for minutes or hours, calls external systems that time out, and gets redeployed mid-task. If step seven of twelve fails on a vendor 503 and the whole run restarts from step one, re-sending the three emails it already sent, you do not have an architecture yet.

The fix is durable execution: persist the state of the run after each step, make every step idempotent, and resume from the last checkpoint instead of from the start. Whether you use a workflow engine or a table of run records and a queue, the property you are buying is the same: any step can be killed and re-run without double effects.

Decide this early, because it changes how you write tools. “Create an invoice” has to become “ensure an invoice with this idempotency key exists.” That is a small change on day one and a painful retrofit on day ninety.

An engineer at a desk reviewing a code diff on one monitor with terminal log output on a second monitor

Decision 3: The tool boundary and its permissions

Tools are where an agent touches the world, which makes the tool layer your security boundary. Three things belong in it. First, schemas: every tool takes typed, validated input and returns typed output, and the model never passes a raw SQL string or shell command through to a system of record. Second, scoped credentials: the agent runs as a service identity with the minimum permissions the task needs, not as the user who kicked it off and never as an admin. Third, a write ledger: every state-changing tool call is logged with its arguments, so “why did the agent do that” is answered from a record, not a guess.

The reason to be strict is prompt injection. An agent that reads untrusted content (emails, web pages, tickets, documents) will eventually read content written to redirect it. The OWASP Top 10 for LLM Applications ranks prompt injection first for a reason. You cannot fully prevent the model from being persuaded; you can make sure a persuaded model has no tool capable of real damage without a human approving it.

One standard worth adopting here is the Model Context Protocol, which gives tools a common schema and transport across models and frameworks. It does not solve permissions, but it keeps the tool surface in one auditable place.

Decision 4: What the agent is allowed to remember

Memory discussions tend to get taxonomic: working, episodic, semantic, procedural. The architectural decision is simpler and more uncomfortable. Memory is state that influences future decisions without a human reviewing it, so every piece of it is a liability as well as an asset.

I separate three stores with three policies. Run-scoped context is the conversation and tool results for the current task; it lives in the context window and the durable run record and is discarded when the run ends. Retrieved knowledge is read-only reference material (documentation, policies, records) fetched per step; the agent cannot write to it. Learned memory is anything the agent persists about a user or task for later runs; it is the smallest store, written only through a schema-validated tool, and a human can list and delete it.

Most agents need the first two and not the third. When a client asks for learned memory, I ask what decision it will change and who will be able to see what was stored. If nobody can answer the second question, we leave it out. Memory is a product and compliance decision dressed up as a technical one.

Decision 5: How done gets proven

This is the decision most architecture guides skip and the one that matters most in practice. A language model will tell you the task is complete, fluently, with a summary of what it did. Sometimes that summary describes work that did not happen.

Developers already know this. In the 2025 Stack Overflow Developer Survey, 66 percent of respondents named “AI solutions that are almost right, but not quite” as their top frustration, and more developers said they distrust the accuracy of AI tools (46 percent) than trust it (33 percent). An agent is that same almost-right behavior running unattended across many steps.

So the architecture needs a verification path that does not rely on the model’s self-report. For a coding agent, that means the tests run in a sandbox and the exit code, not the model’s claim, decides whether the step passed. For a data agent, it means a reconciliation query against the target system. For a customer-facing agent, it means a second evaluator (a model with a narrow rubric, or a rules check, or a person) reviewing the output before it leaves the building. I wrote about this failure mode for coding agents specifically in why coding agents need proof of completion, and the same principle applies to every agent we ship: the agent proposes, something else confirms.

Human gates belong here too. Decide up front which actions are reversible (draft an email, open a pull request, stage a record) and which are not (send, merge, delete, pay). The agent may do the first class freely and must stop at the second until a person approves. With durable execution in place, that approval is just another checkpoint the run resumes from.

Two engineers at a wooden table reviewing printed architecture diagrams with coral sticky notes

A reference shape that holds up in production

Put the five decisions together and a fairly consistent shape emerges. A deterministic workflow owns the stages of the task and persists a run record after each one. Inside a stage, a bounded agent loop (capped steps, capped spend) calls a model that can reach a small set of schema-validated tools through a scoped service identity. Reference knowledge is fetched per step and is read-only. Every state-changing tool call is logged. Each stage ends with a verifier that is independent of the model, and irreversible actions pause for a human.

Notice what is not in that description: a specific framework, a specific model vendor, or a multi-agent swarm. Frameworks help you build the loop faster, but I would not let one choose your state model or your permission model for you. Multi-agent designs earn their place when the task genuinely splits across specialties and the coordination cost is paid back in parallelism; a second agent doubles the surface you have to verify. Where multiple agents do belong, the question becomes how they talk to each other. We compared the two main contenders in our look at A2A and MCP; the short version is to standardize tools before worrying about agent-to-agent messaging.

Write the five decisions down before the first sprint

The cheapest time to make these decisions is before anyone opens an editor, and the cheapest way to make them is in writing. On our projects the agent’s specification has a one-page section with five headings that match the decisions above, and nobody starts building until each heading has an answer a reviewer can argue with.

Under control flow, the spec names the stages that code owns, the stages where the model may loop, and the iteration cap and spend cap for each loop. Under state, it names the run record, the step boundaries where it is written, and the idempotency key for every tool that changes something. Under tools, it lists each tool with its input schema, the service identity it runs as, and whether the model may call it without approval. Under memory, it says which of the three stores exist, what may be written to the learned store, and who can read and delete it. Under verification, it names the check that ends each stage and the actions that wait for a person.

Two things happen when you force this onto one page. First, a surprising share of proposed agents turn out to be workflows: once the stages are written down, the model’s freedom shrinks to one or two steps, and the rest is code. Second, the review conversation changes. Instead of watching a demo, the reviewer reads five answers and looks for the one that is missing or vague, which is almost always verification. A spec that says “the agent confirms the task succeeded” has not made the fifth decision. A spec that says “the stage passes when the reconciliation query returns zero differences” has.

The page also becomes the acceptance test. When the build is done, each heading maps to something a reviewer can open: the run table, the tool registry, the permission configuration, the memory schema and the verifier’s output. Without the page, the question of whether the agent is finished gets answered by the agent itself, which is the problem this whole article is about.

When you should not build an agent at all

The most expensive agent architecture is the one you did not need. If the task has a fixed number of steps and known inputs, a workflow with a few model calls will be cheaper, faster and far easier to test. If the task needs one judgment call, put a human there rather than a loop. Reserve the full agent shape for work where the path cannot be known in advance: investigating an incident, reconciling messy records, migrating code where each file reveals the next problem.

That last category is where we spend most of our time. Modernizing a legacy application with AI coding agents is an agent problem by nature, and the five decisions above are what our AI agent development work and our spec-driven modernization process are built around: a written specification holds the control flow, every change is checkpointed in version control, tools are scoped to the repository, and nothing counts as done until a senior engineer has reviewed the commit and the tests have passed. If you would rather compare firms before deciding how to build, I keep a list of Chicago firms doing AI application and production agent development.

If you are running an aging custom application and wondering whether this approach fits it, the fastest way to find out is to talk. A 30-minute call. We’ll tell you if your app fits in 90 days.

Frequently asked questions

What is the difference between an AI agent and an LLM workflow?

In a workflow, your code decides the sequence of steps and the model fills in content at each step. In an agent, the model decides what to do next based on what it has seen so far. Agents are more flexible and harder to make predictable, so they should be reserved for tasks whose steps cannot be known in advance.

Which AI agent architecture pattern should I start with?

Start with the simplest shape that can do the job, usually a deterministic workflow with one or two model calls. Move to a planner-executor pattern when steps vary, and to a model-driven loop only when the path is unknowable up front. Add a second agent only when the task splits cleanly and parallel work pays for the coordination overhead.

How do I stop an AI agent from claiming it finished work it did not do?

Never let the model’s own report be the signal of success. Each stage should end with an independent check: a test suite exit code, a reconciliation query, a rules-based validator, or a human review. Treat the model’s summary as a proposal that the verifier confirms or rejects.

How should an AI agent handle failures mid-task?

Persist the state of the run after every step and make each step idempotent, so a crash or timeout resumes from the last checkpoint instead of restarting. Add retry limits and a budget cap so a stuck agent stops rather than looping, and route irreversible actions through a human approval that fits the same checkpoint model.

What is the biggest security risk in AI agent architecture?

Prompt injection through untrusted content the agent reads. Because the model can be persuaded, the defense lives in the tool layer: schema-validated inputs, least-privilege credentials, a log of every state change, and human approval for actions that cannot be undone.

Back to top