AI Agent Guardrails: Autonomy Is a Product Decision

The issue that said "Additionally"

Earlier this year, security researchers played a small, devastating trick on GitHub's AI agent. They opened an issue in a public repository — the kind of thing anyone can do — and buried a few instructions inside it. When the agent picked up the issue, it read those instructions as marching orders, reached into a private repository it had access to, and helpfully posted the contents as a public comment for the entire internet to read. The keyword that slipped past the safety training, according to Noma Security's writeup, was as innocent as it gets: "Additionally."

Nobody hacked a server. No password leaked. The agent worked exactly as designed — it read some text and did what the text said. That's the whole problem. And if you build products with agents in them, that story should keep you up at night, because the thing that failed wasn't the model. It was the AI agent guardrails nobody decided to build.

The constraint flipped

For two years the question about AI agents was "can it do enough?" We wanted them to book the meeting, file the ticket, reconcile the invoice, close the loop. Capability was the bottleneck, so capability is what we chased.

In 2026 the bottleneck flipped. Agents can do plenty now — Gartner projects that 40% of enterprise applications will ship task-specific agents by the end of this year, up from under 5% in 2025. They're in production, touching real data, taking real actions. And the new question isn't whether the agent can do enough. It's whether it can do too much — and whether you can live with what happens when someone convinces it to.

There's a proper name for this. OWASP added it to their Top 10 for LLM applications and called it Excessive Agency: "the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM." Note what's not in that sentence: a bug, a breach, a broken model. The damage comes from an agent that was granted more functionality, more permissions, or more autonomy than the job required. You don't need the model to malfunction. You just need it to be gullible while holding the keys.

The lethal trifecta

The clearest way to think about this comes from Simon Willison, who named the failure pattern the lethal trifecta. An agent becomes dangerous the moment it has all three of: access to private data, exposure to untrusted content, and the ability to communicate externally. Any two are fine. All three, and an attacker who controls the untrusted content can walk out with the private data.

The GitLost attack is the trifecta in one screenshot. Private data: the agent could read a private repo. Untrusted content: a public issue anyone could file. External communication: the ability to post a public comment. Line those three up and the exploit writes itself.

Here's the part product people need to sit with. Willison's argument is that this isn't a patch-it-and-move-on bug, because "LLMs are unable to reliably distinguish the importance of instructions based on where they came from." Untrusted text flows into the same context window as your careful system prompt and becomes, functionally, another instruction. The guardrail vendors who claim to catch these attacks tend to stop around 95% of them — which, as he dryly notes, is "very much a failing grade" in security. His blunt conclusion: "we still don't know how to 100% reliably prevent this from happening."

If there's no reliable technical fix at the model layer, then the only real lever is how much you let the agent do — and that lever lives in product, not in the model.

Why guardrails are a product decision, not a security afterthought

It's tempting to file all of this under "security" and hand it to the engineers with the pager. That's the same reflex that's sinking most AI initiatives — the belief that the hard part is the technology, when the hard part is the judgment. It's the reason 95% of AI pilots quietly fail while a few ship something people actually trust.

Look at what "adding guardrails" actually means, using OWASP's own list of mitigations: minimize the tools the agent can reach, minimize what each tool can do, avoid open-ended capabilities, scope permissions to the individual user, and require human approval before any high-impact action. Read that again. Every one of those is a product decision dressed up as a security control.

Which tools does the agent get? Deciding a support agent can read a customer's last three tickets but not issue refunds without a human is scoping, and scoping is product work.
What can it do without asking? The line between "just do it" and "show me first" is a UX decision about trust and friction — the most product decision there is.
Whose permissions does it borrow? An agent that runs with god-mode service credentials instead of the requesting user's own access is a product choosing convenience over blast radius.

None of that is code you can bolt on at the end. It's the shape of the feature. A refund agent that always confirms before moving money is a different product than one that doesn't — different trust, different failure modes, different feeling in the user's stomach. You can't retrofit that; you design it in or you design it out.

Build the dial, then decide where to set it

The mental model I keep coming back to is a dial, not a switch. Autonomy isn't on or off. It's a setting you choose, action by action, and the job is to build the dial before you build the feature.

Concretely, for anything an agent can touch: default to read-only, and make writes — sending, deleting, paying, publishing — the deliberate exception that earns a confirmation step or a human in the loop. Give the agent the narrowest credentials that let it do the job, not the widest that were lying around. And because a runaway agent burns money as fast as it burns trust, put in the same circuit breakers you'd want for the token bill — hard caps on calls and spend before a loop spirals. The teams that skip this aren't being fast; they're just moving the cost to later.

The data says most teams are skipping it. McKinsey's 2026 AI Trust survey found that only about one in three organizations is governance-ready for the autonomous agents it's already deploying, and nearly two-thirds named security and risk as the top barrier to scaling — ahead of cost and regulation. Gartner's forecast lands in the same place: more than 40% of agentic AI projects will be canceled by 2027, done in by runaway costs, unclear value, and agents that behave in ways that create risk. Those cancellations won't come from agents that couldn't do enough. They'll come from agents that did something nobody sanctioned.

The takeaway

The next time your team demos a slick new agent, resist the instinct to ask "what else can it do?" Ask the harder, more useful question first: what can it do that I haven't decided it should? Then answer it on purpose — one tool, one permission, one confirmation at a time.

Capability was the last war. In a world where any patch of untrusted text can whisper "Additionally" and be believed, the teams that win aren't the ones whose agents can do the most. They're the ones who decided, deliberately and in advance, exactly what their agents were allowed to do — and built the guardrails to hold that line.