Context Engineering Has Replaced Prompt Engineering

The demo always works

Here's the pattern I've watched play out a dozen times in the last year. Someone wires up an agent, records a two-minute Loom, and it's genuinely magical — it books the meeting, drafts the reply, reconciles the invoice without being asked twice. The room leans in. We greenlight it. Then it meets real users, real data, and real edge cases, and the magic curdles into "why on earth did it do that?" The reflex is to blame the model. But the thing that actually broke is the work we now call context engineering: everything you assemble around the model before it ever generates a token.

That reflex — ship a bigger model, wait for the next release, bolt one more sentence onto the prompt — is almost always the wrong diagnosis. The model was fine. What failed was the information, tools, and history you handed it. And if you build products for a living, this is quietly becoming the most important skill on your team.

Prompt engineering was the training wheels

Eighteen months ago, "prompt engineering" was the whole conversation. Find the magic phrasing, add "you are an expert," promise the model a tip, and results improved. That worked because most AI features were one-shot: one prompt in, one answer out.

Agents broke that model. An agent runs in a loop — it calls a tool, reads the result, decides the next step, and does it again, sometimes for dozens of turns. There is no single prompt to perfect. There's a constantly changing pile of system instructions, tool definitions, retrieved documents, and message history, and the real question becomes: what should be in front of the model right now? That's why a term that barely existed at the start of 2025 has taken over. LangChain's Harrison Chase defines context engineering as "building dynamic systems to provide the right information and tools in the right format such that the LLM can plausibly accomplish the task." His sharper line is the one worth taping to your monitor: "Most of the time when an agent is not performing reliably the underlying cause is that the appropriate context, instructions and tools have not been communicated to the model."

Anthropic frames it as the natural progression of prompt engineering — less about finding the right words, more about answering "what configuration of context is most likely to generate the behavior we want?" Prompting is a subset. Context is the whole surface.

Context is a budget, not a bucket

Here's the counterintuitive part that trips up most teams. When an agent fails, the instinct is to give it more — more instructions, more documents, more tools, a bigger context window. That instinct is backwards.

Chroma's research team ran a study they called Context Rot, testing 18 leading models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3 — and found that performance degrades as input length grows, even on trivially simple tasks. Models don't read the 10,000th token as carefully as the 100th. Anthropic describes the same phenomenon as an attention budget: every token you add depletes it, because the transformer has to weigh every token against every other one. More context isn't free. Past a point, it actively makes the model dumber.

So the goal of context engineering isn't "stuff everything in just in case." Anthropic puts it well: find "the smallest possible set of high-signal tokens that maximize the likelihood of your desired outcome." You're not filling a bucket. You're spending a budget, and most teams are wildly overspending.

What the job actually looks like

Once you accept that context is scarce, a concrete set of moves falls out — and none of them are about clever wording:

Retrieve just in time. Instead of dumping every document into the prompt up front, give the agent lightweight references — file paths, links, saved queries — and let it pull the specific thing it needs when it needs it. It's how a person works with a filing cabinet instead of memorizing the office.
Compact as you go. When a long conversation approaches the limit, summarize it and start fresh with the summary, preserving decisions and open threads while dropping the redundant tool output nobody needs to see twice.
Take structured notes. Let the agent write to a scratchpad outside the context window and read it back later, so it can hold a goal across dozens of steps without carrying every detail in working memory.
Keep agents small. A focused sub-agent with a clean context window beats one bloated agent trying to hold an entire project in its head. Anthropic's research agents each burn tens of thousands of tokens exploring, then hand back a distilled 1,000–2,000-token summary.

If you want the builder's version of this, Dex Horthy's 12-Factor Agents makes "own your context window" a first-class principle — and his blunt observation is worth sitting with: most things billing themselves as "AI agents" are "mostly deterministic code, with LLM steps sprinkled in at just the right points." The reliable ones aren't the ones that hand the model a bag of tools and pray. They're the ones that engineer, turn by turn, exactly what the model gets to see.

Why this is a product problem, not just an engineering one

It would be easy to read all of this as backend plumbing and hand it to the engineers. That's a mistake, and it's the same mistake that's sinking most AI initiatives — which is why 95% of AI pilots quietly fail while a handful ship something people actually rely on.

The context window is a product surface. Which knowledge base the agent can reach, which tools it's allowed to call, what it remembers about a returning user, what it's told to forget — those aren't implementation details. They're product decisions that determine whether the thing feels magical or unhinged. Deciding that a support agent should see the customer's last three tickets but not their entire billing history is exactly the kind of judgment call product people are supposed to be good at.

Two things follow. First, you can't eyeball this. The difference between a context setup that works and one that rots shows up in aggregate, across thousands of runs, not in the one demo you cherry-picked — which is precisely why evals have become the new PRD for teams shipping AI. Write down what "good" looks like, measure it, and change the context deliberately. Second, tokens are money. Every unnecessary document you cram in is both an attention tax and a literal one, so tighter context is better margins and better answers at the same time. That's a rare two-for-one, and it lands squarely on product to notice it.

The takeaway

Prompt engineering got you the demo. Context engineering is what gets you a product. The teams pulling ahead right now aren't the ones with a secret model or a magic prompt — they're the ones treating the context window as their most precious, most finite resource and spending it with intent.

So the next time an agent does something baffling, resist the urge to reach for a bigger model or a cleverer sentence. Ask the better question first: what did I actually put in front of it — and what should I have left out?