Replace code reviews with verified intentAviator Verify
Aviator
All episodes
AINovember 19, 2026

Your Agent Is Only as Good as Its Context: The CAFE(S) Framework

Brian Houck and Max Kanat-Alexander talk about the CAFE(S) framework for high-quality agent context, why a failure in just one of its dimensions doubles token use, and why human attention is the scarce resource in agentic development.

About the guests

Max Kanat Alexander

Max Kanat Alexander

Developer Experience, Capital One

Brian Houck

Brian Houck

Distinguished Scientist, DX

Brian Houck is a Distinguished Scientist at DX, where he studies developer productivity. Max Kanat-Alexander is one of the technical leads for developer experience at Capital One, where he helps lead AI-assisted software development. Together with Eirini Kalliamvakou (DX, formerly GitHub), Margaret-Anne Storey (University of Victoria), and Nicole Forsgren (Google), they co-authored CAFE(S): Your Agent Is Only as Good as Its Context, published in ACM Queue.

Why CAFE(S)

Brian: As I was looking across 500 different organizations' worth of data, I was seeing that token usage was going up 20x over the last four quarters. At the same time, output was going up 2x, so an order of magnitude less. And outcomes — our ability to deliver high-quality innovation — were sort of flat.

We are using more and more tokens to produce more work, with substantially less growth, and not getting the outcomes we wanted. Clearly, we weren't getting the effectiveness out of our AI agents that we might like. The thesis was that we needed to investigate the context we hand to agents.

Human-agent collaboration starts with an idea, something a person wants to achieve. That idea has to get translated into context — all of the information about the task, the user prompts, the shared prompts, the data, the code, everything you hand to the agent. That then gets passed into a harness for orchestration and retrieval before going to the model, which provides the underlying reasoning.

Everything in this flow is important. But there are already lots of efforts to improve the other parts. There's harness engineering, AI engineering to build more capable models, and product planning to help us solidify our ideas. We didn't find anyone looking at the quality of the context holistically — only bits and pieces.

How Low-Quality Context Fails

In the paper, we looked at real-world use cases and published research, not things that are theoretical, but things that actually result in context failures. We talk about hallucination a lot, but it goes far beyond that.

Weekend runaway. You set a bunch of agents out to solve a task on a Friday afternoon, and you come in Monday morning and see that you've completely exploded your token quota. Those agents didn't have a good definition of where the finish line was, so they just kept going.

Specification ambiguity. Agents don't often stop to clarify requirements. If they have to choose an interpretation, they will often do it silently and then proceed pretty confidently. The earlier it happens in the task, the more those assumptions compound if they are wrong.

Lost in the middle. We have this conversation in the industry about larger context windows. I would posit that if we had infinite context windows, our results would often be lower quality, because agents get distracted. If you embed highly relevant context in much broader context windows, agents struggle to find it.

Prompt injection. We've had SQL injection for years, and we now have well-publicized cases of executable commands being injected into context that agents will then act on. That could leak customer data or introduce vulnerabilities.

Five Durable Properties of High-Quality Context

Clarity. Can the agent interpret the request the way it was intended? To the author, we know what we mean. It's in our head. But we don't always write it down in a way the agent can interpret.

Actionability. Does the agent actually know what it is trying to accomplish? Or are you just throwing spaghetti at the wall? Can the agent proceed and know when it is finished?

Fidelity. Is the context accurate and up to date? Can the agent trust that it is true? This has been a challenge with documentation forever. As recently as February of this year, I found that 88% of engineers regularly waste time trying to find up-to-date documentation. With the scale and speed of agents, this gets compounded even more.

Efficiency. Is the context lean enough to be both usable and affordable? It's not about having the smallest context, it's about having the highest signal to noise. When I investigate outliers of massive token usage, it is almost always a failure of efficiency. Someone is trying to reason about a single method in a single class, so they pass in the entire repo every time. Or they're trying to diagnose a single failure, so they pass in all of their build logs, which the agent will happily go and read.

(S)ecurity. Is the context safe and compliant to use at all? Or is it compromised in some way?

Better context, we believe, will result in better agents, which will lead to better outcomes.

Make Context Accessible and Discoverable

Max: There are prerequisites. First, the context has to be accessible to your agents. Many companies are in a place where they can't even get at the context. If you can't, you're probably not yet at the place where you're worried about its quality. Your first step should be getting your basic needs in place.

Next is discoverability within the accessibility. You have to be able to find the right thing in the giant pile of data you have. Let's say you have 15 different MCP servers connected to your agent, providing a hundred tools, and ten of those tools are named something with "search." The agent cannot figure out what to do. It will often just choose not to search anything. Tool instructions are prompts too, so even the context of how to get the context can be a problem.

Almost exactly the same things you were doing for knowledge management for developers turn out to be exactly the same things that need to be done for agents.

Brian: There's also the question of whether it's written down to begin with. So much knowledge exists in hallway conversations, in people's heads, and on physical whiteboards. As Max has told me many times, AI is an amplifier. Organizations that cared about documentation quality, that rewarded keeping it up to date as a cultural norm, are going to have better outcomes.

Agent-Written Documentation and the Context Flywheel

Brian: It creates a flywheel effect. A human has an agentic session, which results in context. If that's high-quality context, it will feed the next session. If it's low-quality context, it will also feed the next session.

Agents are writing documentation both for humans and for other agents to read, and in both cases it can be high or low quality. A lot depends on the constraints, guardrails, and requirements those agents have been given. Individuals can think about the CAFE(S) dimensions as they create their prompts, their shared skills, and as they assign work to agents to write future context. Make sure anything they write is clear and unambiguous, terms are defined, sources are well cited, and it's secure. These need to be requirements given to agents writing documentation, just like they're requirements for humans.

Max: There's a scale of quality.

If you're writing disposable software, the quality of your context matters less. If you're writing software you intend to be durable over a very long time, you need to start treating your context just like you treated your code in terms of maintenance.

My experience so far, living on the frontier with my side projects and talking to other people who do, is that you want the agent to have access to primary sources, and you want the data in those primary sources to be correct. Ideally, you direct the agent to the primary source and not a transformation of the primary source.

Brian: I believe the future is we will treat context very similarly to how we treat code today. I expect we will be reviewing context, particularly shared skills, and the more consequential they are, the more thoroughly we'll review them. An AGENTS.md we're going to review pretty carefully. Much like with code, we'll have dependency graphs and nested structures of shared system prompts, and our quality threshold will depend on how consequential each piece is.

Can You Write Evals for Context?

Max: There's a cost-benefit trade-off to optimizing the quality of your context. You could take Claude Code or Codex today and say: give a context-free subagent this document, see how well it does at accomplishing this task, and then delete text until you figure out the minimum set of instructions required to get the result. The problem is that's very expensive. The cost of that eval might be greater than all the cost you will ever save by improving that context.

If you're writing long-term durable software with huge consequences and millions of users, you would probably care. If you're writing a one-off tool that's going to be used by ten people, you probably wouldn't.

Brian: I think there are layers of testing that are possible, like with code. Take security: there's a bunch of deterministic static analysis you could do over context, like checking for hard-coded credentials.

One thing I'm looking at right now is whether you can take all of your agentic session transcripts and use an LLM-as-judge technique, not only to evaluate them but to provide coaching to humans for future sessions. How clear were the terms? Was there a lot of back and forth to clarify what you meant? Was there a definition of done? Was a lot of data dumped in that the agent never used?

When evaluating these session transcripts, a failure of just a single dimension of CAFE(S) on average doubles the number of tokens required and decreases accuracy by 30%. You spend more to get worse results.

Don't Look at One Dimension

Brian: An improvement in one dimension can inadvertently lead to a failure in another, and you have to balance them. Efficiency is the clear one. We also see failures travel together: failures in clarity and failures in actionability are often very highly tied. I found this while trying to create semi-synthetic transcripts to run evals over. Even scripting all of my human-side interactions very intentionally, it's really hard to keep these things isolated. They might be conceptually different, but they travel together.

Much like everywhere else in developer productivity, that's why you can't look at just one dimension or distill it down to a single metric. Sometimes you have to make intentional trade-offs.

Max: One thing I've experienced more recently is a trade-off on clarity, actionability, and efficiency between letting the model use its judgment and not. Sometimes you want to be very specific. And sometimes you're wrong and the model would have been right. Some of that is how justified your confidence is as a human being. I have definitely had coding sessions where I got to the end and thought, I should have done the thing you told me to do at the start. You were right, and I was wrong.

Brian: That's why it's collaboration. It's human-agent collaboration.

Who Owns Context

Max: It's the same tragedy of the commons DevEx teams have experienced forever. You have a hundred stakeholders who all want something from you and don't understand the other 99, and the business doesn't understand why DevEx is so expensive.

If you provide a platform, you need to start thinking in your estimates about how to make that platform agentically useful. That's not just context, it's everything. How is the MCP going to really work, not just thrown over the wall for somebody else to hook in? How are you, the owner of the platform, going to make sure the agent continues to find the correct information and instructions about your system?

This is new work that you cannot hide as run-the-business work. And a lot of the time, it is just the work you should have always done to make this a really great developer platform for everybody.

In the past, some of those things were afterthoughts, especially documentation. What happens when something goes wrong? What's the feedback mechanism? What's the telemetry that lets me know things are going wrong, and how do I investigate it? Someone who came from production observability recently told me that building observability for developer platforms is totally different and actually much harder.

Brian: It's going to look a lot like developer platforms, where there was an economic advantage to having centralized teams solve problems for lots of people. It'll probably still be developer platform teams who create central infrastructure to manage ownership of individual pieces of context, handle revisions, and come up with shared standards and tooling for evaluating context quality. Then teams, like with code, will be responsible for adhering to those standards and reviewing their own context. Hub and spoke, in practice.

As a researcher, I have no idea what it will actually look like. If you do something wildly different and it's incredibly successful, please tell me. I want to incorporate that into future revisions of the work.

Human Attention Is the Scarce Resource

Should agents just stop and ask clarifying questions more often?

Max: The cost of agentic software development is attention. The scarce resource is human attention. Everything you're trying to do with your system is to optimize how much human attention it needs.

I think the ideal system is one where there are no decisions to be made.

To anybody in the current world of software, that sounds insane, because you use a bunch of tools that give you choices you don't need to make. You're allowed to write functions in Python a million different ways. You probably don't need that flexibility.

Ideally, the decisions you make are about intent and what you want to build—the creative parts—and not the fiddly bits. I'd like agents to handle all the fiddly bits, and to come to a world where there's only one way to do them and it's the right way. Not one bad way, which has been the danger of abstraction for most of software's existence.

Eliminate the bad ways so the agent can't make bad decisions. And then, where my intent isn't clear, please ask me questions.

Brian: Stopping to ask limits the blast radius, but it should only be necessary if you weren't able to specify what you needed ahead of time. I released a paper recently showing there's a two-way scaffolding between humans and agents. Agents explaining their thinking still helps us understand the systems. But stopping to ask "did you really mean this?" — in general, those are largely failures.

Human attention is under assault with AI. The more someone uses AI, the bigger the spike in interruption frequency, and the lower our ability to do sustained deep work. Constantly checking in on agents and farming them out is not how human creativity works. We still need large uninterrupted blocks of time to think about what we want to build.

To this day, the amount of time available for deep work without interruption remains by far the strongest predictor of innovation delivery of any variable I've tested. It is so important, and we are getting it wrong.

Not All Context Needs Review

Brian: We're going to treat shared context like code, and much like we're seeing with code, we're not going to force humans to review every change anymore. How broadly it's shared and what its impact is will determine the level of rigor.

The bigger risk is conflicting sources of truth, and the problem is exacerbated now. Agents are way better at finding conflicting sources of information. If I have a source of truth I want the agent to use, but an old conflicting source is buried in a downloads folder somewhere, the agent is going to find both, choose the one it wants, and proceed. Oftentimes it won't be right. I run into this in the real world all the time.

Why the S Is in Parentheses

Brian: Among the authors, I was the strongest advocate for putting the S in parentheses, and Max tested that concept the strongest on the other side.

The first four dimensions of CAFE(S) are about whether the context is any good. Security answers a fundamentally different question: should we be using this context at all? Is it safe and compliant?

On a practical level, I was already thinking ahead to measurement. CAFE(S) is a definition framework, not a measurement framework. I could imagine how I'd measure the first four dimensions, but security would be measured very differently because of the inherent trickiness of measuring it. It does make it a pain to write, a pain to search, and a pain to explain. But that is why we did it.