The Hidden Cost of AI-Generated Code: Cognitive Debt

About Margaret-Anne Storey

Dr. Margaret-Anne Storey is a professor of computer science and a Canada Research Chair in Human and Social Aspects of Software Engineering. Together with her students and collaborators, she seeks to understand how software tools, communication media, data visualizations, and social theories can be leveraged to improve how software engineers and knowledge workers explore, understand, analyze, and share complex information and knowledge. She has published widely on these topics and collaborates extensively with high-tech companies and nonprofit organizations to ensure real-world applicability of her research contributions and tools., and is one of the co-authors of the SPACE framework.

‍

The Origins of the SPACE Framework 

DORA was and still is a very successful framework for measuring the delivery of features. SPACE is really taking a step back from that and looking at a more holistic view of developer productivity and experience and trying to understand the space of things.

S is for Satisfaction. A lot of the research that I did in 2017 was about trying to understand what factors help developers feel satisfied and feel that their work is having impact. That research showed us that if developers don't feel satisfied, they're not as likely to feel or be as productive,  and vice versa. If you are productive in your work, you are also more satisfied with your work.

P is for Performance. This is about really focusing on the outcomes or the quality of the work that you're doing. If you're thinking about a company, you can be measuring many activities or countless features being shipped,  but is that actually leading to performance or outcomes at the company or the end-user level?

A is for Activity. This means things like writing lines of code, and number of PRs. They can be useful signals, but they don't necessarily do a good job of measuring developer productivity because software development is not a factory.

C is for Collaboration. A lot of software development projects are a team sport. C looks at how, in your work, you can collaborate and coordinate with others; what kind of feedback you're getting from others, and how much time you're spending helping others in their work.

E is for Efficiency and Flow. I wish flow started with an E because it really is more about that flow. You can think of E as being about the flow of the work, the efficiency of the work. But to me, it also represents the experience that developers have of that flow experience as they're being creative and innovative, which has a connection to their satisfaction.

How AI Is Changing Productivity Frameworks

Everything has changed. All aspects of our work. I think SPACE still stands up pretty well because the dimensions are quite broad. But the ways we ask questions within each of those dimensions, and the things that we look for and the ways we look for them, have shifted a lot.

With collaboration, we still don't have much insight on how interacting with agents is affecting how people collaborate. We did an ethnographic study where we followed people around and watched what they did. The juniors are more likely going to be using AI tools to ask questions first.  It's not brand new that there are resources out there that you can turn to instead of talking to humans. But I think now tools like Claude and Cursor give better answers, and so the juniors are maybe trusting those answers and then not going to the seniors.

That impacts a lot because the relationships aren't built up as well. The seniors now don't know what the junior was struggling with, whereas before they would know those things.

And of course, as each of the developers on a team engages increasingly with agents, they're becoming almost like little managers of the agents they're working with. They're now interacting with the other humans perhaps more in a manager-to-manager capacity rather than as creators or problem-solvers.

‍

What Is Cognitive Debt?

I started using the term "cognitive debt" last October. I was teaching an entrepreneur startup course at the university with business students and computer science students, and this year I was really trying to encourage them to use AI to go faster. I knew that was a risk — that they could lose control a little bit of the code they were writing and the architecture they were building. But in an entrepreneurial situation, you want to go fast. You want to get something into the hands of users and figure out: am I building the right thing? And the fastest way to do that is to have a prototype.

That worked really well. They were building things really fast, putting it into the hands of their users in a way that I've never seen before. But then I started hearing things from the teams. One of the students wanted a change, and the others said, "We can't make that change." I said, "What? Of course you can make that change. Why can't you?"

‍At first I thought they had technical debt, messy code.

But as I started talking to them, I realized it's not about technical debt. It's about the fact that they had lost the plot. They had lost track of what features they were really trying to build and why. They didn't even really know who knew what on the team.

Worse, on one of the teams, it was one person who knew all the code and understood it because they were supervising the AI that generated it, and the rest of the team was unable to do that. And the person who had generated the code didn't really understand what was generated either. So when a change was requested, they were like, "No, I can't really make that change. Every time I try to change something, something else breaks."

‍

That's when I came up with a canvas, a worksheet for my students to work through and say, "Okay, where are you on this prototype-to-product journey?" How much technical debt do you have and how much cognitive debt?

I define cognitive debt as being that understanding of what the system is doing and why, and who knows what across the team. 

‍Peter Naur wrote a paper many years ago that software isn't about the code; it's about the theory that's in developers' heads. The code is less important. It's the theory of what we all together in the team have in our heads that is really the valuable asset. That knowledge is about how we map from the world to the code, how the system is designed, what the architecture looks like, and what the capacity for change is.

The Definition of Intent Debt

Because they were moving so fast with writing code and deploying their software apps, they were losing track of how the features they were building related to user needs. Basically, what they were building was drifting from what they were intending to build at the beginning.

Intent debt is the "Why are we doing this?" Whose problem are we trying to solve? What is the important information? What are the constraints that we need to know about? And what is it also that we're not building?

Why Does AI Make Cognitive Debt Worse?

AI is providing a solution to solving some of it. It's a great way of explaining how some of the code works. It's a great way of sharing knowledge across the team. You could even imagine having an agent that is keeping track of who knows what and who's an expert in any particular system.

The problem lies with cognitive debt if you don't know to ask those questions. If you don't have awareness in your team of who's the go-to when we need to understand a particular part of the system? Do I know what you know? Do you know what I don't know? That's where cognitive debt accrues over time.

I don't use it in the way of describing "my cognitive debt." It's more a property of the system, a property of the project. 

I heard one person say that sometimes it scares him that he created this system and it works and it's running, and he's forgotten what it was he was even trying to create in the first place. It's that loss of control, that loss of memory across the system about what questions to even ask anymore.

When you know that you have a blind spot, you check it when you're in the car. But it's the blind spots you don't know about that you don't check. If you don't know that there are dependencies, maybe hidden dependencies or hidden security flaws, you're not checking for those. And that's where cognitive debt can become a problem.

How to Measure Cognitive and Intent Debt?

The answer is I'm working on it. What I'm worried about is that organizations and teams will rush to create metrics. We have that tendency in software engineering. We are engineers; we like crisp metrics. We like to believe that those metrics represent the things that we care about and that we have control over.

Being careful at developing —  I won't even say the word metrics — but signals of when we have cognitive debt and when we have intent debt is more important at the beginning than trying to find a number.

What I'm trying to do right now is listen a lot to practitioners. I'm trying to look at the different tools and solutions that are being built to address these problems. What is the problem that those tools are trying to address? And then what is the best way to arrive at diagnostic signals of having problems with cognitive debt and intent debt?

Retrospectives Now Make Even More Sense

What I'd like to push more is having the team sit down in retrospectives and ask more pointed questions. Let's throw up an architecture diagram. Who knows how this overall architecture works? Whose name can we put beside the different subsystems or components? Who's the person who really grasps that? How do we map our features to our user needs? Are we doing a good job of that? Are we really evaluating whether our features are addressing those user needs? And when is the last time that we checked?

We've always had challenges with comprehension — program comprehension and team cognition across the team in development. Agile was one of the practices that came out to try to repair that understanding across the team, but also to repair lack of understanding of user needs. And losing track, losing our way, building the wrong software system, that's all been around a long time.

I think what's different now is that AI is an amplifier. It's amplifying the problems from poor understanding across your team. And it's amplifying because you're going so fast, it's easy to lose track of what it is that your users need.

The Need for Slowing Down

I think in software development, we don't think about the team enough. We don't think about how people are feeling. We don't realize that when we support developers and they're not exhausted — which is what's happening with AI right now; we're hearing a lot about AI fatigue — when we slow down, that's often when we do our most creative work.

Software development is not a factory where we're trying to squeeze out every bit of capability out of the developers. It's more about problem solving. That's what excites me about developing software! I think about a problem that a user has, and then I try to come up with the best solution.

And if I'm exhausted and just trying to go fast and manage all these agents and review all this code that I didn't write, or nobody else even wrote, that an agent wrote, I'm less likely to have those creative insights and to innovate.

Some organizations are starting to realize that and starting to introduce intentional friction. Let's slow down. Take some days off, go to the beach, go in nature. That's when your best ideas come — when you take time off.

‍