Engineering Discipline in the AI Era with Dave Farley

About Dave Farley

Dave is a pioneer of Continuous Delivery, thought leader and expert practitioner in CD, DevOps, TDD, and software design, and shares his expertise through his consultancy, YouTube channel, books, and training courses. Dave co-authored the definitive book on Continuous Delivery and has published Modern Software Engineering. 

Dave has been having fun with computers for almost 40 years and has worked on most types of software, from firmware, through tinkering with operating systems and device drivers, to writing games and commercial applications of all shapes and sizes. He was an early adopter of agile development techniques, employing iterative development, continuous integration, and significant levels of automated testing, working in large-scale distributed systems.

AI is Bigger than the Internet, Bigger than Agile

I began writing code in a world that most developers today have never experienced: writing low-level assembler, working directly with memory allocators, squeezing performance out of early-generation PCs. Over the decades, I have witnessed nearly every major shift in the industry: the rise of object-oriented programming, the birth of the internet, the Agile movement, continuous delivery, DevOps, and now AI-assisted development.

I think, without really much shadow of a doubt, the change that we are seeing now is bigger than all of those put together. Bigger than the internet, bigger than object orientation, bigger than the agile transformation.

And I buy into the skepticism as well. It's a huge change, and it's not good enough at the moment.

‍
There are some really naive things that people think about applying AI to helping us build better software more quickly. And people are often looking in the wrong places for some of those wins.

Those who are practicing techniques like continuous delivery and modern software engineering find dramatic, remarkable changes in the rate at which they can produce better software and the quality of the software the scale of the software that they can build. 

That's because they've got good practices. That's because the natural way in which they think about working incrementally, getting feedback, learning, and adapting to what they learn is inbuilt already, and that's a natural fit for working in the chaos of an AI development process.

People that can't cope are working more traditionally, thinking in terms of big waterfalls,  long-term plans, and big changes; hiding their work on feature branches and building weeks- or months-long features in isolation. They just build worse software faster. 

The three problems of AI Coding

English is not a good programming language

Programming languages are tools that were designed to help us think. They allow us to express ideas concretely that a computer can execute, and they help us to compartmentalize problems, decomposing them into pieces that we can fit in our heads and reason about.

Natural language, a human language, is not as good as a programming language as a tool to help us think in that way. It's not precise enough. It's too vague. It's too open to interpretation.

AI is not deterministic

Programming languages are deterministic.  If I write a program and I give it the same input, get it in the same state, and give it the same inputs, I'm going to get the same results. That's not true if I'm talking to an AI. I will get different responses. I can give it exactly the same prompts or inputs, and it can do something different. 

There are ways in which I can try and lock down the AI to be less variable, but it's fundamentally part of how an AI works.  While this can be powerful, it changes the reliability characteristics of our tools.

Verification

When I write a traditional program, I know that if it compiles and the tests pass, I have a high degree of confidence in what it does. With AI, I don’t have that same certainty. It will sometimes claim that tests pass when they don’t. It will occasionally take shortcuts. So we need mechanisms to verify that what it produced is actually what we intended.

BDD: Behavior-Driven Development

The obvious way to address these challenges is to lean even more heavily on engineering discipline, particularly test-driven development and behavior-driven development. If I specify what I want in terms of testable, verifiable examples, then I have something concrete to measure against. If I describe a behavior—for example, searching for a book in a webshop, adding it to a cart, and completing checkout—that's a testable outcome. It’s not just a description. It’s something I can execute and verify. 

In an AI-driven world, verification becomes the bottleneck. Code generation is cheap. Understanding and validating behavior is the hard part.

‍That changes the nature of programming. Historically, much of our craft has been about defining solutions; choosing data structures, algorithms, control flow. Increasingly, I think our role will shift toward defining the problem in greater detail rather than hand-crafting the solution.

That doesn’t make this any less of a technical discipline. In fact, it may require more technical thinking. Decomposition, modularity, separation of concerns, cohesion—all of those ideas still matter. I don’t think the foundational principles of modern software engineering change one jot.

Spec-driven or behavior-driven?

Some of the spec-driven development folks seem to be saying or implying that you've got to come up with all the specifications upfront. And I think that's a mistake. I think that software development as a practice, whether with or without AI, is profoundly an exploratory process.

You need to incrementally explore a problem so that you can see it and understand the details that you  haven't thought of yet. You’re not going to get a specification right in blank.

Rather, you’ll specify something, interact with the system in some ways, learn from that, and then specify something new. Ideas like modularity, cohesion, separation of concerns, and testability are still important. Testability is deeply important because we want to be able to test the pieces that we're creating in isolation from each other and understand them.

We need models of how we're to think about those. We need to think about the shapes that we're painting, even if it's the AI that's actually filling in the colors. 

If the AI is writing the code and the tests, what are we doing? What are we bringing to the party? How do we know that it's giving us what it is that we want? There has to be some way in which we specify what it is that we want of the systems that we're working on.

Can Humans Review 12,000 lines of Code?

Test-driven development and acceptance test-driven development using BDD are essential because then we can say we'd like to achieve this outcome; we don't really mind too much what the solution looks like underneath as long as it achieves the outcome and it continues to achieve the outcome, so verification becomes the bottleneck. 

The scale of AI-generated code introduces a practical problem. I recently talked to Steve Yegge, and he said he produces 12,000 lines of code a day.

‍

I can’t read 12,000 lines of code carefully enough to feel that I truly understand and own them. That means we need a different mechanism for building trust. The trust has to come from executable specifications and continuous verification, not from manual inspection of every line.

The skill of programming is going to turn from using programming languages to building our own feature-specific programming language to express the specifications of what we want from the system. 

We're still going to have to think in technical terms. We're still going to have to decompose problems. It's just that the tools that we will use and the way that we think about this will change and mean that we can move more quickly.

Researching AI's Impact on Software Quality

I was very peripherally involved in academic research into the quality of software that AI presented. We wanted to look into how maintainable the code that AI wrote was. Researchers did a controlled experiment: they had a bunch of programmers who were asked to add a new feature to some very nasty Java code. And another group of programmers was asked to do the same, only with AI assistance. And then there was a third group who blindly took one of the outputs and were asked to do some more changes.

What was really surprising was that AI didn't really make any difference. It didn't make it any better, and it didn't make it any worse.

So I think that the scaremongers saying that AI can't code and it can't write systems are wrong. But I also think that people claiming that you get tens or hundreds of times more productivity are wrong too. There were some improvements in productivity, but it had more to do with the experience and the approach of the individuals.

The research said it was about 55% more improvement in productivity, which is not nothing. It's just not 10 times. 

The DORA research into AI came to similar conclusions, which is that if you're already working well, AI will be a big win. It's going to help you move faster, forward even faster.  But if you're working poorly, you're just going to dig a deeper hole faster. 

‍

‍