We built dev productivity tools before AI. Here’s what we’ve unlearned since

I started Aviator with a mission to provide Google-level developer productivity capabilities to the rest of the world. This was pre-LLMs, in the summer of 2021. As a team of former Googlers, we started looking for tools to fill the gap in the real world. Since Google builds everything in-house, not everything was easily replicable.

We had no idea that 5 years later the software engineering world would turn completely upside down. And that developer productivity would mean something entirely different. 

How we build software at Aviator now

From AI skeptics to parallel sessions, 40+ skills, and 20+ connectors

We were not the earliest adopters of AI. Even as late as the summer of 2024, the AI capabilities felt limited to smart auto-completes. We didn’t use Copilot for a while and didn’t use Claude Code until Apr 2025, and even then we were skeptical of the benefits. Even though it was broadly helpful and fun, most times the output was not right, and we had to fix it. It was clear that things would improve, and models would get better, but we had to carefully review the syntax, the logic, and the structure of any code generated by AI.

Today, we are not only running 5 parallel sessions with 40+ skills and 20+ connectors, but as a player in the developer productivity space, we are exploring where software engineering is headed.

Our engineering team is five engineers of various levels of seniority, but now they are producing work of 20+ pre-LLM engineers.

We are a heavy Claude shop, and every engineer gets access to the Max 5X plan. I’m surprised when I hear from folks who run out of 2+ Max 20x weekly limits. I guess these are either solo engineers working on greenfield projects or enterprise engineers whose codebase is so large and who have so many skills and MCPs that their baseline context is already large.

For us, the bigger issue with Claude has been reliability. Their outages impact us noticeably a few times every week, so much so that some of the engineers have now started using Codex as a backup. Unfortunately, we still have to pause our existing sessions and start new ones.

We do use Codex for reviews. Having a separate model critique the work of one helps catch issues better.

We also use git-worktrees very aggressively. I have 5 worktrees where I would have sessions running, that means at any given time I’m running at max 5 parallel things. I have not been able to go beyond that because context switching, even across 5 things, gets overwhelming.

Our standard code review skill is called “ofer-review,” after one of the engineers who created it. This skill creates a temporary worktree to check out a branch for review. We also have a separate skill that clears out old worktrees and branches as regular hygiene.

What’s in our mixed bag of skills 

Our skills can be broken down into a few categories:

Coding skills – Separate skills for backend (Python), frontend (TypeScript), and GraphQL design. Having separate skills helps keep our baseline context light. The skills are loaded based on the work being done. It’s common to use more than one skill for a coding task.

Debugging skills – Agentic debugging is now a critical part of our workflow.
Our product has two different installations (cloud and on-prem), so the debugging skills have different instructions for each of them. The logs are fetched from multiple sources. We have maintained great discipline with logs, with just enough to debug but also maintaining clear PII separation. 

We have one engineer “on call” every week, including me on the rotation. The on-call here is to respond to customers, not to manage infra issues. Our typical workflow is posting the question from the customer in Claude code, letting it debug, and then verifying the analysis before sharing with customers. If we find ourselves spending time doing back and forth or pushing back the agent, we also ensure that we update the skill to capture the learnings for next time.

I’ve noticed that agents are really good at finding workarounds to access the information; that means we had to lock it down completely from accessing production data. We had to remove access to kubectl (Kubernetes); it has only limited access to gcloud commands (including logs and metrics, no db access), and also it’s not allowed to execute arbitrary scripts without permissions.

Documentation skills – meant to create release notes, changelogs, and update public documentation. These are mostly style guides on what’s acceptable language for avoiding verbose AI tone and keeping information direct and simple.

Testing skills – There are multiple skills associated with testing. One is focused around the Playwright CLI to access the browser, navigate our app, and take screenshots. The other is focused around driving custom end-to-end testing, using a real GitHub repository, checking the database state and GitHub, and validating race conditions. We have a suite of libraries (sort of tools) that can perform operations like creating a PR, approving a PR, triggering CI, and pushing commits. The real beauty of the skill is that it can use these libraries to create just-in-time test plans based on the code change. So we don’t keep a formal set of tests that always run; instead, we create a test plan using these tools and then verify end-to-end using the skills. Sometimes that uses Playwright, but most times, APIs and DB snapshots are sufficient to validate the changes.

Review skill – this is the “ofer-review” skill that I mentioned.
It serves two main goals – help us understand what the code change is about (the intent and behavior) and then do adversarial review. The one place where we spend a lot of time refining the skill is on defining “what’s acceptable.” I find that most models are trained to find odd edge case issues, highly unlikely to ever happen in an actual application. In such cases, we ask the review agent to flag this as “Not proposing as comments.” These are just side notes worth reading, but not important enough to implement or even report as review comments. In a way, we are refining what a tradeoff means. As a startup, we don’t expect we will get hit by a million requests per hour, so we don’t need to build for such resilience. These things become acceptable risks.

There are also a couple of personal skills that help review the code. These skills help us understand the code without needing to read line by line. We can poke holes, confirm or invalidate our hypothesis, and in a way argue pros and cons of various approaches.

Some of those things we also do eventually check with the author, but more interestingly, we sometimes do this for our own PRs. This is realistically the main place where we are building our mental models, but without needing to read the code. Just a couple of skills, that’s all.

On human code review

Yes, we still do human reviews. Ironically, because one thing that probably got me the most attention in the engineering community was the article titled How to kill the code review.

We don’t expect a line-by-line review, but we don’t want everyone to rubber-stamp either. We find that depending on the complexity and ephemeral nature of the code, a code change requires different scrutiny. But, if we kill these reviews, we kill the only place where we have these debates and conversations, learn from each other, teach junior engineers, and build our mental models.

A team of 5 engineers can now do what 20+ engineers could do pre-AI, but the AI engineering space moves at insane speed, so, as a founder, I still write and review code. 

How do I review the code? It depends.

I don’t read any frontend code at all. We have ephemeral previews, so I’ll play around with the UI. We also have an agent that does the same and captures screenshots, most times that’s sufficient.

For a backend change, I use the skill to understand the core architecture and changes, I pressure test my assumptions and push back on complexity, and I review the intent and the expected behavior. I also look at what another AI agent says about the code change. And finally, have the agents perform an end-to-end verification to make sure that the change actually behaves the way it’s supposed to.

The only two roles that make sense now: product and systems engineer

We don’t hire backend and frontend engineers. This was the case for most early-stage startups, and it’s more true now. Realistically there are only two dimensions that make sense anymore: a product engineer and a systems engineer.

A product engineer is a person who owns the responsibility of understanding business needs and turning them into capabilities for the customers. This person also owns the outcomes. In a way, they are a blend of a PM, a designer, and the traditional product engineer. 

A systems engineer is what we used to call a platform engineer five years ago. They are responsible for building “the software factory,” or the tools that are used by the product engineers. They build the harnesses, manage the engineering context, and make sure that AI does not go off-track. Whether we call it “harness engineering,” “loop engineering,” or “context engineering,” they are all doing the same thing—enabling product engineers to use AI effectively. Even though there are thousands of products out there selling some versions of these capabilities, every company serious about AI efficiency would need a full-time team managing these capabilities.

Interviewing changed completely, and onboarding intensified

Bring your own agent to the interview

We had to change our entire interview process. If no one is writing the code by hand, the understanding of algorithms and logical reasoning becomes less important. What is left is:

• Ability to understand the requirements well
• Being able to provide the right guidance to navigate the agent
• Being able to choose the appropriate models and build the right skills for a task
• Ability to understand where AI goes off-track and correct it
• Ability to comprehend the code and understand the implications
• High agency to gather the requirements and show business impact

That means all of our interviews now allow candidates to bring their own coding agent and show us how to solve a given problem. The problems we give have gotten more complex and more ambiguous, because that’s where the true judgment can be assessed. We watch how candidates interact with their agents, who is in the driver’s seat—is it the agent driving the candidate or the candidate driving the agent? We then ask them to explain and critique the code produced by the agent. 

This is where an average candidate and a great candidate split up. If you can’t comprehend what an agent is producing even when you are running the agent collaboratively, how would you do that when we run the agents in the background and you are just looking at the final results?

I’ve heard some teams also interview for how many parallel agents you can run, what does your orchestrator looks like, and what your skills look like. I’m less bullish on the speed side of things. These are great skills to have, but these are things a smart engineer can learn, not an assessment of their own skills.

Onboarding: Figure it out with the agent

The process itself hasn’t changed much; just the intensity has increased. We would give a new hire a bunch of backlog tasks to tackle. These are intentionally meant to be a wide variety so they get to touch different parts of the system. The difference now is that we don’t handhold them much, rather we let them figure it out with the agent.

We start with simple tasks with very little ambiguity, and by the second week, we expect them to start picking up tasks that require critical thinking, may have multiple integration pieces, and might not have an obvious answer.

By the third week, they are also reviewing the code. I find this is quite a missed opportunity in a typical onboarding.

Having the new hires do review means that we are having someone look at our implementations from an outside perspective. Once they get into their daily routine, it’s difficult to bring in the outside perspective.

By the fourth week, we expect them to start participating in our regular sprints and work on critical projects with time sensitivity.

What I believed as a dev tool founder and how AI has changed it

I mentioned that we started Aviator in 2021, with the idea of bringing Google-level developer productivity to engineering teams of all sizes and various tech stacks. For years, Google has been making significant investments in engineering productivity via EngProd, but the rest of the tech industry only started to focus on it with the rise of remote work that came with COVID.

Our dream was not to treat developer productivity as something to be measured and tracked on a dashboard, so our approach was decidedly an anti-metrics approach. Developers back then were pretty clear about what makes them less productive – friction, so we wanted to focus on solving real developer problems, not just pointing at them.

Instead of building shiny dashboards, we focused on improving developer experience and built automated workflows across the entire SDLC—development, code reviews, builds, tests, and deployments.

So if everything in software engineering turned upside down, what stayed? Ironically, the same principles of better software engineering are still true.

Trunk-based development, continuous delivery, monorepos, small PRs (stacked PRs), and clear and accurate ownership. These were the principles we started Aviator with in 2021, and none of them have changed. The tools have, and the workflows have, but the processes and principles have remained the same.

Monorepos

One of the foundation principles that we built Aviator’s stack on is trunk-based development in monorepos.

“A monorepo setup helps establish consistent standards across projects by centralizing build configurations, linting rules, and development workflows. This standardization reduces cognitive overhead for developers moving between projects and ensures uniform quality controls. Having all code in one place also simplifies dependency management and cross-project changes.”

Many engineers rolled their eyes at the mention of monorepos—it was considered unscalable, unnecessary, or even an anti-pattern. 

With agents, monorepos become a lot more important. Since agents are great at searching through code, having all the code in one place makes it easy for them to find all references and understand cross-project dependencies. Having a single verification loop, standard tools, and conventions of monorepos make it easier for agents to navigate. We are also a monorepo, managing our backend, frontend, GraphQL, and REST interfaces all in a single repository.

On the flip side, with agents, the codebases are getting large quickly. Much of the Git tooling today isn’t built for that scale. 

Small reviews (Stacked PRs)

Google’s  Small CLs became the inspiration for Stacked PRs. This is what we wrote in our manifesto:
“Small, focused code reviews help maintain high code quality and developer productivity. By breaking changes into smaller, digestible pieces, reviewers can provide more thorough feedback and catch potential issues earlier. This approach also reduces cognitive load on reviewers and speeds up the review process, leading to faster iteration cycles.”

Reducing cognitive load is not something the industry is concerned about anymore, the number of parallel sessions and context switching is almost a source of pride for some engineers. The code review bottleneck created by the amount and size of AI-generated PRs is top of mind for basically every engineering org that has adopted AI coding tools.

There are mixed opinions on AI reviews in the industry. In our case, we find that they can still catch bugs, and edge cases that the coding agent might have missed. We just use a different model to review vs. the one for coding. Instead of paying for an AI reviewer tool, we have just set up a GitHub action that internally uses Codex to review the PR. This saves us a lot of cost.

However, in the last few weeks, we have been exploring this concept: what if we move the AI review before the PR is generated? If two agents are going to argue over the code change, why do we need to use a UI for it? And we are noticing that we would often read these AI comments and resolutions to understand the decisions. 

So, now instead of doing reviews in the UI, we are building a way to push these decisions to the UI. In a way, this provides us with the best of both worlds, saving the cycles back and forth where a human is not involved but also providing a space to have a debate.

We are doing the same for the coding agents: capture the decisions that the author (human) is making while implementing something interactively with the agent and post those in the UI to review. These decisions are acting as a proxy for the code itself, or rather, surfacing the more interesting nuances of a code change worth analyzing.

Old problems, new scale

Even though the principles remain the same, our tools are falling apart. I don’t need to restate the concerns around failing Git or GitHub, our overloaded CI, and rubber-stamped reviews, but we are collectively responsible for the world we live in. 

We got so used to the software delivery gates in the last 20 years that we forgot to question their value; we forgot to question whether there could be a better flow. And now that the software development velocity has increased, the cracks that only the companies operating at the top 1% of scale and complexity felt are seen even at teams of 10 engineers. 

I don’t believe that we will need fewer software engineers in the future; rather, we will need more engineers owning bigger systems. There will be no small systems when AI is the one doing the work. 

Runbooks: Our failed AI product

Everyone is building a software factory, so did we, as a spec-driven multiplayer framework called Runbooks. At least we tried. Runbooks failed.

Not because the product wasn’t useful, but because software factories require a lot of business context to be effective. They need to be customized to understand specific business needs, business context, and all the internal systems, and that requires months of intentional effort put in by a team of engineers to codify the business context, and define the knowledge graphs and the guardrails needed for these factories to succeed. This is a full-time job, not a plug-and-play that many vendors might claim.

Secondly, the models just became better. We started Runbooks in the summer of 2025, and by the end of 2025, when the product was ready for users, the models didn’t need detailed specs, they didn’t need tasks broken down in multiple steps; all they needed was a clear and current context. That was also a lesson in building products with layers that wouldn’t become obsolete with the next model update.

Where is software engineering headed?

I don’t know. But I still very much believe we still need strong software engineers to make the world a better place. And it’s still our responsibility to help grow the next generation of software engineers. The biggest bottleneck we have today is our own understanding of the last two decades of how software was built and continuing to explore what’s possible.

Today at Aviator, we accept that 90% (if not 100%) of the code is written by agents, and we build workflows around what parts get harder or stay harder when agents write all the code.

It’s easy to spin up dozens of agents to build the entire product without staring at the screen and typing, but understanding how things work, understanding what customers need, and bridging those gaps remain as hard as they always were. We don’t need to make developers faster at writing code, but we are far from making organizations confident in providing value to the customer.

If I were starting a dev tool startup today, I wouldn’t base it on bold beliefs about how software should be made; this is a time of change, and no one knows what software engineering will look like next year. But the principle of reducing toil for humans remains true with an extra layer of bridging the gap between humans and agents.