What we already measure
DORA and SPACE metrics aren’t enough
Measuring engineering productivity is a complicated process. Neither DORA nor SPACE capture the hidden pockets of developer productivity gaps.
Lack of context
DORA doesn’t reveal why your deployment frequency is low, or change lead time is increasing. It could be due to an issue you can solve, such as too much friction during code review, or slow, unreliable builds.
Changing metrics creates uncertainty
DORA is just data. It doesn’t tell you what to do when the metrics change. This causes uncertainty because developers aren’t informed of what happens when the change failure rate increases or deployment frequency slows down.
Developer pushback
Meaningful use of SPACE is dependent on developer participation. It’s designed to provide a true picture of what’s happening at both the individual and team level. If individuals aren’t contributing, then the insights you obtain will be unreliable
What we built
Measure “hidden pockets” of time wasted that are hard to measure using common engineering productivity metrics.
Aviator’s calculator focuses on how much time you and your team lose due to build and test failures in developer workflows. You can input data based on your GitHub activities and how you use GitHub branches.
M – PRs merged per day
X – mainline failures in a week
T – average CI time
F – flakiness factor %
Why this matters
It’s hard to get a full picture of how developers spend their time.
When we start adding those things up the numbers can be alarming.
For instance, consider the amount of time a developer spends debugging a flaky test trying to figure out if it failed because of their change or not. Or, the time spent by a developer who is trying to resolve a mainline build failure.
Although most of these impact the DORA metrics associated with Lead time for changes, we are still just scratching the surface in terms of measuring the inefficiencies in engineering team workflows. The impact of build and test failures also leads to delayed releases impacting the other DORA metrics like deployment frequency, time to restore service, and persistence of flaky tests in the system can lead to a higher change Failure rate.
What to do with this data
Today, many engineering organizations are scaling merge workflows without breaking builds using Aviator MergeQueue. Combined with a flaky test suppression system like Aviator TestDeck, teams are saving hundreds of engineering hours every week.
How it works
We ask you to input data based on your GitHub activities and how you use GitHub branches.
Based on these inputs, we estimate how much time your engineering team wastes weekly on managing build failures and dealing with flaky tests.