Scaling monorepos with MergeQueue and Nx

Today we’re happy to announce that affected targets now supports Nx.
CEO @ Aviator

Scaling monorepos with MergeQueue and Nx

Today we’re happy to announce that affected targets now supports Nx.

Recently, tooling advancement has led to the proliferation of monorepo usage among large teams. Humbly, we also wrote about the benefits of using a monorepo. One of the more popular build tools for managing Javascript monorepos is Nx. Today we’re happy to announce that affected targets now supports Nx.

Nx is an open-source build system optimized for monorepos that comes with built-in tooling and advanced CI capabilities. Nx platform is language agnostic and can be used with any language and framework. The native Nx plugins support most of the popular frontend frameworks like React, Vue, node.js, Angular, NextJS, and have recently added support for non-Javascript languages such as Java, C#, Go, etc.

Affected targets in Nx

Local and remote caching is one of the primary benefits of using a build tool like Nx. Remote caching allows users to break down a large codebase of a monorepo into both components and their dependencies. During the initial code build, builds are cached for these components remotely. Subsequent rebuilds leverage this remote cache, the build system smartly pulls the dependencies from remote servers, and only compiles the code that has changed.

To calculate the targets affected between two Git SHAs, Nx provides a handy CLI tool called affected:

nx show projects --affected --base main --head feature/my_feature

This will show all the projects that are affected between the feature branch and the main branch. You can read more about how to fetch affected targets using nx CLI in their documentation.

MergeQueue for monorepos

If you are unfamiliar with merge queues, here’s a good primer. The typical modes of MergeQueue work well in a small to medium-sized repository. Still, once you go past a certain size (we estimated about 100 changes a day with an average CI time of about 30 minutes), the queue can become sluggish.

Now imagine a team of 1000+ developers working in a monorepo. Even when using batching and parallel mode, a single failure would reset the entire optimistic queue, causing merges to slow down. This is where affected targets can play a big role. Using the knowledge of affected targets, you can split the queue into multiple disjoint queues:

This way a failure in one queue is localized and does not impact other queues running completely independently.

Passing Nx targets to MergeQueue

To pass the affected targets to MergeQueue automatically, we can build a quick GitHub action. We will use the following steps to generate a GitHub action:

  • Fetch the base SHA and head SHA of the PR
  • Use Nx CLI to pull affected targets based on these SHAs
  • Send this information to Aviator’s server to be used for queue selection.

Fetch the base SHA and head SHA

GitHub provides the head SHA information but not the base SHA. Instead, Aviator MergeQueue reads directly from the GitHub API. For this, we use GitHub API request action:

- uses: octokit/request-action@v2.x
  id: get_pr
  with:
    route: GET /repos/${{ github.repository }}/pulls/${{ github.event.number }}
    owner: octokit
    repo: request-action
  env:
    GITHUB_TOKEN: ${{ github.token }}

Note that this API call only works on pull_request workflow events:

on:
  pull_request:

This will give you the entire payload of pull_request as provided by GitHub Rest API. Now, to fetch the base SHA, we can use fromJSON:

${{ fromJson(steps.get_pr.outputs.data).base.sha }}

The head SHA can be easily referenced using: ${{ github.sha }}.

It’s important to note that to compare base and head SHAs, we need to do a partial clone when checking out the GitHub repository:

- uses: actions/checkout@v4
  with:
    filter: 'blob:none'
    fetch-depth: 0

Use Nx CLI to pull affected targets

The next step is to pull the affected targets for these changes. Nx CLI makes fetching the targets extremely easy. We just need to pass the base and head commit SHA to the CLI:

nx show projects --affected \\
--base ${{ fromJson(steps.get_pr.outputs.data).base.sha }} --head ${{ github.sha }}

These are the targets that were impacted by all changes collectively between the base SHA and the head SHA. This will print out a list of affected targets, one in each line. Since we will use this list to be sent as JSON to the Aviator API, we can fetch this in JSON format using the --json flag.

Send the targets to Aviator

The last step is to send the target information to the Aviator using the Aviator API. To keep it simple, we will just use curl here:

curl -X POST -H "Authorization: Bearer ${{ secrets.AVIATOR_API_TOKEN }}" \\
  -H "Content-Type: application/json" \\
  -d '{
    "action": "update",
    "pull_request": {
        "number": '"$PR_NUMBER"',
        "repository": {"name": "nx-examples", "org": "aviator-co"},
        "affected_targets": '"$TARGETS_JSON"'
    }
  }' <https://api.aviator.co/api/v1/pull_request/>

To authenticate with the Aviator API, you will need to set AVIATOR_API_TOKEN in GitHub secrets. It’s important for security reasons to not hard code the token in your actions code. Once targets are sent to Aviator you should see the targets also updated in the GitHub comment of Aviator status:

Now, when a developer queues a PR, Aviator will automatically assign the PR to a separate queue based on these targets. Using this concept, you can scale your MergeQueues to thousands of daily merges without breaking the builds.

You can also find this GitHub action code in the nx-examples repository fork, and read the Aviator Nx documentation for further details.

Aviator.co | Blog

Subscribe

Be the first to know once we publish a new blog post

Join our Discord

Join us for a workshop on Mastering Releases

Uncover key release challenges and master the best practices for safe releases. When: Friday, Oct 18th, 10am PDT