Blue-green deployments and rollbacks with Spinnaker

Spinnaker is an open-source continuous delivery platform. Build workflows to avoid any downtimes while making critical changes to your services using Spinnaker.
I like to solve technical infrastructure issues ;)

Blue-green deployments and rollbacks with Spinnaker

Spinnaker is an open-source continuous delivery platform. Build workflows to avoid any downtimes while making critical changes to your services using Spinnaker.

Managing deployments and rollbacks is key to achieving continuous delivery and minimizing downtime in DevOps workflows. One approach that has recently gained popularity is blue-green deployments. This method involves switching between two identical environments — one serving as the active production environment and the other as the inactive backup — as new code is deployed. 

However, manually managing these deployments can be complex and time-consuming. This is where Spinnaker comes in: Spinnaker is a powerful open-source, multi-cloud continuous delivery platform that automates your deployment processes and helps you easily manage your infrastructure. It provides a user-friendly interface and allows you to create automated deployment pipelines. And the best part? It supports various cloud providers like AWS, Azure, and GCP. So no matter where your infrastructure resides, Spinnaker has got you covered. 

This blog post will provide a step-by-step guide to setting up Spinnaker for blue-green deployments, defining blue-green deployment pipelines, and executing blue-green deployments. In it, we’ll also discuss how Spinnaker supports managing rollbacks efficiently, and we’ll share best practices for optimizing deployment workflows and avoiding common pitfalls.

Understanding blue-green deployments

A blue-green deployment strategy entails maintaining two identical production environments, cleverly named blue and green. One side is active and visible to users, and the other is idle. The active environment (blue) serves the current live version of the application. The idle one (green) is where you can safely deploy and test the new version. 

When a new version is ready, you deploy it to the green environment — while the blue environment is still live and serving the current version of the application to users. The QA team can then test the new version in the green environment, so you can catch and fix any bugs or issues before they reach users. 

Once the new version in the green environment is deemed ready, you switch the load balancer to redirect traffic from the blue environment to the green one. Users are seamlessly transitioned to the new version of the application with zero downtime. Now the blue environment becomes idle and serves as a safety net. If you encounter any issues with the new version, you can quickly switch back to the blue environment, effectively rolling back to the previous version.

While blue-green deployments enable seamless transitions and easy rollbacks, there are a couple of catches: You can’t route the latest version to specific users: the switch from blue to green happens for all users at once. Maintaining two identical production environments also doubles your infrastructure and resource needs. You could spin down the idle environment between deployments, but managing two parallel production environments and ensuring seamless data synchronization can add significant complexity to the deployment process and requires sophisticated infrastructure management and tooling.

This is where Spinnaker plays an important role.

Overview of Spinnaker

In the introduction, we described Spinnaker as an open-source, multi-cloud continuous delivery platform, which is a mouthful, so let’s unpack that statement. This means that Spinnaker can deploy to any cloud provider you use: AWS, Google Cloud, or Kubernetes. You can use it to automate delivery to all of the environments you use throughout the software development lifecycle, from test to stage to production. 

Spinnaker provides essential insight into an application’s health and helps drive the deployment and scaling decisions you’ll make throughout the application’s life. Spinnaker does this by abstracting away some of the common cloud infrastructure you can find throughout these providers, such as load balancers, security groups, and server groups. 

Server groups are just an abstraction on top of your application running in one of these cloud providers, so in AWS, that would be an autoscale group made up of EC2 instances; in Kubernetes, that would be a deployment or replica set made up of pods. One of the most incredible things Spinnaker provides is this robust pipeline framework that you can use to build out any deployment workflow you want. This works because you can compose a series of stages representing various steps throughout your deployment pipeline. Let’s say you’ve pushed code to GitHub. Next, you want to have Jenkins build your artifact — you can have Spinnaker pick up that artifact, bake it into an AMI, and then deploy that AMI into AWS. 

All of these stages have been battle-tested and built out so that your application stays available, and some of the biggest companies are already using Spinnaker to do this. 

Now that we’ve provided an overview of Spinnaker, we’ll explain how to implement a blue-green deployment strategy.

Implementing blue-green deployments with Spinnaker

You can configure Spinnaker to dynamically route traffic to particular applications, which can then be used to configure blue-green deployments.

Start by creating a pipeline with two stages: deploy and disable. The process is simple. First, you deploy your new application version and wait for all the health checks to pass. Once that’s done, you’ll disable the old version of the running application. Then the pipeline will look like this:

spinnaker ui

Before you get started with the implementation, there are some prerequisites:

  • A Spinnaker with Kubernetes provider configured — if you are new to Spinnaker, check out the installation guide.
  • A running service inside your cluster and namespace that your pipeline will deploy to — as an example, we are naming this service blue-green-service, and the specifications for this service are:
kind: Service
apiVersion: v1
metadata:
  name: blue-green-service
spec:
  selector:
    frontedBy: blue-green-service # will be applied to backends by Spinnaker
  ports:
  - protocol: TCP
    port: 80

1. Setting up the deployment stage

With these prerequisites in place, you can create a pipeline, add the application deployment stage, and configure the manifest below to deploy:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  annotations:
    strategy.spinnaker.io/max-version-history: '2'
    traffic.spinnaker.io/load-balancers: '["service blue-green-service"]'
  labels:
    tier: frontend
  name: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      tier: frontend
  template:
    metadata:
      labels:
        tier: frontend
    spec:
      containers:
        - image: 'us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5'
          name: frontend

There are two things you should pay attention to here:

spinnaker config

  1. traffic.spinnaker.io/load-balancers tells Spinnaker which service to attach during the deployment stage and which service to detach during the disabled stage.  
  2. strategy.spinnaker.io/max-version-history ensures that no more than the mentioned number of versions of these replica sets are available for garbage collection.

Now that the deployment stage is set up, you can move on to configuring the disable stage in the pipeline. 

2. Setting up the disable stage

This step is easier than the deployment stage. All you have to do is add the disable step after the deployment step with the following values:

FieldValue
Namespacedefault
KindreplicaSet
SelectorChoose a target dynamically
ClusterreplicaSet frontend
TargetSecond Newest

The configuration window should look something like this:

spinnaker manifest

That’s all — your blue-green strategy is set up and can be used for the next deployment.

But now that you have your blue-green pipeline set up to take care of your deployments, what happens if an application fails? How do you respond if customers experience issues on your website? This is when Spinnaker’s most powerful feature comes to the rescue.

Managing rollbacks with Spinnaker

Spinnaker provides a quick way to roll back our applications in case of issues or failures. The rollback feature will simply disable the version that you’re rolling back from and re-enable the version we are rolling back to.

Step 1: Click on the version of the application you want to roll back from 

You’ll see a list of all running applications on the cluster screen. From here, you have to click on the version of the application you want to roll back from.

spinnaker rollback start

In our example, we’re clicking on V001. 

Step 2: Select Rollback from the Server Group Actions menu

From the Server Group Actions menu, select the Rollback option.

spinnaker rollback

Step 3: Select the version you want to roll back to

In the next popup menu, select the version you want to roll back to and enter a reason for the rollback (entering a reason is optional).

rollback spinnaker blue green

Step 4: Click on Submit

That’s all!! 

Spinnaker will now start running a bunch of tasks in order to get you to the rollback state. In a couple of minutes, Spinnaker will disable the version you rolled back from and enable the version you rolled back to. 

Tips for improving your deployment

Now that you know how to configure a blue-green deployment and manage rollback, take a look at these tips for improving your deployments:

  • Automation: Use automation in the deployment pipeline wherever possible. Most importantly, the testing should be automated. Set up the testing frameworks in your pipeline to cover all types of test cases. 
  • Monitoring and alert setup: Monitoring should be a part of your process, to ensure that everything with respect to your deployment and application is working perfectly. To do this, setting up a monitoring dashboard for Spinnaker is critical. 
  • Health checks: Set up health checks for both of the application environments in the blue-green pipeline, to ensure that the application is working perfectly all the time.
  • CO (continuous optimization): Don’t stop after your pipeline is set up. Over time, you can monitor feedback and other data to find ways to improve efficiency — by optimizing performance metrics, configurations, and allocated resources.

Conclusion

In this blog post, we described blue-green deployment and how Spinnaker, an open-source, multi-cloud continuous delivery platform, makes setting up your deployment process easy. We also explained how to set up a blue-green pipeline on Spinnaker and how to manage rollbacks using Spinnaker’s integrated rollback feature. Lastly, we covered best practices for optimizing your pipelines. We  hope you’ve found this article insightful and valuable.

aviator releases

Aviator.co | Blog

Subscribe

Be the first to know once we publish a new blog post

Join our Discord

Join us for a workshop on Mastering Releases

Uncover key release challenges and master the best practices for safe releases. When: Friday, Oct 18th, 10am PDT