{"id":2226,"date":"2024-06-05T00:15:02","date_gmt":"2024-06-05T00:15:02","guid":{"rendered":"https:\/\/www.aviator.co\/blog\/?p=2226"},"modified":"2025-08-19T12:57:52","modified_gmt":"2025-08-19T12:57:52","slug":"how-to-manage-rollouts-and-rollbacks-using-flux-cd","status":"publish","type":"post","link":"https:\/\/www.aviator.co\/blog\/how-to-manage-rollouts-and-rollbacks-using-flux-cd\/","title":{"rendered":"How to manage rollouts and rollbacks using Flux CD"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2-1024x576.png\" alt=\"\" class=\"wp-image-2227\" title=\"rollouts-fluxcd\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2-1024x576.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2-300x169.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2-768x432.png 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2-1536x864.png 1536w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2-2048x1152.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Controlled deployments allow teams to minimize disruptions and ensure that new releases improve an application and don\u2019t introduce problems. They allow you to gradually expose new features to users, and give you room to monitor and validate performance, before a full-scale rollout.<\/p>\n\n\n\n<p>This article will walk you through the essentials of rollouts and rollbacks. It will introduce Flux CD and explain how to implement both rollouts and rollbacks with Flux CD \u2014 covering setup and configuration, and including an example of a canary deployment strategy.<\/p>\n\n\n\n<p>It will also explain how to integrate Flagger with Flux CD, to further enhance deployment strategies, especially for progressive delivery.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prerequisites<\/strong><\/h2>\n\n\n\n<p>Before starting, make sure you have the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A Kubernetes cluster.<\/li>\n\n\n\n<li>kubectl installed and configured to interact with your cluster.<\/li>\n\n\n\n<li>A Git repository for storing Kubernetes manifests.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Setting up Flux CD<\/strong><\/h2>\n\n\n\n<p>First, you\u2019ll install Flux CLI:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -s https:\/\/fluxcd.io\/install.sh | sudo bash<\/code><\/pre>\n\n\n\n<p>Flux needs your GitHub credentials to log in and perform some actions on your repository. Export your GitHub personal access token and username:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>export GITHUB_TOKEN=&lt;your-token&gt; #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\nexport GITHUB_USER=&lt;your-username&gt;<\/code><\/pre>\n\n\n\n<p>The flux bootstrap GitHub command sets up the Flux controllers on a Kubernetes cluster and sets them to synchronize the cluster\u2019s state with a Git repository. It also uploads the Flux manifests to the Git repository and sets up Flux CD to automatically update itself based on changes in the Git repository.<\/p>\n\n\n\n<p>To do this, run the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo $GITHUB_TOKEN | flux bootstrap github \\\n&nbsp;&nbsp;--owner=$GITHUB_USER \\\n&nbsp;&nbsp;--repository=&lt;github-repo-name&gt; \\\n&nbsp;&nbsp;--branch=main \\\n&nbsp;&nbsp;--path=.\/flux-cluster \\\n&nbsp;&nbsp;--personal\n&nbsp;&nbsp;--private=false<\/code><\/pre>\n\n\n\n<p>The command above does the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Creates a Git repository, <code>&lt;github-repo-name&gt;<\/code> (in this example, <code>roll-flux<\/code>), on your GitHub account.<\/li>\n\n\n\n<li>Adds Flux component manifests to the repository.<\/li>\n\n\n\n<li>Deploys Flux components to your Kubernetes cluster. You can run <code>kubectl get all -n flux-system<\/code> to check out the components.<\/li>\n\n\n\n<li>Configures Flux components to track the path <code>\/flux-cluster<\/code> in the repository.<\/li>\n\n\n\n<li>Uses the <code>--private=false<\/code> flag to create a public repository.<\/li>\n<\/ul>\n\n\n\n<p>Now clone the repository you created (in my example, roll-flux) to your local machine:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>git clone git@github.com:$GITHUB_USER\/roll-flux.git\ncd roll-flux<\/code><\/pre>\n\n\n\n<p>Now run the following to create a<a href=\"https:\/\/fluxcd.io\/flux\/components\/source\/gitrepositories\/\"> GitRepository<\/a> manifest pointing to<a href=\"https:\/\/github.com\/khabdrick\/roll-flux\/\"> the https:\/\/github.com\/khabdrick\/roll-flux\/<\/a> main branch.&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>flux create source git nginx \\\n&nbsp;&nbsp;--url=https:\/\/github.com\/khabdrick\/roll-flux \\\n&nbsp;&nbsp;--branch=main \\\n&nbsp;&nbsp;--interval=2m \\\n&nbsp;&nbsp;--export &gt; .\/flux-cluster\/rollflux-source.yaml<\/code><\/pre>\n\n\n\n<p>In the command above:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A Git repository named <code>nginx<\/code> is created.<\/li>\n\n\n\n<li>The <code>source-controller<\/code> checks the Git repository every two minutes, as indicated by the <code>--interval<\/code> flag.<\/li>\n\n\n\n<li>It clones the main branch of the<a href=\"https:\/\/github.com\/khabdrick\/roll-flux\/\"> https:\/\/github.com\/khabdrick\/roll-flux\/<\/a> repository.<\/li>\n\n\n\n<li>When the current Git repository revision differs from the latest fetched revision, a new artifact is archived.<\/li>\n<\/ul>\n\n\n\n<p>After the command is run, you should have the corresponding file <code>rollflux-source.yaml<\/code>.<\/p>\n\n\n\n<p>Now deploy the roll-flux application using GitOps. Configure Flux CD to build and apply the<a href=\"https:\/\/github.com\/khabdrick\/roll-flux\/tree\/main\/kustomize\"> kustomize<\/a> directory located in the roll-flux repository.&nbsp;<\/p>\n\n\n\n<p>Use the following <code>flux create<\/code> command to create a<a href=\"https:\/\/fluxcd.io\/flux\/components\/kustomize\/kustomizations\/\"> Kustomization<\/a> that applies the roll-flux manifest files:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>flux create kustomization nginx \\\n&nbsp;&nbsp;--target-namespace=default \\\n&nbsp;&nbsp;--source=quotes \\\n&nbsp;&nbsp;--path=\".\/kustomize\" \\\n&nbsp;&nbsp;--prune=true \\\n&nbsp;&nbsp;--wait=true \\\n&nbsp;&nbsp;--interval=10m \\\n&nbsp;&nbsp;--retry-interval=2m \\\n&nbsp;&nbsp;--export &gt; .\/flux-cluster\/nginx-kustomization.yaml<\/code><\/pre>\n\n\n\n<p>Add the manifest to your Git repository:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>git add -A\ngit commit -m \"Add Kustomize settings\"\ngit push origin main<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding rollouts<\/strong><\/h2>\n\n\n\n<p>A rollout is the process of deploying a new version of an application to your Kubernetes cluster. Flux CD uses kustomize or helm controllers to manage rollouts.<\/p>\n\n\n\n<p>Let\u2019s say we have a simple Nginx deployment. We\u2019ll create a<a href=\"https:\/\/github.com\/khabdrick\/roll-flux\/blob\/main\/kustomize\/deployment.yaml\"> file for our deployment<\/a> and the<a href=\"https:\/\/github.com\/khabdrick\/roll-flux\/blob\/main\/kustomize\/kustomization.yaml\"> kustomization.yaml<\/a> file in the <code>.\/kustomize<\/code> directory. Take note of the file path in the<a href=\"https:\/\/github.com\/khabdrick\/roll-flux\"> repo<\/a>.<\/p>\n\n\n\n<p>The deployment file we have in the repository contains a Kubernetes <code>Service<\/code> and a <code>Deployment<\/code> for an Nginx application. The <code>Service<\/code> of type <code>LoadBalancer<\/code> named <code>nginx<\/code> directs incoming TCP traffic on port 80 to the pods labeled <code>app: nginx<\/code> on the same port, facilitating external access to the Nginx application. The <code>Deployment<\/code> named <code>nginx<\/code> specifies running three replicas of the Nginx container with version <code>1.19.2<\/code>.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Performing a rollout<\/strong><\/h3>\n\n\n\n<p>To perform a rollout, update the image version in your Kustomize configuration and push the changes to your Git repository:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Kustomize\/deployment.yaml\n...\n&nbsp;&nbsp;&nbsp;spec:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;containers:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- name: nginx\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;image: nginx:1.19.3\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ports:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- containerPort: 80<\/code><\/pre>\n\n\n\n<p>Commit and push the changes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>bash git add base\/deployment.yaml\ngit commit -m \"Update nginx to 1.19.3\"\ngit push origin main<\/code><\/pre>\n\n\n\n<p>Flux will automatically sync the changes and deploy your application. You can check by running the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl -n default get deployments,services<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" width=\"1047\" height=\"166\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/image.png\" alt=\"\" class=\"wp-image-2229\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/image.png 1047w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/image-300x48.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/image-1024x162.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/image-768x122.png 768w\" sizes=\"(max-width: 1047px) 100vw, 1047px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding rollbacks<\/strong><\/h2>\n\n\n\n<p>A rollback is the process of reverting to a previous version of an application when a new deployment fails or causes issues. Flux CD makes rollbacks straightforward by allowing you to undo changes in your Git repository.<\/p>\n\n\n\n<p>If the new version of Nginx (<code>1.19.3<\/code>) causes issues, you can easily roll back to the previous version (<code>1.19.2<\/code>) by undoing<strong> <\/strong>the change, or reverting to the previous version, in your Git repository:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>git revert &lt;commit-hash-of-update&gt;\ngit push origin main<\/code><\/pre>\n\n\n\n<p>Flux CD will automatically detect the rollback in your Git repository and apply the previous version (<code>1.19.2<\/code>) to your Kubernetes cluster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Advanced rollout strategies with Flux CD<\/strong><\/h3>\n\n\n\n<p>Flux CD supports advanced rollout strategies through integration with Flagger. Flagger provides automated canary deployments, A\/B testing, and blue\/green deployments.<\/p>\n\n\n\n<p>In this section, we will use Flagger to create a<a href=\"https:\/\/fluxcd.io\/flagger\/usage\/deployment-strategies\/#canary-release\"> canary deployment<\/a> for the Nginx application and gradually shift traffic to the new version while monitoring metrics. If the new version meets the specified thresholds, the canary becomes the primary deployment.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Canary deployment with Flagger<\/strong><\/h4>\n\n\n\n<p>First, install Flagger with the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl apply -f <a href=\"https:\/\/raw.githubusercontent.com\/fluxcd\/flagger\/main\/artifacts\/flagger\/crd.yaml\">https:\/\/raw.githubusercontent.com\/fluxcd\/flagger\/main\/artifacts\/flagger\/crd.yaml<\/a><\/code><\/pre>\n\n\n\n<p>Create a canary resource:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apiVersion: flagger.app\/v1beta1\nkind: Canary\nmetadata:\n&nbsp;name: nginx\nspec:\n&nbsp;targetRef:\n&nbsp;&nbsp;&nbsp;apiVersion: apps\/v1\n&nbsp;&nbsp;&nbsp;kind: Deployment\n&nbsp;&nbsp;&nbsp;name: nginx\n\n&nbsp;service:\n&nbsp;&nbsp;&nbsp;port: 80\n\n&nbsp;analysis:\n&nbsp;&nbsp;&nbsp;interval: 1m\n&nbsp;&nbsp;&nbsp;threshold: 10\n&nbsp;&nbsp;&nbsp;metrics:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- name: request-success-rate\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;threshold: 99\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;interval: 1m\n\n&nbsp;&nbsp;&nbsp;webhooks:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- name: load-test\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type: pre-rollout\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url: http:\/\/flagger-loadtester.test\/\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;timeout: 30s<\/code><\/pre>\n\n\n\n<p>Push to GitHub:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>git add kutomize\/deployment.yaml&nbsp;\ngit commit -m \"added flagger to Kustomize\"\ngit push origin main<\/code><\/pre>\n\n\n\n<p>This YAML configuration defines a Flagger canary resource for the <code>nginx<\/code> deployment in the <code>default<\/code> namespace. It specifies that Flagger should monitor the <code>nginx<\/code> deployment (with API version <code>apps\/v1<\/code>) and expose it on port 80.<\/p>\n\n\n\n<p>The canary analysis will run every minute (<code>interval: 1m<\/code>) for a total of 10 iterations (<code>iterations: 10<\/code>) and allow a maximum of 10 failed checks (<code>threshold: 10<\/code>). Each iteration checks whether the <code>request-success-rate<\/code> metric meets or exceeds the 99% success rate (<code>threshold: 99<\/code>) over a one-minute interval (<code>interval: 1m<\/code>). Additionally, a pre-rollout webhook (<code>type: pre-rollout<\/code>) is configured to run a load test via the specified URL (<code>http:\/\/flagger-loadtester.test\/<\/code>) with a timeout of 30 seconds (<code>timeout: 30s<\/code>). If the canary deployment fails to meet the specified metrics or the webhook test, Flagger will rollback to the previous stable version.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Canary deployment analysis with Flagger<\/strong><\/h4>\n\n\n\n<p>To trigger the canary analysis, you can update the deployment (<code>Kustomize\/deployment.yaml<\/code>).<\/p>\n\n\n\n<p>In your <code>Kustomize\/deployment.yaml<\/code>, update the Nginx image version to a new tag \u2014 for example,&nbsp; nginx:1.19.4 \u2014 and push to GitHub:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\n&nbsp;&nbsp;&nbsp;&nbsp;spec:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;containers:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- name: nginx\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;image: nginx:1.19.4\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ports:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- containerPort: 80<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>How does Flagger work?<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Monitoring the deployment:<\/strong> Flagger continuously monitors the target deployment specified in the targetRef of the canary resource. In our example, this is the nginx deployment.<\/li>\n\n\n\n<li><strong>Detecting new versions:<\/strong> When a new version of the application is pushed to the container registry and the deployment manifest is updated (for instance, when a new image tag is applied), Flagger detects this change. This is typically done by changing the image field in the deployment specification.<\/li>\n\n\n\n<li><strong>Initiating canary analysis:<\/strong> Once a change is detected, Flagger starts the canary analysis process. It creates a canary version of the deployment and routes a small percentage of traffic to it while monitoring key performance indicators (KPIs) such as success rate and request duration.<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Rollback with Flagger<\/strong><\/h4>\n\n\n\n<p>Let\u2019s assume the new version of the Nginx deployment (nginx:1.19.4) is failing to meet the success rate criteria:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Flagger detects that the success rate for nginx:1.19.4 has fallen to 95%.<\/li>\n\n\n\n<li>The threshold is set to 99%, so Flagger initiates a rollback.<\/li>\n\n\n\n<li>Flagger reverts the deployment to the previous stable version .<\/li>\n\n\n\n<li>The traffic is shifted back to the stable version, ensuring the stability of the application.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p>Flux CD simplifies the process of rollouts and rollbacks in Kubernetes clusters by leveraging GitOps principles. By continuously monitoring your Git repository, Flux CD ensures that your cluster state matches the desired state defined in your manifests.<\/p>\n\n\n\n<p>By following the steps outlined in this article, you can confidently manage application updates and rollbacks.<\/p>\n\n\n\n<p>To deepen the knowledge you\u2019ve gained from this tutorial, explore advanced concepts such as integrating Flagger with other monitoring tools like Prometheus for more granular metrics.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.aviator.co\/releases\"><img decoding=\"async\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/08\/blog-cta-9Release_CTA.svg\" alt=\"aviator releases\" class=\"wp-image-2489\"\/><\/a><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>A guide to configuring various deployment strategies to enhance software delivery with Flux CD and Flagger.<\/p>\n","protected":false},"author":23,"featured_media":2227,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[302],"tags":[31,97],"class_list":["post-2226","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tooling-integrations"],"blocksy_meta":[],"acf":[],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/06\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-2.png","post_mailing_queue_ids":[],"_links":{"self":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/2226","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/comments?post=2226"}],"version-history":[{"count":3,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/2226\/revisions"}],"predecessor-version":[{"id":3547,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/2226\/revisions\/3547"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/media\/2227"}],"wp:attachment":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/media?parent=2226"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/categories?post=2226"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/tags?post=2226"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}