{"id":2405,"date":"2024-07-19T02:33:24","date_gmt":"2024-07-19T02:33:24","guid":{"rendered":"https:\/\/www.aviator.co\/blog\/?p=2405"},"modified":"2025-08-19T12:34:05","modified_gmt":"2025-08-19T12:34:05","slug":"deployments-and-rollbacks-using-ecs-and-github-actions","status":"publish","type":"post","link":"https:\/\/www.aviator.co\/blog\/deployments-and-rollbacks-using-ecs-and-github-actions\/","title":{"rendered":"Deployments and Rollbacks Using ECS and GitHub Actions"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1-1024x576.jpg\" alt=\"\" class=\"wp-image-2408\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1-1024x576.jpg 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1-300x169.jpg 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1-768x432.jpg 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1-1536x864.jpg 1536w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1-2048x1152.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Amazon ECS offers <a href=\"https:\/\/aws.amazon.com\/about-aws\/whats-new\/2022\/12\/amazon-ecs-cloudwatch-alarms-safety-deployments\/\" target=\"_blank\" rel=\"noopener\" title=\"\">native support<\/a> for monitoring and automatically managing updates using Amazon CloudWatch metric alarms. However, in this article, we&#8217;ll explore how to accomplish this with GitHub Actions, providing more flexibility and integration with existing workflows.<\/p>\n\n\n\n<p>We will set up a workflow to deploy releases when changes are pushed to the main branch. For rollbacks, we&#8217;ll configure CloudWatch to monitor HTTP 5xx errors, high memory utilization, and high CPU utilization. If any of these metrics show issues, the rollback will be triggered.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<p>To follow this tutorial, you need:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Basic understanding of ECS<\/li>\n\n\n\n<li>An application already running on ECS<\/li>\n\n\n\n<li>The following GitHub repository secrets:\n<ul class=\"wp-block-list\">\n<li><code>AWS_ACCESS_KEY_ID<\/code><\/li>\n\n\n\n<li><code>AWS_SECRET_ACCESS_KEY<\/code><\/li>\n\n\n\n<li><code>AWS_REGION<\/code><\/li>\n\n\n\n<li><code>ECS_CLUSTER<\/code><\/li>\n\n\n\n<li><code>ECS_SERVICE<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Workflow for Releases<\/h2>\n\n\n\n<p>Assuming you have a project on GitHub, create a workflow file for releases (<code>.github\/workflows\/releases.yaml<\/code>) and use the following code. This will build and push changes to Docker Hub and trigger a service update via the AWS CLI, allowing ECS to deploy the latest version of your project.<\/p>\n\n\n\n<p><em>Note: Some of the credentials left in the config file below should be stored secretly.<\/em><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>name: Deploy to ECS\non:\n  push:\n    branches:\n      - main\njobs:\n  deploy:\n    runs-on: ubuntu-latest\n    steps:\n      - name: Checkout\n        uses: actions\/checkout@v2\n      \n      - name: Log in to Docker Hub\n        run: echo \"${{ secrets.DOCKER_HUB_PASSWORD }}\" | docker login -u \"${{ secrets.DOCKER_HUB_USERNAME }}\" --password-stdin\n      - name: Set up QEMU\n        uses: docker\/setup-qemu-action@v2\n      - name: Set up Docker Buildx\n        uses: docker\/setup-buildx-action@v2\n      - name: Build and push Docker image\n        run: |\n          docker build -t khabdrick\/ecsproject:${{ github.sha }} .\n          docker push khabdrick\/ecsproject:${{ github.sha }}\n          echo \"IMAGE_TAG=khabdrick\/ecsproject:${{ github.sha }}\" &gt;&gt; $GITHUB_ENV\n      - name: Install AWS CLI\n        run: sudo apt-get update &amp;&amp; sudo apt-get install -y awscli\n      - name: Configure AWS CLI\n        run: |\n          aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }}\n          aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n          aws configure set region ${{ secrets.AWS_REGION }}\n      - name: Register new task definition revision\n        run: |\n          aws ecs register-task-definition \\\n            --family ecsproject_task \\\n            --execution-role-arn arn:aws:iam::925248302005:role\/ecstaskrole \\\n            --task-role-arn arn:aws:iam::925248302005:role\/ecstaskrole \\\n            --network-mode awsvpc \\\n            --requires-compatibilities FARGATE \\\n            --cpu \"1024\" \\\n            --memory \"3072\" \\\n            --container-definitions '&#91;\n                {\n                    \"name\": \"mongo\",\n                    \"image\": \"mongo:latest\",\n                    \"cpu\": 0,\n                    \"memory\": 2048,\n                    \"portMappings\": &#91;\n                        {\n                            \"appProtocol\": \"http\",\n                            \"containerPort\": 27017,\n                            \"hostPort\": 27017,\n                            \"name\": \"mongo-27017-tcp\",\n                            \"protocol\": \"tcp\"\n                        }\n                    ],\n                    \"essential\": true,\n                    \"environment\": &#91;\n                        {\n                            \"name\": \"MONGO_INITDB_ROOT_USERNAME\",\n                            \"value\": \"mongo\"\n                        },\n                        {\n                            \"name\": \"MONGO_INITDB_ROOT_PASSWORD\",\n                            \"value\": \"password\"\n                        }\n                    ],\n                    \"mountPoints\": &#91;\n                        {\n                            \"sourceVolume\": \"mongo-mount\",\n                            \"containerPath\": \"\/data\/db\",\n                            \"readOnly\": false\n                        }\n                    ],\n                    \"logConfiguration\": {\n                        \"logDriver\": \"awslogs\",\n                        \"options\": {\n                            \"awslogs-group\": \"\/ecs\/ecsproject_task\",\n                            \"awslogs-create-group\": \"true\",\n                            \"awslogs-region\": \"us-east-1\",\n                            \"awslogs-stream-prefix\": \"ecs\"\n                        }\n                    }\n                },\n                {\n                    \"name\": \"project_container\",\n                    \"image\": \"${{ env.IMAGE_TAG }}\",\n                    \"cpu\": 0,\n                    \"memory\": 1024,\n                    \"portMappings\": &#91;\n                        {\n                            \"containerPort\": 3000,\n                            \"hostPort\": 3000,\n                            \"name\": \"project_container-3000-tcp\",\n                            \"protocol\": \"tcp\"\n                        }\n                    ],\n                    \"essential\": false,\n                    \"environment\": &#91;\n                        {\n                            \"name\": \"MONGO_USER\",\n                            \"value\": \"mongo\"\n                        },\n                        {\n                            \"name\": \"MONGO_IP\",\n                            \"value\": \"localhost\"\n                        },\n                        {\n                            \"name\": \"MONGO_PORT\",\n                            \"value\": \"27017\"\n                        },\n                        {\n                            \"name\": \"MONGO_PASSWORD\",\n                            \"value\": \"password\"\n                        }\n                    ]\n                }\n            ]' \\\n            --volumes '&#91;\n                {\n                    \"name\": \"mongo-mount\",\n                    \"efsVolumeConfiguration\": {\n                        \"fileSystemId\": \"fs-0ae93a5984f5ff5c0\",\n                        \"rootDirectory\": \"\/\"\n                    }\n                }\n            ]' \\\n            --runtime-platform '{\"cpuArchitecture\": \"X86_64\", \"operatingSystemFamily\": \"LINUX\"}' \\\n            --output json &gt; new-task-def.json\n      - name: Update ECS service to use new task definition\n        run: |\n          NEW_TASK_DEF_ARN=$(jq -r '.taskDefinition.taskDefinitionArn' new-task-def.json)\n          aws ecs update-service \\\n            --cluster ${{ secrets.ECS_CLUSTER }} \\\n            --service ${{ secrets.ECS_SERVICE }} \\\n            --task-definition $NEW_TASK_DEF_ARN<\/code><\/pre>\n\n\n\n<p>This workflow automates deploying a Docker-based application to Amazon ECS whenever changes are pushed to the main branch. It sets up QEMU for multi-platform builds and Docker Buildx for building and pushing Docker images.<\/p>\n\n\n\n<p>Once the Docker image is built and pushed, the workflow installs and configures the AWS CLI using credentials stored in GitHub Secrets. It then registers a new task definition revision in ECS. This task definition includes two containers: one for a MongoDB database and another for the application itself. You can modify this portion to fit the specific requirements of your application running on ECS.<\/p>\n\n\n\n<p>Finally, the script updates the ECS service to use the newly registered task definition. It extracts the ARN (Amazon Resource Name) of the new task definition from the output JSON file and updates the ECS service using this ARN, ensuring that the ECS service runs the latest version of the application.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Monitor and Rollback<\/h2>\n\n\n\n<p>Create another workflow (<code>.github\/workflows\/rollback.yaml<\/code>) to run every ten minutes, five times after deployment, checking CloudWatch alarms. If any issues are detected, the rollback to the previous task will be triggered.<\/p>\n\n\n\n<p>First, create an SNS topic for alarm actions:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Open the AWS Management Console and navigate to Amazon SNS.<\/li>\n\n\n\n<li>Create a new topic.<\/li>\n\n\n\n<li>Note the ARN of the created topic (e.g., <code>arn:aws:sns:us-east-1:123456789012:MyTopic<\/code>).<\/li>\n<\/ol>\n\n\n\n<p>Next, create CloudWatch alarms for <code>HighHTTP5xxErrors<\/code>, <code>HighMemoryUtilization<\/code>, and <code>HighCPUUtilization<\/code> using the AWS CLI. Replace <code>&lt;arn:aws:sns:us-east-1:123456789012:MyTopic&gt;<\/code> with your SNS topic ARN and <code>ECSproject<\/code> with your cluster name.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>aws cloudwatch put-metric-alarm \\\n    --alarm-name HighHTTP5xxErrors \\\n    --metric-name HTTPCode_Backend_5XX \\\n    --namespace AWS\/ApplicationELB \\\n    --statistic Sum \\\n    --period 300 \\\n    --evaluation-periods 3 \\\n    --threshold 10 \\\n    --comparison-operator GreaterThanThreshold \\\n    --dimensions Name=LoadBalancer,Value=note-api-lb \\\n    --alarm-actions &lt;arn:aws:sns:us-east-1:123456789012:MyTopic&gt; \\\n    --unit Count<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>aws cloudwatch put-metric-alarm \\\n    --alarm-name HighMemoryUtilization \\\n    --metric-name MemoryUtilization \\\n    --namespace AWS\/ECS \\\n    --statistic Average \\\n    --period 300 \\\n    --evaluation-periods 3 \\\n    --threshold 80 \\\n    --comparison-operator GreaterThanThreshold \\\n    --dimensions Name=ClusterName,Value=ECSproject \\\n    --alarm-actions &lt;arn:aws:sns:us-east-1:123456789012:MyTopic&gt; \\\n    --unit Percent<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>aws cloudwatch put-metric-alarm \\\n    --alarm-name HighCPUUtilization \\\n    --metric-name CPUUtilization \\\n    --namespace AWS\/ECS \\\n    --statistic Average \\\n    --period 300 \\\n    --evaluation-periods 3 \\\n    --threshold 80 \\\n    --comparison-operator GreaterThanThreshold \\\n    --dimensions Name=ClusterName,Value=ECSproject \\\n    --alarm-actions &lt;arn:aws:sns:us-east-1:123456789012:MyTopic&gt; \\\n    --unit Percent<\/code><\/pre>\n\n\n\n<p><br>And paste in the workflow for rolling back:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>name: Rollback to Previous Deployment\non:\n  workflow_run:\n    workflows: &#91;\"Deploy to ECS\"]\n    types:\n      - completed\njobs:\n  rollback:\n    runs-on: ubuntu-latest\n    if: ${{ github.event.workflow_run.conclusion == 'success' }}\n    strategy:\n      matrix:\n        attempt: &#91;1, 2, 3, 4, 5]\n    steps:\n      - name: Checkout\n        uses: actions\/checkout@v2\n      - name: Wait before rollback attempt ${{ matrix.attempt }}\n        run: sleep $(( ${{ matrix.attempt }} * 600 ))\n      \n      - name: Install AWS CLI\n        run: |\n          sudo apt-get update\n          sudo apt-get install -y awscli\n      - name: Configure AWS CLI\n        run: |\n          aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }}\n          aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n          aws configure set region ${{ secrets.AWS_REGION }}\n\n      - name: Check for CloudWatch Alarms\n        id: check_alarm_state\n        run: |\n          CPU_ALARM_STATE=$(aws cloudwatch describe-alarms --alarm-names \"HighCPUUtilization\" --state-value ALARM --query 'MetricAlarms&#91;0].StateValue' --region ${{ secrets.AWS_REGION }})\n          MEMORY_ALARM_STATE=$(aws cloudwatch describe-alarms --alarm-names \"HighMemoryUtilization\" --state-value ALARM --query 'MetricAlarms&#91;0].StateValue' --region ${{ secrets.AWS_REGION }})\n          HTTP_ALARM_STATE=$(aws cloudwatch describe-alarms --alarm-names \"HighHTTP5xxErrors\" --state-value ALARM --query 'MetricAlarms&#91;0].StateValue' --region ${{ secrets.AWS_REGION }})\n          if &#91; \"$CPU_ALARM_STATE\" == \"ALARM\" ] || &#91; \"$MEMORY_ALARM_STATE\" == \"ALARM\" ] || &#91; \"$HTTP_ALARM_STATE\" == \"ALARM\" ]; then\n            echo \"ALARM\"\n            echo \"::set-output name=alarm_state::ALARM\"\n          else\n            echo \"OK\"\n            echo \"::set-output name=alarm_state::OK\"\n          fi\n\n      - name: Get the second-to-last task definition revision\n        id: get_previous_task_definition\n        run: |\n          if &#91; \"${{ steps.check_alarm_state.outputs.alarm_state }}\" == \"ALARM\" ]; then\n            TASK_DEFINITION=$(aws ecs describe-services --cluster ${{ secrets.ECS_CLUSTER }} --services ${{ secrets.ECS_SERVICE }} --query 'services&#91;0].deployments&#91;1].taskDefinition' --output text)\n            echo \"::set-output name=task_definition::${TASK_DEFINITION}\"\n          else\n            echo \"No alarm, no rollback needed.\"\n            exit 0\n          fi\n      \n      - name: Rollback to previous task definition\n        if: steps.check_alarm_state.outputs.alarm_state == 'ALARM'\n        run: |\n          aws ecs update-service \\\n            --cluster ${{ secrets.ECS_CLUSTER }} \\\n            --service ${{ secrets.ECS_SERVICE }} \\\n            --task-definition ${{ steps.get_previous_task_definition.outputs.task_definition }}<\/code><\/pre>\n\n\n\n<p>This workflow will rollback a deployment on Amazon ECS if specific alarms are triggered after a successful deployment. It is triggered upon the completion of the &#8220;Deploy to ECS&#8221; workflow and only proceeds if the deployment was successful. The rollback job uses a matrix strategy to attempt the rollback up to five times, with each attempt spaced 10 minutes apart.<\/p>\n\n\n\n<p>The core function of the workflow is to monitor specific CloudWatch alarms for CPU utilization, memory utilization, and HTTP 5xx errors. The script checks the state of these alarms and sets an output variable, <code>alarm_state<\/code>, to &#8220;ALARM&#8221; if any are triggered. This condition determines whether the rollback should proceed.<\/p>\n\n\n\n<p>If any alarms are in the &#8220;ALARM&#8221; state, the workflow retrieves the second-to-last task definition revision for the ECS service, representing the previous stable deployment. This task definition is then used to update the ECS service, effectively rolling back to the prior version. This ensures that if the latest deployment causes issues, the system can quickly revert to a stable state.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Conclusion<\/h3>\n\n\n\n<p>We covered how to use GitHub Actions to automate the deployment and rollback processes for an application running on Amazon ECS. This includes setting up a workflow for releasing updates when changes are pushed to the main branch and configuring CloudWatch alarms to monitor key metrics. If any alarms are triggered, the workflow initiates a rollback to a previous task definition to maintain application reliability.<\/p>\n\n\n\n<p>To further improve your deployment strategy, consider advanced techniques like blue-green deployments, canary deployments, using AWS CodeDeploy, or integrating monitoring tools like Prometheus and Grafana for better insights.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.aviator.co\/releases\"><img decoding=\"async\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/08\/blog-cta-9Release_CTA.svg\" alt=\"aviator releases\" class=\"wp-image-2489\"\/><\/a><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>This post will walk you through setting up automated deployments as well as automatic rollbacks for an ECS set up using GitHub Actions.<\/p>\n","protected":false},"author":23,"featured_media":2408,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[58],"tags":[109],"class_list":["post-2405","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci-cd-deployment"],"blocksy_meta":[],"acf":[],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/07\/Blue-Black-Futuristic-Light-Leak-Automotive-Reviewer-Youtube-Channel-Art-1.jpg","post_mailing_queue_ids":[],"_links":{"self":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/2405","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/comments?post=2405"}],"version-history":[{"count":4,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/2405\/revisions"}],"predecessor-version":[{"id":3208,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/2405\/revisions\/3208"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/media\/2408"}],"wp:attachment":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/media?parent=2405"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/categories?post=2405"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/tags?post=2405"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}