{"id":1016,"date":"2023-01-30T15:40:15","date_gmt":"2023-01-30T15:40:15","guid":{"rendered":"https:\/\/www.aviator.co\/blog\/?p=1016"},"modified":"2025-09-25T13:59:41","modified_gmt":"2025-09-25T13:59:41","slug":"how-to-monitor-and-alert-on-nginx-ingress-in-kubernetes","status":"publish","type":"post","link":"https:\/\/www.aviator.co\/blog\/how-to-monitor-and-alert-on-nginx-ingress-in-kubernetes\/","title":{"rendered":"How to monitor and alert on Nginx ingress in Kubernetes"},"content":{"rendered":"\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2-1024x576.jpeg\" alt=\"\" class=\"wp-image-1026\" style=\"width:512px;height:288px\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2-1024x576.jpeg 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2-300x169.jpeg 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2-768x432.jpeg 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2-1536x864.jpeg 1536w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2.jpeg 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>In this blog post, we will discuss how to set up monitoring and alerting for Nginx ingress in a Kubernetes environment.<\/p>\n\n\n\n<p>We will cover the installation and configuration of ingress-nginx, Prometheus, and Grafana, and the setup of alerts for key Ingress metrics.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/docs.aviator.co\/mergequeue\" target=\"_blank\" rel=\" noreferrer noopener\"><img decoding=\"async\" width=\"1024\" height=\"264\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/04\/aviator-mergequeue-blog-documentation-cta-photo-min-1-1024x264.png\" alt=\"MergeQueue CTA\" class=\"wp-image-4921\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/04\/aviator-mergequeue-blog-documentation-cta-photo-min-1-1024x264.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/04\/aviator-mergequeue-blog-documentation-cta-photo-min-1-300x77.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/04\/aviator-mergequeue-blog-documentation-cta-photo-min-1-768x198.png 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/04\/aviator-mergequeue-blog-documentation-cta-photo-min-1-1536x396.png 1536w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2024\/04\/aviator-mergequeue-blog-documentation-cta-photo-min-1.png 1940w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Pre-requisites :<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A Kubernetes cluster<\/li>\n\n\n\n<li>Helm v3<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Install Prometheus and Grafana<\/h2>\n\n\n\n<p>In this step, we will install Prometheus to collect metrics, and Grafana to visualize and create alerts based on those metrics.<\/p>\n\n\n\n<p>Let&#8217;s install the <strong><a href=\"https:\/\/github.com\/prometheus-community\/helm-charts\/tree\/main\/charts\/kube-prometheus-stack\">kube-prometheus-stack<\/a><\/strong> helm chart by copying the below-mentioned commands to your terminal. This will set up Grafana, Prometheus, and other monitoring components.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Add and update the prometheus-community helm repository.\nhelm repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts\nhelm repo update<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>cat &lt;&lt;EOF | helm install kube-prometheus-stack prometheus-community\/kube-prometheus-stack \\\n--create-namespace -n monitoring -f -\n\ngrafana:\n  enabled: true\n  adminPassword: \"admin\"\n  persistence:\n    enabled: true\n    accessModes: &#91;\"ReadWriteOnce\"]\n    size: 1Gi\n  ingress:\n    enabled: true\n    ingressClassName: nginx\n    hosts:\n      - grafana.localdev.me\nEOF<\/code><\/pre>\n\n\n\n<p>Let&#8217;s see if the installed components are up and running :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl get pods -n monitoring\n\nNAME                                                        READY   STATUS    RESTARTS        AGE\nkube-prometheus-stack-grafana-7bb55544c9-qwkrg              3\/3     Running   0               3m38s\nprometheus-kube-prometheus-stack-prometheus-0               2\/2     Running   0               3m14s\n...<\/code><\/pre>\n\n\n\n<p>Looks great, let&#8217;s proceed to the next section.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Install &amp; configure Ingress Nginx<\/h3>\n\n\n\n<p>In this step, we will install and configure Nginx ingress controller and enable metrics that can be scraped by Prometheus.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Let&#8217;s install <strong>ingress-nginx<\/strong> into our cluster using the command below:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>helm upgrade --install ingress-nginx ingress-nginx \\\n  --repo https:\/\/kubernetes.github.io\/ingress-nginx \\\n  --namespace ingress-nginx --create-namespace \\\n  --set controller.metrics.enabled=true \\\n  --set controller.metrics.serviceMonitor.enabled=true \\\n  --set controller.metrics.serviceMonitor.additionalLabels.release=\"kube-prometheus-stack\"<\/code><\/pre>\n\n\n\n<p>Here we&#8217;re specifying <code>serviceMonitor.additionalLabels<\/code> to be <code>release: kube-prometheus-stack<\/code> so that Prometheus can discover the service monitor and automatically scrape metrics from it.<\/p>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li>Once the chart is installed, let\u2019s deploy a sample app <strong><a href=\"https:\/\/github.com\/stefanprodan\/podinfo\">podinfo<\/a><\/strong> into <code>default<\/code> namespace.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>helm install --wait podinfo --namespace default \\\noci:\/\/ghcr.io\/stefanprodan\/charts\/podinfo<\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Now, create an ingress for the deployed <strong>podinfo<\/strong> deployment :<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>cat &lt;&lt;EOF | kubectl apply -f -\napiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n  name: podinfo-ingress\nspec:\n  ingressClassName: nginx\n  rules :\n  - host: podinfo.localdev.me\n  defaultBackend:\n    service:\n      name: podinfo\n      port:\n        number: 9898\nEOF<\/code><\/pre>\n\n\n\n<p>Let\u2019s understand a bit about the above ingress config :<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We\u2019re using <code>ingress-nginx<\/code> as our ingress controller, hence the ingress class is defined as <code>nginx<\/code>.<\/li>\n\n\n\n<li>In the above config, I\u2019ve used the host address for my Ingress as <code>podinfo.localdev.me<\/code>.<\/li>\n\n\n\n<li>The DNS <code>*.localdev.me<\/code> resolves to <code>127.0.0.1<\/code>, hence for any local testing, this DNS can be used without the hassle of adding an entry into <code>\/etc\/hosts<\/code> file.<\/li>\n\n\n\n<li>Podinfo app serves HTTP API over port <code>9898<\/code>, hence it&#8217;s specified for the backend port i.e. when the traffic arrives for the domain <code>http:\/\/podinfo.localdev.me<\/code>, it will be forwarded to <code>9898<\/code> of podinfo service.<\/li>\n<\/ul>\n\n\n\n<p> 4. Next, from your terminal, port-forward the <code>ingress-nginx<\/code> service so that you can send traffic from your local terminal.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl port-forward -n ingress-nginx service\/ingress-nginx-controller 8080:80  &gt; \/dev\/null &amp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Port <code>80<\/code> on the host is a privileged port, so we\u2019re not using that. Instead, we\u2019re binding port <code>80<\/code> of nginx service to <code>8080<\/code> of host machine. You can specify any valid port of your choice.<\/li>\n<\/ul>\n\n\n\n<p>Note: If you\u2019re running this in any cloud, port-forwarding is not required as LoadBalancer for <strong>ingress-nginx<\/strong> service will be auto-created since the service type is defined as <code>LoadBalancer<\/code> by default.<\/p>\n\n\n\n<p>5. Now, you can run the below curl request to the podinfo endpoint, which should respond with :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; curl http:\/\/podinfo.localdev.me:8080\n\n\"hostname\": \"podinfo-59cd496d88-8dcsx\"\n\"message\": \"greetings from podinfo v6.2.2\"<\/code><\/pre>\n\n\n\n<p>6. You can also get the prettier look in the browser with URL : http:\/\/podinfo.localdev.me:8080\/<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"545\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/podinfo-web-1024x545.png\" alt=\"\" class=\"wp-image-1017\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/podinfo-web-1024x545.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/podinfo-web-300x160.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/podinfo-web-768x409.png 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/podinfo-web.png 1277w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Configure Grafana Dashboards for Ingress Nginx monitoring<\/h2>\n\n\n\n<p>To access Grafana, you can open the below URL in your browser with the credential <code>admin:admin<\/code> : http:\/\/grafana.localdev.me:8080\/.<\/p>\n\n\n\n<p>Copy the<code>nginx.json<\/code> from <a href=\"https:\/\/raw.githubusercontent.com\/kubernetes\/ingress-nginx\/main\/deploy\/grafana\/dashboards\/nginx.json\">here<\/a> and paste it into http:\/\/grafana.localdev.me:8080\/dashboard\/import to import the dashboard.<\/p>\n\n\n\n<p>Once imported, the dashboard should look like this :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"389\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-dashboard-1024x389.png\" alt=\"\" class=\"wp-image-1018\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-dashboard-1024x389.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-dashboard-300x114.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-dashboard-768x292.png 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-dashboard-1536x583.png 1536w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-dashboard.png 1843w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Generate sample loads<\/h3>\n\n\n\n<p>In order to get traffic on our my podinfo application, we\u2019ll be using vegeta as a load testing tool. Please install it from <a href=\"https:\/\/github.com\/tsenart\/vegeta\">here<\/a>.<\/p>\n\n\n\n<p>Let&#8217;s generate a sample HTTP 4xx traffic. To do that, you can run the below command which will run at a request rate of 10 RPS for 10 minutes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"GET http:\/\/podinfo.localdev.me:8080\/status\/400\" | vegeta attack -duration=10m -rate=10\/s<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You can just change the status code from 400 to 500 and run as well for test 5xx throughput.<\/li>\n<\/ul>\n\n\n\n<p>For latency tests, I\u2019ve used the below command as <code>GET \/delay\/{seconds} waits for the specified period<\/code> :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"GET http:\/\/podinfo.localdev.me:8080\/delay\/3\" | vegeta attack -duration=10m -rate=100\/s<\/code><\/pre>\n\n\n\n<p>Note: You can read more on the endpoints available in podinfo app from <a href=\"https:\/\/github.com\/stefanprodan\/podinfo\">here<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Alerting over SLI metrics with&nbsp;Grafana<\/h3>\n\n\n\n<p>The newer Grafana comes with its own alerting engine. That helps in keeping all config, rules, and even firing alerts in one place. Let&#8217;s configure alerts for common SLI.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4XX Error Rate<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Let&#8217;s create an alert by going to http:\/\/grafana.localdev.me:8080\/alerting\/new<\/li>\n\n\n\n<li>We can use the following formula to get 4xx error rate percentage :<\/li>\n<\/ol>\n\n\n\n<p><code>(total number of 4xx requests \/ total number of requests) * 100<\/code><\/p>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Add the below expression for the query :<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>(sum(rate(nginx_ingress_controller_requests{status=~'4..'}&#91;1m])) by (ingress) \/ sum(rate(nginx_ingress_controller_requests&#91;1m])) by (ingress)) * 100 &gt; 5<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"454\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-config-1024x454.png\" alt=\"\" class=\"wp-image-1019\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-config-1024x454.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-config-300x133.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-config-768x340.png 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-config.png 1496w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ol start=\"4\" class=\"wp-block-list\">\n<li>In Expression <code>B<\/code>, use <code>Reduce<\/code> operation with the function <code>Mean<\/code> for input <code>A<\/code>.<\/li>\n<\/ol>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li>In Alert Details, Name the alert as per your liking, I\u2019ve named it <code>Ingress_Nginx_4xx<\/code>.<\/li>\n\n\n\n<li>For Summary, we can keep it as short as possible, by just displaying the Ingress name with label <code>{{ $labels.ingress }}<\/code>.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>Ingress High Error Rate : 4xx on *{{ $labels.ingress }}*<\/code><\/pre>\n\n\n\n<ol start=\"7\" class=\"wp-block-list\">\n<li>For Description, I\u2019ve used <code>printf \"%0.2f\"<\/code> to display up to two decimals on the percentage value.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>4xx : High Error rate : `{{ printf \"%0.2f\" $values.B.Value }}%` on *{{ $labels.ingress }}*.<\/code><\/pre>\n\n\n\n<ol start=\"8\" class=\"wp-block-list\">\n<li>Overall alert should look similar to the below snapshot :<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"926\" height=\"548\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-details.png\" alt=\"\" class=\"wp-image-1020\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-details.png 926w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-details-300x178.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/nginx-ingress-alert-details-768x454.png 768w\" sizes=\"(max-width: 926px) 100vw, 926px\" \/><\/figure>\n\n\n\n<ol start=\"9\" class=\"wp-block-list\">\n<li>In the end, you can add a custom label like <code>severity : critical<\/code>.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">5XX Error Rate<\/h3>\n\n\n\n<p>Similar to 4xx alert config, 5xx error rate can also be queried with the below query :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sum(rate(nginx_ingress_controller_requests{status=~'5..'}&#91;1m])) by (ingress,cluster) \/ sum(rate(nginx_ingress_controller_requests&#91;1m]))by (ingress) * 100 &gt; 5<\/code><\/pre>\n\n\n\n<p>Note: I\u2019ve configured the alert to be triggered then the 5xx\/4xx percentage is &gt; 5%. You can set it as per your error budget.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">High Latency (p95)<\/h3>\n\n\n\n<p>To calculate the 95th percentile of request durations over the last 15m we can use the <code>nginx_ingress_controller_request_duration_seconds_bucket<\/code> metric.<\/p>\n\n\n\n<p>It gives you <strong>The request processing time in milliseconds<\/strong> and since its a bucket we can use <code>histogram_quantile<\/code> function.<\/p>\n\n\n\n<p>Create an alert similar to the above examples and use the below query&nbsp;:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>histogram_quantile(0.95,sum(rate(nginx_ingress_controller_request_duration_seconds_bucket&#91;15m])) by (le,ingress)) &gt; 1.5<\/code><\/pre>\n\n\n\n<p>I\u2019ve set the threshold to 1.5 seconds, it can be updated as per your SLO.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">High request rate<\/h3>\n\n\n\n<p>To get the request rate per second (RPS), we can use the below query :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sum(rate(nginx_ingress_controller_requests&#91;5m])) by (ingress) &gt; 2000<\/code><\/pre>\n\n\n\n<p>The above query will trigger an alert when the request rate is greater than 2000 RPS.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other SLIs to consider<\/h3>\n\n\n\n<p><strong>Connection rate<\/strong>: This measures the number of active connections to Nginx ingress, and can be used to identify potential issues with connection handling.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>rate(nginx_ingress_controller_nginx_process_connections{ingress=\"ingress-name\"}&#91;5m])<\/code><\/pre>\n\n\n\n<p><strong>Upstream response time<\/strong>: The time it takes for the underlying service to respond to a request, this will help to identify issues with the service and not just the ingress.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>histogram_quantile(0.95,sum(rate(nginx_ingress_controller_response_duration_seconds_bucket&#91;15m])) by (le,ingress)) <\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Slack Alert Template<\/h3>\n\n\n\n<p>To make alert messages meaningful, we can use <a href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/alerting\/contact-points\/message-templating\/\">alert templates in Grafana.<\/a><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>In order to configure them, go to http:\/\/grafana.localdev.me:8080\/alerting\/notifications and create a new template named <code>slack<\/code> by pasting the below code block :<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>{{ define \"alert_severity_prefix_emoji\" -}}\n    {{- if ne .Status \"firing\" -}}\n        :white_check_mark:\n    {{- else if eq .CommonLabels.severity \"critical\" -}}\n        :fire:\n    {{- else if eq .CommonLabels.severity \"warning\" -}}\n        :warning:\n    {{- end -}}\n{{- end -}}\n\n{{ define \"slack.title\" -}}\n    {{ template \"alert_severity_prefix_emoji\" . }}  {{- .Status | toUpper -}}{{- if eq .Status \"firing\" }} x {{ .Alerts.Firing | len -}}{{- end }}  |  {{ .CommonLabels.alertname -}}\n{{- end -}}\n\n{{- define \"slack.text\" -}}\n{{- range .Alerts -}}\n{{ if gt (len .Annotations) 0 }}\n*Summary*: {{ .Annotations.summary}}\n*Description*: {{ .Annotations.description }}\nLabels: \n{{ range .Labels.SortedPairs }}{{ if or (eq .Name \"ingress\") (eq .Name \"cluster\") }}\u2022 {{ .Name }}: `{{ .Value }}`\n{{ end }}{{ end }}\n{{ end }}\n{{ end }}\n{{ end }}<\/code><\/pre>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li>Configure a new contact point of type Slack. For this, you need to create an incoming webhook from Slack. Refer <a href=\"https:\/\/api.slack.com\/messaging\/webhooks#create_a_webhook\">this doc<\/a> for more detailed steps.<\/li>\n\n\n\n<li>Edit the contact point <strong>slack<\/strong> and scroll down and select the option <strong><code>Optional Slack settings<\/code>.<\/strong><\/li>\n\n\n\n<li>In the <strong>Title,<\/strong> enter the below to specify the template to use:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>{{ template \"slack.title\" . }}<\/code><\/pre>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li>In the <strong>Text Body,<\/strong> enter the below and save it :<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>{{ template \"slack.text\" . }}<\/code><\/pre>\n\n\n\n<ol start=\"6\" class=\"wp-block-list\">\n<li>Go to http:\/\/grafana.localdev.me:8080\/alerting\/routes and configure the <strong>Default contact point<\/strong> to be <strong>Slack<\/strong>.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Finally, the alert message arrives!<\/h3>\n\n\n\n<p>After configuring all the steps, finally we arrive at the end, and below are the snapshots of how the alert will look on your slack.<\/p>\n\n\n\n<p>4xx Error Rate :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"846\" height=\"196\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/4xx-slack-alert.png\" alt=\"\" class=\"wp-image-1021\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/4xx-slack-alert.png 846w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/4xx-slack-alert-300x70.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/4xx-slack-alert-768x178.png 768w\" sizes=\"(max-width: 846px) 100vw, 846px\" \/><\/figure>\n\n\n\n<p>5xx Error Rate :<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"846\" height=\"193\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/5xx-slack-alert.png\" alt=\"\" class=\"wp-image-1022\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/5xx-slack-alert.png 846w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/5xx-slack-alert-300x68.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/5xx-slack-alert-768x175.png 768w\" sizes=\"(max-width: 846px) 100vw, 846px\" \/><\/figure>\n\n\n\n<p>Latency P95 :<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"763\" height=\"187\" src=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/p65-slack-alert.png\" alt=\"\" class=\"wp-image-1023\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/p65-slack-alert.png 763w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/p65-slack-alert-300x74.png 300w\" sizes=\"(max-width: 763px) 100vw, 763px\" \/><\/figure>\n\n\n\n<p>There are lots of things one can improve according to their requirements. For example, if you have multiple Kubernetes clusters, you can add a <code>cluster<\/code> label that will help in identifying the source cluster for the alert.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">References<\/h3>\n\n\n\n<p><a href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/alerting\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/alerting\/<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/kubernetes.github.io\/ingress-nginx\/user-guide\/monitoring\/\">https:\/\/kubernetes.github.io\/ingress-nginx\/user-guide\/monitoring\/<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a href=\"https:\/\/www.aviator.co\/\" target=\"_blank\" rel=\"noreferrer noopener\">Aviator<\/a>: Automate your cumbersome merge processes<\/h2>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/www.aviator.co\/\" target=\"_blank\" rel=\"noreferrer noopener\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"727\" src=\"https:\/\/blog.aviator.co\/wp-content\/uploads\/2022\/08\/blog-cta-1024x727.png\" alt=\"\" class=\"wp-image-57\" srcset=\"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2022\/08\/blog-cta-1024x727.png 1024w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2022\/08\/blog-cta-300x213.png 300w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2022\/08\/blog-cta-768x545.png 768w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2022\/08\/blog-cta-1536x1090.png 1536w, https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2022\/08\/blog-cta-2048x1454.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p>Aviator automates tedious developer workflows by managing git Pull Requests (PRs) and continuous integration test (CI) runs to help your team avoid broken builds, streamline cumbersome merge processes, manage cross-PR dependencies, and handle flaky tests while maintaining their security compliance.<\/p>\n\n\n\n<p>There are 4 key components to Aviator:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>MergeQueue<\/strong>&nbsp;\u2013 an automated queue that manages the merging workflow for your GitHub repository to help protect important branches from broken builds. The Aviator bot uses GitHub Labels to identify Pull Requests (PRs) that are ready to be merged, validates CI checks, processes semantic conflicts, and merges the PRs automatically.<\/li>\n\n\n\n<li><strong>ChangeSets<\/strong>&nbsp;\u2013 workflows to synchronize validating and merging multiple PRs within the same repository or multiple repositories. Useful when your team often sees groups of related PRs that need to be merged together, or otherwise treated as a single broader unit of change.<\/li>\n\n\n\n<li><strong>FlakyBot<\/strong>&nbsp;\u2013 a tool to automatically detect, take action on, and process results from flaky tests in your CI infrastructure.<\/li>\n\n\n\n<li><strong>Stacked PRs CLI<\/strong>&nbsp;\u2013 a command line tool that helps developers manage cross-PR dependencies. This tool also automates syncing and merging of stacked PRs. Useful when your team wants to promote a culture of smaller, incremental PRs instead of large changes, or when your workflows involve keeping multiple, dependent PRs in sync.<\/li>\n<\/ol>\n\n\n\n<p><a href=\"https:\/\/www.aviator.co\/\" target=\"_blank\" rel=\"noopener\" title=\"\">Try it for free.<\/a><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to set up monitoring and alerting for Nginx ingress in a Kubernetes environment.<\/p>\n","protected":false},"author":11,"featured_media":1026,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[302],"tags":[36,35],"class_list":["post-1016","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tooling-integrations"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":6}},"acf":[],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/www.aviator.co\/blog\/wp-content\/uploads\/2023\/01\/Studio_Project-2.jpeg","post_mailing_queue_ids":[],"_links":{"self":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/1016","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/comments?post=1016"}],"version-history":[{"count":10,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/1016\/revisions"}],"predecessor-version":[{"id":4967,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/posts\/1016\/revisions\/4967"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/media\/1026"}],"wp:attachment":[{"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/media?parent=1016"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/categories?post=1016"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aviator.co\/blog\/wp-json\/wp\/v2\/tags?post=1016"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}