Add EKS Istio Dashboards (#194)

* Update readme * checking in new readme example * new istio readme * Updated mkdocs.yml * checking in readme files * checking in all changes * Test * Test * Test * Testing * Testing * test * Update main.tf * Update outputs.tf * Adding Istio dashboards * Updating URL's * Removing dashboards * Updates from pre-commit * Update istio.md * Removing empty file * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update README.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update istio.md * Update README.md * Update README.md * Istio Documentation Updates Added istioctl prereq, updated dashboards image * Updated setup and cleanup of Istio Bookinfo Updated Istio Bookinfo sample app instructions, included clean up of Bookinfo resources * Updating image and removing advanced configuration -Switching image to Github generated link -Removing advanced configuration section --------- Co-authored-by: Prithvi Reddy <ypreddy@amazon.com> Co-authored-by: Prithvi Reddy <47993564+preddy727@users.noreply.github.com>
2026-10-09 00:09:43 +00:00 · 2023-07-27 12:44:52 -04:00
parent 7f60bd72d6
commit 1ceecb3041
18 changed files with 904 additions and 0 deletions
@@ -40,6 +40,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
 | <a name="module_external_secrets"></a> [external\_secrets](#module\_external\_secrets) | ./add-ons/external-secrets | n/a |
 | <a name="module_fluentbit_logs"></a> [fluentbit\_logs](#module\_fluentbit\_logs) | ./add-ons/aws-for-fluentbit | n/a |
 | <a name="module_helm_addon"></a> [helm\_addon](#module\_helm\_addon) | github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons/helm-addon | v4.32.0 |
+| <a name="module_istio_monitoring"></a> [istio\_monitoring](#module\_istio\_monitoring) | ./patterns/istio | n/a |
 | <a name="module_java_monitoring"></a> [java\_monitoring](#module\_java\_monitoring) | ./patterns/java | n/a |
 | <a name="module_nginx_monitoring"></a> [nginx\_monitoring](#module\_nginx\_monitoring) | ./patterns/nginx | n/a |
 | <a name="module_operator"></a> [operator](#module\_operator) | ./add-ons/adot-operator | n/a |
@@ -76,6 +77,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
 | <a name="input_enable_external_secrets"></a> [enable\_external\_secrets](#input\_enable\_external\_secrets) | Installs External Secrets to EKS Cluster | `bool` | `true` | no |
 | <a name="input_enable_fluxcd"></a> [enable\_fluxcd](#input\_enable\_fluxcd) | Enables or disables FluxCD. Disabling this might affect some data in the dashboards | `bool` | `true` | no |
 | <a name="input_enable_grafana_operator"></a> [enable\_grafana\_operator](#input\_enable\_grafana\_operator) | Deploys Grafana Operator to EKS Cluster | `bool` | `true` | no |
+| <a name="input_enable_istio"></a> [enable\_istio](#input\_enable\_istio) | Enable ISTIO workloads monitoring, alerting and default dashboards | `bool` | `false` | no |
 | <a name="input_enable_java"></a> [enable\_java](#input\_enable\_java) | Enable Java workloads monitoring, alerting and default dashboards | `bool` | `false` | no |
 | <a name="input_enable_kube_state_metrics"></a> [enable\_kube\_state\_metrics](#input\_enable\_kube\_state\_metrics) | Enables or disables Kube State metrics exporter. Disabling this might affect some data in the dashboards | `bool` | `true` | no |
 | <a name="input_enable_logs"></a> [enable\_logs](#input\_enable\_logs) | Using AWS For FluentBit to collect cluster and application logs to Amazon CloudWatch | `bool` | `true` | no |
@@ -101,6 +103,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
 | <a name="input_helm_config"></a> [helm\_config](#input\_helm\_config) | Helm Config for Prometheus | `any` | `{}` | no |
 | <a name="input_irsa_iam_permissions_boundary"></a> [irsa\_iam\_permissions\_boundary](#input\_irsa\_iam\_permissions\_boundary) | IAM permissions boundary for IRSA roles | `string` | `null` | no |
 | <a name="input_irsa_iam_role_path"></a> [irsa\_iam\_role\_path](#input\_irsa\_iam\_role\_path) | IAM role path for IRSA roles | `string` | `"/"` | no |
+| <a name="input_istio_config"></a> [istio\_config](#input\_istio\_config) | Configuration object for ISTIO monitoring | <pre>object({<br>    enable_alerting_rules  = bool<br>    enable_recording_rules = bool<br>    enable_dashboards      = bool<br>    scrape_sample_limit    = number<br><br>    flux_gitrepository_name   = string<br>    flux_gitrepository_url    = string<br>    flux_gitrepository_branch = string<br>    flux_kustomization_name   = string<br>    flux_kustomization_path   = string<br><br>    grafana_url                             = string<br>    grafana_istio_cp_dashboard_url          = string<br>    grafana_istio_mesh_dashboard_url        = string<br>    grafana_istio_performance_dashboard_url = string<br>    grafana_istio_service_dashboard_url     = string<br><br>    prometheus_metrics_endpoint = string<br>  })</pre> | `null` | no |
 | <a name="input_java_config"></a> [java\_config](#input\_java\_config) | Configuration object for Java/JMX monitoring | <pre>object({<br>    enable_alerting_rules  = bool<br>    enable_recording_rules = bool<br>    enable_dashboards      = bool<br>    scrape_sample_limit    = number<br><br><br>    flux_gitrepository_name   = string<br>    flux_gitrepository_url    = string<br>    flux_gitrepository_branch = string<br>    flux_kustomization_name   = string<br>    flux_kustomization_path   = string<br><br>    grafana_dashboard_url = string<br><br>    prometheus_metrics_endpoint = string<br>  })</pre> | `null` | no |
 | <a name="input_ksm_config"></a> [ksm\_config](#input\_ksm\_config) | Kube State metrics configuration | <pre>object({<br>    create_namespace   = bool<br>    k8s_namespace      = string<br>    helm_chart_name    = string<br>    helm_chart_version = string<br>    helm_release_name  = string<br>    helm_repo_url      = string<br>    helm_settings      = map(string)<br>    helm_values        = map(any)<br><br>    scrape_interval = string<br>    scrape_timeout  = string<br>  })</pre> | <pre>{<br>  "create_namespace": true,<br>  "helm_chart_name": "kube-state-metrics",<br>  "helm_chart_version": "4.24.0",<br>  "helm_release_name": "kube-state-metrics",<br>  "helm_repo_url": "https://prometheus-community.github.io/helm-charts",<br>  "helm_settings": {},<br>  "helm_values": {},<br>  "k8s_namespace": "kube-system",<br>  "scrape_interval": "60s",<br>  "scrape_timeout": "15s"<br>}</pre> | no |
 | <a name="input_logs_config"></a> [logs\_config](#input\_logs\_config) | Configuration object for logs collection | <pre>object({<br>    cw_log_retention_days = number<br>  })</pre> | <pre>{<br>  "cw_log_retention_days": 90<br>}</pre> | no |
@@ -77,4 +77,32 @@ locals {
    grafana_url           = var.grafana_url
    grafana_dashboard_url = "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/nginx/nginx.json"
  }
+
+  istio_pattern_config = {
+    # disabled if options from module are disabled, by default
+    # can be overriden by providing a config
+    enable_alerting_rules  = var.enable_alerting_rules
+    enable_recording_rules = var.enable_recording_rules
+    enable_dashboards      = var.enable_dashboards
+
+    scrape_sample_limit = 1000
+
+    flux_gitrepository_name   = "aws-observability-accelerator"
+    flux_gitrepository_url    = "https://github.com/aws-observability/aws-observability-accelerator"
+    flux_gitrepository_branch = "main"
+    flux_kustomization_name   = "grafana-dashboards-istio"
+    flux_kustomization_path   = "./artifacts/grafana-operator-manifests/eks/istio"
+
+    managed_prometheus_workspace_id       = var.managed_prometheus_workspace_id
+    managed_prometheus_workspace_region   = var.managed_prometheus_workspace_region
+    managed_prometheus_workspace_endpoint = var.managed_prometheus_workspace_endpoint
+    prometheus_metrics_endpoint           = "/metrics"
+
+    grafana_url                             = var.grafana_url
+    grafana_istio_cp_dashboard_url          = "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/istio/istio-control-plane-dashboard.json"
+    grafana_istio_mesh_dashboard_url        = "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/istio/istio-mesh-dashboard.json"
+    grafana_istio_performance_dashboard_url = "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/istio/istio-performance-dashboard.json"
+    grafana_istio_service_dashboard_url     = "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/istio/istio-service-dashboard.json"
+  }
+
 }
@@ -169,6 +169,18 @@ module "helm_addon" {
      name  = "nginxPrometheusMetricsEndpoint"
      value = try(var.nginx_config.prometheus_metrics_endpoint, local.nginx_pattern_config.prometheus_metrics_endpoint)
    },
+    {
+      name  = "enableIstio"
+      value = var.enable_istio
+    },
+    {
+      name  = "istioScrapeSampleLimit"
+      value = try(var.istio_config.scrape_sample_limit, local.istio_pattern_config.scrape_sample_limit)
+    },
+    {
+      name  = "istioPrometheusMetricsEndpoint"
+      value = try(var.istio_config.prometheus_metrics_endpoint, local.istio_pattern_config.prometheus_metrics_endpoint)
+    }
  ]

  irsa_config = {
@@ -202,6 +214,13 @@ module "nginx_monitoring" {
  pattern_config = coalesce(var.nginx_config, local.nginx_pattern_config)
 }

+module "istio_monitoring" {
+  source = "./patterns/istio"
+  count  = var.enable_istio ? 1 : 0
+
+  pattern_config = coalesce(var.istio_config, local.istio_pattern_config)
+}
+
 module "fluentbit_logs" {
  source = "./add-ons/aws-for-fluentbit"
  count  = var.enable_logs ? 1 : 0
@@ -1562,6 +1562,67 @@ spec:
                  action: labeldrop
            {{ end }}

+         {{ if .Values.enableIstio }}
+            - honor_labels: true
+              job_name: kubernetes-istio
+              kubernetes_sd_configs:
+              - role: pod
+              relabel_configs:
+              - action: keep
+                regex: true
+                source_labels:
+                - __meta_kubernetes_pod_annotation_prometheus_io_scrape
+              - action: drop
+                regex: true
+                source_labels:
+                - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
+              - action: replace
+                regex: (https?)
+                source_labels:
+                - __meta_kubernetes_pod_annotation_prometheus_io_scheme
+                target_label: __scheme__
+              - action: replace
+                regex: (.+)
+                source_labels:
+                - __meta_kubernetes_pod_annotation_prometheus_io_path
+                target_label: __metrics_path__
+              - action: replace
+                regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
+                replacement: '[$$2]:$$1'
+                source_labels:
+                - __meta_kubernetes_pod_annotation_prometheus_io_port
+                - __meta_kubernetes_pod_ip
+                target_label: __address__
+              - action: replace
+                regex: (\d+);((([0-9]+?)(\.|$)){4})
+                replacement: $$2:$$1
+                source_labels:
+                - __meta_kubernetes_pod_annotation_prometheus_io_port
+                - __meta_kubernetes_pod_ip
+                target_label: __address__
+              - action: labelmap
+                regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
+                replacement: __param_$1
+              - action: labelmap
+                regex: __meta_kubernetes_pod_label_(.+)
+              - action: replace
+                source_labels:
+                - __meta_kubernetes_namespace
+                target_label: namespace
+              - action: replace
+                source_labels:
+                - __meta_kubernetes_pod_name
+                target_label: pod
+              - action: keep
+                source_labels: [ __address__ ]
+                regex: '.*:15020$$'
+              - action: drop
+                regex: Pending|Succeeded|Failed|Completed
+                source_labels:
+                - __meta_kubernetes_pod_phase
+            {{ end }}
+
+
    exporters:
      {{ if .Values.enableTracing }}
      awsxray:
@@ -24,4 +24,8 @@ enableNginx: ${enable_nginx}
 nginxScrapeSampleLimit: ${nginx_scrape_sample_limit}
 nginxPrometheusMetricsEndpoint: ${nginx_prometheus_metrics_endpoint}

+enableIstio: ${enable_istio}
+istioScrapeSampleLimit: ${istio_scrape_sample_limit}
+istioPrometheusMetricsEndpoint: ${istio_prometheus_metrics_endpoint}
+
 adotLoglevel: ${adot_loglevel}
@@ -0,0 +1,47 @@
+# Istio patterns module
+
+Provides monitoring for Istio based workloads with the following resources:
+
+- AWS Managed Grafana Dashboard and data source
+- Alerts and recording rules with AWS Managed Service for Prometheus
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.0.0 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |
+| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 1.14 |
+| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.0.0 |
+| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 1.14 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_prometheus_rule_group_namespace.alerting_rules](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_rule_group_namespace) | resource |
+| [aws_prometheus_rule_group_namespace.recording_rules](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_rule_group_namespace) | resource |
+| [kubectl_manifest.flux_kustomization](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_pattern_config"></a> [pattern\_config](#input\_pattern\_config) | Configuration object for ISTIO monitoring | <pre>object({<br>    enable_alerting_rules  = bool<br>    enable_recording_rules = bool<br>    scrape_sample_limit    = number<br><br>    enable_recording_rules = bool<br><br>    enable_dashboards = bool<br><br>    flux_gitrepository_name   = string<br>    flux_gitrepository_url    = string<br>    flux_gitrepository_branch = string<br>    flux_kustomization_name   = string<br>    flux_kustomization_path   = string<br><br>    managed_prometheus_workspace_id       = string<br>    managed_prometheus_workspace_region   = string<br>    managed_prometheus_workspace_endpoint = string<br><br>    grafana_url                             = string<br>    grafana_istio_cp_dashboard_url          = string<br>    grafana_istio_mesh_dashboard_url        = string<br>    grafana_istio_performance_dashboard_url = string<br>    grafana_istio_service_dashboard_url     = string<br>  })</pre> | n/a | yes |
+
+## Outputs
+
+No outputs.
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
@@ -0,0 +1,217 @@
+resource "aws_prometheus_rule_group_namespace" "recording_rules" {
+  count = var.pattern_config.enable_recording_rules ? 1 : 0
+
+  name         = "accelerator-istio-rules"
+  workspace_id = var.pattern_config.managed_prometheus_workspace_id
+  data         = <<EOF
+ groups:
+  - name: "istio.recording-rules"
+    interval: 5s
+    rules:
+    - record: "workload:istio_requests_total"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_requests_total)
+
+    - record: "workload:istio_request_duration_milliseconds_count"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_count)
+
+    - record: "workload:istio_request_duration_milliseconds_sum"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_sum)
+
+    - record: "workload:istio_request_duration_milliseconds_bucket"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_bucket)
+
+    - record: "workload:istio_request_bytes_count"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_count)
+
+    - record: "workload:istio_request_bytes_sum"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_sum)
+
+    - record: "workload:istio_request_bytes_bucket"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_bucket)
+
+    - record: "workload:istio_response_bytes_count"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_count)
+
+    - record: "workload:istio_response_bytes_sum"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_sum)
+
+    - record: "workload:istio_response_bytes_bucket"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_bucket)
+
+    - record: "workload:istio_tcp_sent_bytes_total"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total)
+
+    - record: "workload:istio_tcp_received_bytes_total"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total)
+
+    - record: "workload:istio_tcp_connections_opened_total"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_connections_opened_total)
+
+    - record: "workload:istio_tcp_connections_closed_total"
+      expr: |
+        sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_connections_closed_total)
+EOF
+}
+
+resource "aws_prometheus_rule_group_namespace" "alerting_rules" {
+  count = var.pattern_config.enable_alerting_rules ? 1 : 0
+
+  name         = "accelerator-istio-alerting"
+  workspace_id = var.pattern_config.managed_prometheus_workspace_id
+  data         = <<EOF
+  groups:
+    - name: "istio.basic.alerting-rules"
+      rules:
+        - alert: IngressTrafficMissing
+          annotations:
+            summary: 'ingress gateway traffic missing'
+            description: '[Critical]: ingress gateway traffic missing, likely other monitors are misleading, check client logs'
+          expr: >
+              absent(istio_requests_total{destination_service_namespace=~"service-graph.*",reporter="source",source_workload="istio-ingressgateway"})==1
+          for: 5m
+        - alert: IstioMetricsMissing
+          annotations:
+            summary: 'Istio Metrics missing'
+            description: '[Critical]: Check prometheus deployment or whether the prometheus filters are applied correctly'
+          expr: >
+            absent(istio_request_total)==1 or absent(istio_request_duration_milliseconds_bucket)==1
+          for: 5m
+    - name: "istio.workload.alerting-rules"
+      rules:
+        - alert: HTTP5xxRateHigh
+          annotations:
+            summary: '5xx rate too high'
+            description: 'The HTTP 5xx errors rate higher than 0.05 in 5 mins'
+          expr: >
+            sum(irate(istio_requests_total{reporter="destination", response_code=~"5.*"}[5m])) / sum(irate(istio_requests_total{reporter="destination"}[5m])) > 0.05
+          for: 5m
+        - alert: WorkloadLatencyP99High
+          expr: histogram_quantile(0.99, sum(irate(istio_request_duration_milliseconds_bucket{source_workload=~"svc.*"}[5m])) by (source_workload,namespace, le)) > 160
+          for: 10m
+          annotations:
+            description: 'The workload request latency P99 > 160ms '
+            message:  "Request duration has slowed down for workload: {{`{{$labels.source_workload}}`}} in namespace: {{`{{$labels.namespace}}`}}. Response duration is {{`{{$value}}`}} milliseconds"
+        - alert: IngressLatencyP99High
+          expr: histogram_quantile(0.99, sum(irate(istio_request_duration_milliseconds_bucket{source_workload=~"istio.*"}[5m])) by (source_workload,namespace, le)) > 250
+          for: 10m
+          annotations:
+            description: 'The ingress latency P99 > 250ms '
+            message:  "Request duration has slowed down for ingress: {{`{{$labels.source_workload}}`}} in namespace: {{`{{$labels.namespace}}`}}. Response duration is {{`{{$value}}`}} milliseconds"
+    - name: "istio.infra.alerting-rules"
+      rules:
+        - alert: ProxyContainerCPUUsageHigh
+          expr: (sum(rate(container_cpu_usage_seconds_total{namespace!="kube-system", container=~"istio-proxy", namespace!=""}[5m])) BY (namespace, pod, container) * 100) > 80
+          for: 5m
+          annotations:
+            summary: "Proxy Container CPU usage (namespace {{ $labels.namespace }}) (pod {{ $labels.pod }}) (container {{ $labels.container }})  VALUE = {{ $value }}\n"
+            description: "Proxy Container CPU usage is above 80%"
+        - alert: ProxyContainerMemoryUsageHigh
+          expr: (sum(container_memory_working_set_bytes{namespace!="kube-system", container=~"istio-proxy", namespace!=""}) BY (container, pod, namespace)  / (sum(container_spec_memory_limit_bytes{namespace!="kube-system", container!="POD"}) BY (container, pod, namespace) > 0)* 100) > 80
+          for: 5m
+          annotations:
+            summary: "Proxy Container Memory usage (namespace {{ $labels.namespace }}) (pod {{ $labels.pod }}) (container {{ $labels.container }})  VALUE = {{ $value }}\n"
+            description: "Proxy Container Memory usage is above 80%"
+        - alert: IngressMemoryUsageIncreaseRateHigh
+          expr: avg(deriv(container_memory_working_set_bytes{container=~"istio-proxy",namespace="istio-system"}[60m])) > 200
+          for: 180m
+          annotations:
+            summary: "Ingress proxy Memory change rate, VALUE = {{ $value }}\n"
+            description: "Ingress proxy Memory Usage increases more than 200 Bytes/sec"
+        - alert: IstiodContainerCPUUsageHigh
+          expr: (sum(rate(container_cpu_usage_seconds_total{namespace="istio-system", container="discovery"}[5m])) BY (pod) * 100) > 80
+          for: 5m
+          annotations:
+            summary: "Istiod Container CPU usage (namespace {{ $labels.namespace }}) (pod {{ $labels.pod }}) (container {{ $labels.container }}) VALUE = {{ $value }}\n"
+            description: "Isitod Container CPU usage is above 80%"
+        - alert: IstiodMemoryUsageHigh
+          expr: (sum(container_memory_working_set_bytes{namespace="istio-system", container="discovery"}) BY (pod)  / (sum(container_spec_memory_limit_bytes{namespace="istio-system", container="discovery"}) BY (pod) > 0)* 100) > 80
+          for: 5m
+          annotations:
+            summary: "Istiod Container Memory usage (namespace {{ $labels.namespace }}) (pod {{ $labels.pod }}) (container {{ $labels.container }}) VALUE = {{ $value }}\n"
+            description: "Istiod Container Memory usage is above 80%"
+        - alert: IstiodMemoryUsageIncreaseRateHigh
+          expr: sum(deriv(container_memory_working_set_bytes{namespace="istio-system",pod=~"istiod-.*"}[60m])) > 1000
+          for: 300m
+          annotations:
+            summary: "Istiod Container Memory usage increase rate high, VALUE = {{ $value }}\n"
+            description: "Istiod Container Memory usage increases more than 1k Bytes/sec"
+    - name: "istio.controlplane.alerting-rules"
+      rules:
+        - alert: IstiodxdsPushErrorsHigh
+          annotations:
+            summary: 'istiod push errors is too high'
+            description: 'istiod push error rate is higher than 0.05'
+          expr: >
+            sum(irate(pilot_xds_push_errors{app="istiod"}[5m])) / sum(irate(pilot_xds_pushes{app="istiod"}[5m])) > 0.05
+          for: 5m
+        - alert: IstiodxdsRejectHigh
+          annotations:
+            summary: 'istiod rejects rate is too high'
+            description: 'istiod rejects rate is higher than 0.05'
+          expr: >
+            sum(irate(pilot_total_xds_rejects{app="istiod"}[5m])) / sum(irate(pilot_xds_pushes{app="istiod"}[5m])) > 0.05
+          for: 5m
+        - alert: IstiodContainerNotReady
+          annotations:
+            summary: 'istiod container not ready'
+            description: 'container: discovery not running'
+          expr: >
+            kube_pod_container_status_running{namespace="istio-system", container="discovery", component=""} == 0
+          for: 5m
+        - alert: IstiodUnavailableReplica
+          annotations:
+            summary: 'Istiod unavailable pod'
+            description: 'Istiod unavailable replica > 0'
+          expr: >
+            kube_deployment_status_replicas_unavailable{deployment="istiod", component=""} > 0
+          for: 5m
+        - alert: Ingress200RateLow
+          annotations:
+            summary: 'ingress gateway 200 rate drops'
+            description: 'The expected rate is 100 per ns, the limit is set based on 15ns'
+          expr: >
+            sum(rate(istio_requests_total{reporter="source", source_workload="istio-ingressgateway",response_code="200",destination_service_namespace=~"service-graph.*"}[5m])) < 1490
+          for: 30m
+    EOF
+}
+
+resource "kubectl_manifest" "flux_kustomization" {
+  count = var.pattern_config.enable_dashboards ? 1 : 0
+
+  yaml_body = <<YAML
+apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
+kind: Kustomization
+metadata:
+  name: ${var.pattern_config.flux_kustomization_name}
+  namespace: flux-system
+spec:
+  interval: 1m0s
+  path: ${var.pattern_config.flux_kustomization_path}
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: ${var.pattern_config.flux_gitrepository_name}
+  postBuild:
+    substitute:
+      AMG_AWS_REGION: ${var.pattern_config.managed_prometheus_workspace_region}
+      AMP_ENDPOINT_URL: ${var.pattern_config.managed_prometheus_workspace_endpoint}
+      AMG_ENDPOINT_URL: ${var.pattern_config.grafana_url}
+      GRAFANA_ISTIO_CP_DASH_URL: ${var.pattern_config.grafana_istio_cp_dashboard_url}
+      GRAFANA_ISTIO_MESH_DASH_URL: ${var.pattern_config.grafana_istio_mesh_dashboard_url}
+      GRAFANA_ISTIO_PERF_DASH_URL: ${var.pattern_config.grafana_istio_performance_dashboard_url}
+      GRAFANA_ISTIO_SERVICE_DASH_URL: ${var.pattern_config.grafana_istio_service_dashboard_url}
+YAML
+}
@@ -0,0 +1,29 @@
+variable "pattern_config" {
+  description = "Configuration object for ISTIO monitoring"
+  type = object({
+    enable_alerting_rules  = bool
+    enable_recording_rules = bool
+    scrape_sample_limit    = number
+
+    enable_recording_rules = bool
+
+    enable_dashboards = bool
+
+    flux_gitrepository_name   = string
+    flux_gitrepository_url    = string
+    flux_gitrepository_branch = string
+    flux_kustomization_name   = string
+    flux_kustomization_path   = string
+
+    managed_prometheus_workspace_id       = string
+    managed_prometheus_workspace_region   = string
+    managed_prometheus_workspace_endpoint = string
+
+    grafana_url                             = string
+    grafana_istio_cp_dashboard_url          = string
+    grafana_istio_mesh_dashboard_url        = string
+    grafana_istio_performance_dashboard_url = string
+    grafana_istio_service_dashboard_url     = string
+  })
+  nullable = false
+}
@@ -0,0 +1,22 @@
+terraform {
+  required_version = ">= 1.1.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 4.0.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.10"
+    }
+    kubectl = {
+      source  = "gavinbunney/kubectl"
+      version = ">= 1.14"
+    }
+    helm = {
+      source  = "hashicorp/helm"
+      version = ">= 2.4.1"
+    }
+  }
+}
@@ -305,6 +305,40 @@ variable "nginx_config" {
  default = null
 }

+variable "enable_istio" {
+  description = "Enable ISTIO workloads monitoring, alerting and default dashboards"
+  type        = bool
+  default     = false
+}
+
+
+variable "istio_config" {
+  description = "Configuration object for ISTIO monitoring"
+  type = object({
+    enable_alerting_rules  = bool
+    enable_recording_rules = bool
+    enable_dashboards      = bool
+    scrape_sample_limit    = number
+
+    flux_gitrepository_name   = string
+    flux_gitrepository_url    = string
+    flux_gitrepository_branch = string
+    flux_kustomization_name   = string
+    flux_kustomization_path   = string
+
+    grafana_url                             = string
+    grafana_istio_cp_dashboard_url          = string
+    grafana_istio_mesh_dashboard_url        = string
+    grafana_istio_performance_dashboard_url = string
+    grafana_istio_service_dashboard_url     = string
+
+    prometheus_metrics_endpoint = string
+  })
+
+  # defaults are pre-computed in locals.tf, provide a full definition to override
+  default = null
+}
+
 variable "enable_logs" {
  description = "Using AWS For FluentBit to collect cluster and application logs to Amazon CloudWatch"
  type        = bool