EKS container insights (#204)

* adot-container-insight-tf-code by Rajat Omar

* Documentation and naming convention added

* PR review fixes

* PR CI pipeline fixes

* pr ci run fixes

* added the variables mentioned in ci build

* ci variable fixes

* changed the module name

---------

Co-authored-by: Omar <merajat@3c0630162a5a.ant.amazon.com>

docs/container-insights/eks.md

@@ -0,0 +1,59 @@
+# Setting Up Container Insights for your EKS Cluster
+
+This example deploys AWS Distro of OpenTelemetry on your EKS cluster as a Daemonset which will enable
+Container Insights metrics Dashboard on Amazon CloudWatch.
+
+
+## Prerequisites
+
+!!! note
+    Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.
+
+## Setup
+
+### 1. Download sources and initialize Terraform
+
+```
+git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
+cd terraform-aws-observability-accelerator/examples/eks-container-insights
+terraform init
+```
+
+### 2. AWS Region
+
+Specify the AWS Region where the resources will be deployed:
+
+```bash
+export TF_VAR_aws_region=xxx
+```
+### 2. EKS Cluster Name
+
+Specify the EKS Cluster Name where the resources will be deployed:
+
+```bash
+export TF_VAR_eks_cluster_id=xxx
+```
+
+## Deploy
+
+Simply run this command to deploy the example
+
+```bash
+terraform apply
+```
+
+## Visualization
+
+After apply, open your Amazon CloudWatch console in the same region as your EKS cluster, then from the left hand side choose `Insights -> Container Insights`, there choose the `Performance montoring` from the drop down, choose the `cluster name` and you will see the metrics shown on the dashboard:
+
+
+<img width="1423" alt="Screenshot 2023-08-08 at 1.15.14 PM" src="https://github.com/RJrocks/terraform-aws-observability-accelerator/assets/5756583/4c5e4ed3-2e1f-4d41-b568-01976fbfd303">
+
+
+## Cleanup
+
+To clean up your environment, destroy the Terraform example by running
+
+```sh
+terraform destroy
+```

examples/eks-container-insights/README.md

@@ -0,0 +1,54 @@
+# Enable Container Insights for EKS cluster
+
+This example deploys ADOT as a daemonset on your EKS cluster which enables Container Insights metrics on CloudWatch.
+
+Step-by-step instructions available on our [docs site](https://aws-observability.github.io/terraform-aws-observability-accelerator/)
+under **Amazon CloudWatch Container Insights**
+
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0.0 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |
+| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 1.14 |
+| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0.0 |
+
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_eks_container_insights"></a> [eks\_container\_insights](#module\_eks\_container\_insights) | ../../modules/eks-container-insights | n/a |
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
+| [aws_eks_cluster.eks_cluster](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/eks_cluster) | data source |
+| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_aws_region"></a> [aws\_region](#input\_aws\_region) | EKS cluster region | `string` | n/a | yes |
+| <a name="input_eks_cluster_id"></a> [eks\_cluster\_id](#input\_eks\_cluster\_id) | EKS cluster name | `string` | n/a | yes |
+| <a name="input_irsa_iam_permissions_boundary"></a> [irsa\_iam\_permissions\_boundary](#input\_irsa\_iam\_permissions\_boundary) | IAM permissions boundary for IRSA roles | `string` | `null` | no |
+| <a name="input_irsa_iam_role_path"></a> [irsa\_iam\_role\_path](#input\_irsa\_iam\_role\_path) | IAM role path for IRSA roles | `string` | `"/"` | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | Additional tags (e.g. `map('BusinessUnit`,`XYZ`) | `map(string)` | `{}` | no |
+
+## Outputs
+
+No outputs.
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

examples/eks-container-insights/locals.tf

@@ -0,0 +1,9 @@
+data "aws_partition" "current" {}
+
+data "aws_caller_identity" "current" {}
+
+data "aws_region" "current" {}
+
+data "aws_eks_cluster" "eks_cluster" {
+  name = var.eks_cluster_id
+}

examples/eks-container-insights/main.tf

@@ -0,0 +1,34 @@
+provider "aws" {
+  region = var.aws_region
+}
+
+provider "kubernetes" {
+  host                   = local.eks_cluster_endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    args        = ["eks", "get-token", "--cluster-name", var.eks_cluster_id]
+    command     = "aws"
+  }
+}
+
+provider "helm" {
+  kubernetes {
+    host                   = local.eks_cluster_endpoint
+    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
+    exec {
+      api_version = "client.authentication.k8s.io/v1beta1"
+      args        = ["eks", "get-token", "--cluster-name", var.eks_cluster_id]
+      command     = "aws"
+    }
+  }
+}
+
+
+# Deploy the ADOT Container Insights
+
+module "eks_container_insights" {
+  source = "../../modules/eks-container-insights"
+  # source = "github.com/aws-observability/terraform-aws-observability-accelerator//modules/eks-container-insights?ref=v2.5.4"
+  eks_cluster_id = var.eks_cluster_id
+}

examples/eks-container-insights/variables.tf

@@ -0,0 +1,27 @@
+variable "eks_cluster_id" {
+  description = "EKS cluster name"
+  type        = string
+}
+
+variable "aws_region" {
+  description = "EKS cluster region"
+  type        = string
+}
+
+variable "irsa_iam_role_path" {
+  description = "IAM role path for IRSA roles"
+  type        = string
+  default     = "/"
+}
+
+variable "irsa_iam_permissions_boundary" {
+  description = "IAM permissions boundary for IRSA roles"
+  type        = string
+  default     = null
+}
+
+variable "tags" {
+  description = "Additional tags (e.g. `map('BusinessUnit`,`XYZ`)"
+  type        = map(string)
+  default     = {}
+}

examples/eks-container-insights/versions.tf

@@ -0,0 +1,30 @@
+terraform {
+  required_version = ">= 1.1.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.0.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.10"
+    }
+    kubectl = {
+      source  = "gavinbunney/kubectl"
+      version = ">= 1.14"
+    }
+    helm = {
+      source  = "hashicorp/helm"
+      version = ">= 2.4.1"
+    }
+  }
+
+  # ##  Used for end-to-end testing on project; update to suit your needs
+  # backend "s3" {
+  #   bucket = "aws-observability-accelerator-terraform-states"
+  #   region = "us-west-2"
+  #   key    = "e2e/eks_container_insights/terraform.tfstate"
+  # }
+
+}

mkdocs.yml

@@ -35,6 +35,8 @@ nav:
       - Viewing logs: eks/logs.md
       - Tracing: eks/tracing.md
       - Teardown: eks/destroy.md
+  - Amazon CloudWatch Container Insights:
+      - Amazon EKS: container-insights/eks.md
   - Monitoring Managed Service for Prometheus Workspaces: workloads/managed-prometheus.md
   - Supporting Examples:
       - EKS Cluster with VPC: helpers/new-eks-cluster.md

modules/eks-container-insights/README.md

@@ -0,0 +1,61 @@
+# Container Insights ADOT implementation for EKS Cluster Observability
+
+This module provides an automated experience around enabling Container Insights for your EKS cluster using ADOT (AWS Distro for OpenTelemetry).
+It provides the following resources:
+
+- ADOT Collector Deployment to your EKS cluster
+- Enabling Container Insights on CloudWatch
+
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0.0 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |
+| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 1.14 |
+| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0.0 |
+
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_helm_addon"></a> [helm\_addon](#module\_helm\_addon) | github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons/helm-addon | v4.32.1 |
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
+| [aws_eks_cluster.eks_cluster](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/eks_cluster) | data source |
+| [aws_iam_policy.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy) | data source |
+| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_adot_otel_helm_chart_verison"></a> [adot\_otel\_helm\_chart\_verison](#input\_adot\_otel\_helm\_chart\_verison) | ADOT collector helm chart version | `string` | `"0.17.0"` | no |
+| <a name="input_eks_cluster_id"></a> [eks\_cluster\_id](#input\_eks\_cluster\_id) | EKS Cluster Id | `string` | n/a | yes |
+| <a name="input_helm_config"></a> [helm\_config](#input\_helm\_config) | Helm provider config for adot-exporter-for-eks-on-ec2 | `any` | `{}` | no |
+| <a name="input_irsa_iam_permissions_boundary"></a> [irsa\_iam\_permissions\_boundary](#input\_irsa\_iam\_permissions\_boundary) | IAM permissions boundary for IRSA roles | `string` | `null` | no |
+| <a name="input_irsa_iam_role_path"></a> [irsa\_iam\_role\_path](#input\_irsa\_iam\_role\_path) | IAM role path for IRSA roles | `string` | `"/"` | no |
+| <a name="input_irsa_policies"></a> [irsa\_policies](#input\_irsa\_policies) | Additional IAM policies for a IAM role for service accounts | `list(string)` | `[]` | no |
+| <a name="input_manage_via_gitops"></a> [manage\_via\_gitops](#input\_manage\_via\_gitops) | Determines if the add-on should be managed via GitOps. | `bool` | `false` | no |
+| <a name="input_service_exporters"></a> [service\_exporters](#input\_service\_exporters) | exporter for adot-ci setup | `string` | `"awsemf"` | no |
+| <a name="input_service_receivers"></a> [service\_receivers](#input\_service\_receivers) | receiver for adot-ci setup | `string` | `"awscontainerinsightreceiver"` | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | Additional tags (e.g. `map('BusinessUnit`,`XYZ`) | `map(string)` | `{}` | no |
+
+## Outputs
+
+No outputs.
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

modules/eks-container-insights/locals.tf

@@ -0,0 +1,78 @@
+data "aws_partition" "current" {}
+
+data "aws_caller_identity" "current" {}
+
+data "aws_region" "current" {}
+
+data "aws_eks_cluster" "eks_cluster" {
+  name = var.eks_cluster_id
+}
+
+data "aws_iam_policy" "irsa" {
+  arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
+}
+
+locals {
+  name            = "adot-exporter-for-eks-on-ec2"
+  service_account = try(var.helm_config.service_account, "${local.name}-sa")
+
+  set_values = [
+    {
+      name  = "serviceAccount.name"
+      value = local.service_account
+    },
+    {
+      name  = "serviceAccount.create"
+      value = false
+    }
+  ]
+  # https://github.com/aws-observability/aws-otel-helm-charts/tree/main/charts/adot-exporter-for-eks-on-ec2
+  default_helm_config = {
+    name        = local.name
+    chart       = "adot-exporter-for-eks-on-ec2"
+    repository  = "https://aws-observability.github.io/aws-otel-helm-charts"
+    version     = var.adot_otel_helm_chart_verison
+    namespace   = "amazon-metrics"
+    values      = local.default_helm_values
+    description = "ADOT Helm Chart Deployment Configuration for Container Insights"
+  }
+
+  helm_config = merge(
+    local.default_helm_config,
+    var.helm_config
+  )
+
+  default_helm_values = [templatefile("${path.module}/values.yaml", {
+    aws_region        = local.addon_context.aws_region_name
+    cluster_name      = local.addon_context.eks_cluster_id
+    service_receivers = format("[\"%s\"]", var.service_receivers)
+    service_exporters = format("[\"%s\"]", var.service_exporters)
+    service_account   = local.service_account
+  })]
+
+  irsa_config = {
+    kubernetes_namespace                = local.helm_config["namespace"]
+    kubernetes_service_account          = local.service_account
+    create_kubernetes_namespace         = try(local.helm_config["create_namespace"], true)
+    create_kubernetes_service_account   = true
+    create_service_account_secret_token = try(local.helm_config["create_service_account_secret_token"], false)
+    irsa_iam_policies                   = concat([data.aws_iam_policy.irsa.arn], var.irsa_policies)
+  }
+
+  eks_oidc_issuer_url = replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")
+
+  addon_context = {
+    aws_caller_identity_account_id = data.aws_caller_identity.current.account_id
+    aws_caller_identity_arn        = data.aws_caller_identity.current.arn
+    aws_eks_cluster_endpoint       = data.aws_eks_cluster.eks_cluster.endpoint
+    aws_partition_id               = data.aws_partition.current.partition
+    aws_region_name                = data.aws_region.current.name
+    eks_cluster_id                 = var.eks_cluster_id
+    eks_oidc_issuer_url            = replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")
+    eks_oidc_provider_arn          = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.eks_oidc_issuer_url}"
+    tags                           = var.tags
+    irsa_iam_role_path             = var.irsa_iam_role_path
+    irsa_iam_permissions_boundary  = var.irsa_iam_permissions_boundary
+  }
+
+}

modules/eks-container-insights/main.tf

@@ -0,0 +1,30 @@
+provider "kubernetes" {
+  host                   = data.aws_eks_cluster.eks_cluster.endpoint
+  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
+  exec {
+    api_version = "client.authentication.k8s.io/v1beta1"
+    args        = ["eks", "get-token", "--cluster-name", local.addon_context.eks_cluster_id]
+    command     = "aws"
+  }
+}
+
+provider "helm" {
+  kubernetes {
+    host                   = data.aws_eks_cluster.eks_cluster.endpoint
+    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)
+    exec {
+      api_version = "client.authentication.k8s.io/v1beta1"
+      args        = ["eks", "get-token", "--cluster-name", local.addon_context.eks_cluster_id]
+      command     = "aws"
+    }
+  }
+}
+
+module "helm_addon" {
+  source            = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons/helm-addon?ref=v4.32.1"
+  manage_via_gitops = var.manage_via_gitops
+  set_values        = local.set_values
+  helm_config       = local.helm_config
+  irsa_config       = local.irsa_config
+  addon_context     = local.addon_context
+}

modules/eks-container-insights/values.yaml

@@ -0,0 +1,20 @@
+
+awsRegion: ${aws_region}
+clusterName: ${cluster_name}
+
+
+adotCollector:
+  daemonSet:
+    serviceAccount:
+      create: false
+      name: ${service_account}
+    createNamespace: false
+    extensions:
+      sigv4auth:
+        region: ${aws_region}
+    service:
+      metrics:
+        receivers: ${service_receivers}
+        exporters: ${service_exporters}
+  sidecar:
+    regionS3: ${aws_region}

modules/eks-container-insights/variables.tf

@@ -0,0 +1,58 @@
+variable "helm_config" {
+  description = "Helm provider config for adot-exporter-for-eks-on-ec2"
+  type        = any
+  default     = {}
+}
+
+variable "manage_via_gitops" {
+  type        = bool
+  description = "Determines if the add-on should be managed via GitOps."
+  default     = false
+}
+
+variable "service_receivers" {
+  type        = string
+  description = "receiver for adot-ci setup"
+  default     = "awscontainerinsightreceiver"
+}
+
+variable "service_exporters" {
+  type        = string
+  description = "exporter for adot-ci setup"
+  default     = "awsemf"
+}
+
+variable "irsa_policies" {
+  description = "Additional IAM policies for a IAM role for service accounts"
+  type        = list(string)
+  default     = []
+}
+
+variable "eks_cluster_id" {
+  description = "EKS Cluster Id"
+  type        = string
+}
+
+variable "adot_otel_helm_chart_verison" {
+  description = "ADOT collector helm chart version"
+  type        = string
+  default     = "0.17.0"
+}
+
+variable "tags" {
+  description = "Additional tags (e.g. `map('BusinessUnit`,`XYZ`)"
+  type        = map(string)
+  default     = {}
+}
+
+variable "irsa_iam_role_path" {
+  description = "IAM role path for IRSA roles"
+  type        = string
+  default     = "/"
+}
+
+variable "irsa_iam_permissions_boundary" {
+  description = "IAM permissions boundary for IRSA roles"
+  type        = string
+  default     = null
+}

modules/eks-container-insights/versions.tf

@@ -0,0 +1,22 @@
+terraform {
+  required_version = ">= 1.1.0"
+
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.0.0"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = ">= 2.10"
+    }
+    kubectl = {
+      source  = "gavinbunney/kubectl"
+      version = ">= 1.14"
+    }
+    helm = {
+      source  = "hashicorp/helm"
+      version = ">= 2.4.1"
+    }
+  }
+}