diff --git a/docs/eks/multicluster.md b/docs/eks/multicluster.md new file mode 100644 index 0000000..48bd1cc --- /dev/null +++ b/docs/eks/multicluster.md @@ -0,0 +1,96 @@ +# AWS EKS Multicluster Observability + +This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with more than one EKS cluster and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace. + +## Prerequisites + +#### 1. EKS clusters + +Using the example [eks-cluster-with-vpc](../../examples/eks-cluster-with-vpc/), create two EKS clusters with the names: + 1. `eks-cluster-1` + 2. `eks-cluster-2` + +#### 2. Amazon Managed Serivce for Prometheus (AMP) workspace + +We recommend that you create a new AMP workspace. To do that you can run the following command. + +Ensure you have the following necessary IAM permissions +* `aps.CreateWorkspace` + +```sh +export TF_VAR_managed_prometheus_workspace_id=$(aws amp create-workspace --alias observability-accelerator --query='workspaceId' --output text) +``` + +#### 3. Amazon Managed Grafana (AMG) workspace + +To run this example you need an AMG workspace. If you have +an existing workspace, create an environment variable as described below. +To create a new workspace, visit our supporting example for managed Grafana. + +!!! note + For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz` + +```sh +export TF_VAR_managed_grafana_workspace_id=g-xxx +``` + +#### 4. Grafana API Key + +AMG provides a control plane API for generating Grafana API keys. +As a security best practice, we will provide to Terraform a short lived API key to +run the `apply` or `destroy` command. + +Ensure you have the following necessary IAM permissions +* `grafana.CreateWorkspaceApiKey` +* `grafana.DeleteWorkspaceApiKey` + +```sh +export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text` +``` + +## Setup + +#### 1. Download sources and initialize Terraform + +```sh +git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git +cd terraform-aws-observability-accelerator/examples/eks-multicluster +terraform init +``` + +#### 2. Deploy + +Verify by looking at the file `variables.tf` that there are two EKS clusters targeted for deployment by the names/ids: +1. `eks-cluster-1` +2. `eks-cluster-2` + +The difference in deployment between these clusters is that Terraform, when setting up the EKS cluster behind variable `eks_cluster_1_id` for observability, also sets up: +* Dashboard folder and files in `AMG` +* Prometheus and Java, alerting and recording rules in `AMP` + +!!! warning + To override the defaults, create a `terraform.tfvars` and change the default values of the variables. + +Run the following command to deploy + +```sh +terraform apply --auto-approve +``` + +## Verifying Multicluster Observability + +One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace. + +Note how you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster. + +eks-multicluster-1 + +eks-multicluster-2 + +## Cleanup + +To clean up entirely, run the following command: + +```sh +terraform destroy --auto-approve +``` diff --git a/examples/eks-multicluster/README.md b/examples/eks-multicluster/README.md new file mode 100644 index 0000000..48bd1cc --- /dev/null +++ b/examples/eks-multicluster/README.md @@ -0,0 +1,96 @@ +# AWS EKS Multicluster Observability + +This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with more than one EKS cluster and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace. + +## Prerequisites + +#### 1. EKS clusters + +Using the example [eks-cluster-with-vpc](../../examples/eks-cluster-with-vpc/), create two EKS clusters with the names: + 1. `eks-cluster-1` + 2. `eks-cluster-2` + +#### 2. Amazon Managed Serivce for Prometheus (AMP) workspace + +We recommend that you create a new AMP workspace. To do that you can run the following command. + +Ensure you have the following necessary IAM permissions +* `aps.CreateWorkspace` + +```sh +export TF_VAR_managed_prometheus_workspace_id=$(aws amp create-workspace --alias observability-accelerator --query='workspaceId' --output text) +``` + +#### 3. Amazon Managed Grafana (AMG) workspace + +To run this example you need an AMG workspace. If you have +an existing workspace, create an environment variable as described below. +To create a new workspace, visit our supporting example for managed Grafana. + +!!! note + For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz` + +```sh +export TF_VAR_managed_grafana_workspace_id=g-xxx +``` + +#### 4. Grafana API Key + +AMG provides a control plane API for generating Grafana API keys. +As a security best practice, we will provide to Terraform a short lived API key to +run the `apply` or `destroy` command. + +Ensure you have the following necessary IAM permissions +* `grafana.CreateWorkspaceApiKey` +* `grafana.DeleteWorkspaceApiKey` + +```sh +export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text` +``` + +## Setup + +#### 1. Download sources and initialize Terraform + +```sh +git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git +cd terraform-aws-observability-accelerator/examples/eks-multicluster +terraform init +``` + +#### 2. Deploy + +Verify by looking at the file `variables.tf` that there are two EKS clusters targeted for deployment by the names/ids: +1. `eks-cluster-1` +2. `eks-cluster-2` + +The difference in deployment between these clusters is that Terraform, when setting up the EKS cluster behind variable `eks_cluster_1_id` for observability, also sets up: +* Dashboard folder and files in `AMG` +* Prometheus and Java, alerting and recording rules in `AMP` + +!!! warning + To override the defaults, create a `terraform.tfvars` and change the default values of the variables. + +Run the following command to deploy + +```sh +terraform apply --auto-approve +``` + +## Verifying Multicluster Observability + +One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace. + +Note how you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster. + +eks-multicluster-1 + +eks-multicluster-2 + +## Cleanup + +To clean up entirely, run the following command: + +```sh +terraform destroy --auto-approve +``` diff --git a/examples/eks-multicluster/data.tf b/examples/eks-multicluster/data.tf new file mode 100644 index 0000000..acc5d55 --- /dev/null +++ b/examples/eks-multicluster/data.tf @@ -0,0 +1,19 @@ +data "aws_eks_cluster_auth" "eks_cluster_1" { + name = var.eks_cluster_1_id + provider = aws.eks_cluster_1 +} + +data "aws_eks_cluster_auth" "eks_cluster_2" { + name = var.eks_cluster_2_id + provider = aws.eks_cluster_2 +} + +data "aws_eks_cluster" "eks_cluster_1" { + name = var.eks_cluster_1_id + provider = aws.eks_cluster_1 +} + +data "aws_eks_cluster" "eks_cluster_2" { + name = var.eks_cluster_2_id + provider = aws.eks_cluster_2 +} diff --git a/examples/eks-multicluster/main.tf b/examples/eks-multicluster/main.tf new file mode 100644 index 0000000..e0d08a3 --- /dev/null +++ b/examples/eks-multicluster/main.tf @@ -0,0 +1,102 @@ +module "aws_observability_accelerator" { + source = "../../../terraform-aws-observability-accelerator" + aws_region = var.eks_cluster_1_region + enable_managed_prometheus = false + enable_alertmanager = true + create_dashboard_folder = true + create_prometheus_data_source = true + grafana_api_key = var.grafana_api_key + managed_prometheus_workspace_region = null + managed_prometheus_workspace_id = var.managed_prometheus_workspace_id + managed_grafana_workspace_id = var.managed_grafana_workspace_id + + providers = { + aws = aws.eks_cluster_1 + } +} + +module "eks_cluster_1_monitoring" { + source = "../../../terraform-aws-observability-accelerator//modules/eks-monitoring" + eks_cluster_id = var.eks_cluster_1_id + enable_amazon_eks_adot = true + enable_cert_manager = true + enable_java = true + + # This configuration section results in actions performed on AMG and AMP; and it needs to be done just once + # And hence, this in performed in conjunction with the setup of the eks_cluster_1 EKS cluster + enable_dashboards = true + enable_alerting_rules = true + enable_recording_rules = true + + grafana_api_key = var.grafana_api_key + dashboards_folder_id = module.aws_observability_accelerator.grafana_dashboards_folder_id + managed_prometheus_workspace_id = module.aws_observability_accelerator.managed_prometheus_workspace_id + managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint + managed_prometheus_workspace_region = module.aws_observability_accelerator.managed_prometheus_workspace_region + + java_config = { + enable_alerting_rules = true + enable_recording_rules = true + scrape_sample_limit = 1 + } + + prometheus_config = { + global_scrape_interval = "60s" + global_scrape_timeout = "15s" + scrape_sample_limit = 2000 + } + + providers = { + aws = aws.eks_cluster_1 + kubernetes = kubernetes.eks_cluster_1 + helm = helm.eks_cluster_1 + grafana = grafana + } + + depends_on = [ + module.aws_observability_accelerator + ] +} + +module "eks_cluster_2_monitoring" { + source = "../../../terraform-aws-observability-accelerator//modules/eks-monitoring" + eks_cluster_id = var.eks_cluster_2_id + enable_amazon_eks_adot = true + enable_cert_manager = true + enable_java = true + + # Since the following were enabled in conjunction with the set up of the eks_cluster_1 EKS cluster, we will skip + # them with the eks_cluster_2 EKS cluster + enable_dashboards = false + enable_alerting_rules = false + enable_recording_rules = false + + grafana_api_key = var.grafana_api_key + dashboards_folder_id = module.aws_observability_accelerator.grafana_dashboards_folder_id + managed_prometheus_workspace_id = module.aws_observability_accelerator.managed_prometheus_workspace_id + managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint + managed_prometheus_workspace_region = module.aws_observability_accelerator.managed_prometheus_workspace_region + + java_config = { + enable_alerting_rules = false # addressed while setting up the eks_cluster_1 EKS cluster + enable_recording_rules = false # addressed while setting up the eks_cluster_1 EKS cluster + scrape_sample_limit = 1 + } + + prometheus_config = { + global_scrape_interval = "60s" + global_scrape_timeout = "15s" + scrape_sample_limit = 2000 + } + + providers = { + aws = aws.eks_cluster_2 + kubernetes = kubernetes.eks_cluster_2 + helm = helm.eks_cluster_2 + grafana = grafana + } + + depends_on = [ + module.aws_observability_accelerator + ] +} diff --git a/examples/eks-multicluster/outputs.tf b/examples/eks-multicluster/outputs.tf new file mode 100644 index 0000000..e69de29 diff --git a/examples/eks-multicluster/providers.tf b/examples/eks-multicluster/providers.tf new file mode 100644 index 0000000..a84abec --- /dev/null +++ b/examples/eks-multicluster/providers.tf @@ -0,0 +1,46 @@ +provider "kubernetes" { + host = data.aws_eks_cluster.eks_cluster_1.endpoint + cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_1.certificate_authority[0].data) + token = data.aws_eks_cluster_auth.eks_cluster_1.token + alias = "eks_cluster_1" +} + +provider "kubernetes" { + host = data.aws_eks_cluster.eks_cluster_2.endpoint + cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_2.certificate_authority[0].data) + token = data.aws_eks_cluster_auth.eks_cluster_2.token + alias = "eks_cluster_2" +} + +provider "helm" { + kubernetes { + host = data.aws_eks_cluster.eks_cluster_1.endpoint + cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_1.certificate_authority[0].data) + token = data.aws_eks_cluster_auth.eks_cluster_1.token + } + alias = "eks_cluster_1" +} + +provider "helm" { + kubernetes { + host = data.aws_eks_cluster.eks_cluster_2.endpoint + cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_2.certificate_authority[0].data) + token = data.aws_eks_cluster_auth.eks_cluster_2.token + } + alias = "eks_cluster_2" +} + +provider "aws" { + region = var.eks_cluster_1_region + alias = "eks_cluster_1" +} + +provider "aws" { + region = var.eks_cluster_2_region + alias = "eks_cluster_2" +} + +provider "grafana" { + url = module.aws_observability_accelerator.managed_grafana_workspace_endpoint + auth = var.grafana_api_key +} diff --git a/examples/eks-multicluster/variables.tf b/examples/eks-multicluster/variables.tf new file mode 100644 index 0000000..29561d1 --- /dev/null +++ b/examples/eks-multicluster/variables.tf @@ -0,0 +1,46 @@ +variable "eks_cluster_1_id" { + description = "Name or ID of the EKS cluster 1" + type = string + default = "eks-cluster-1" + nullable = false +} + +variable "eks_cluster_1_region" { + description = "AWS region of the EKS cluster 1" + type = string + default = "us-west-2" + nullable = false +} + +variable "eks_cluster_2_id" { + description = "Name or ID of the EKS cluster 2" + type = string + default = "eks-cluster-2" + nullable = true +} + +variable "eks_cluster_2_region" { + description = "AWS region of the EKS cluster 2" + type = string + default = "us-west-2" + nullable = true +} + +variable "managed_prometheus_workspace_id" { + description = "Amazon Managed Service for Prometheus Workspace ID" + type = string + default = "" +} + +variable "managed_grafana_workspace_id" { + description = "Amazon Managed Grafana Workspace ID" + type = string + default = "" +} + +variable "grafana_api_key" { + description = "API key for authorizing the Grafana provider to make changes to Amazon Managed Grafana" + type = string + default = "" + sensitive = true +} diff --git a/examples/eks-multicluster/versions.tf b/examples/eks-multicluster/versions.tf new file mode 100644 index 0000000..2bb0686 --- /dev/null +++ b/examples/eks-multicluster/versions.tf @@ -0,0 +1,29 @@ +terraform { + required_version = ">= 1.3.9" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 4.55.0" + configuration_aliases = [aws.eks_cluster_1, aws.eks_cluster_2] + } + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.18.0" + configuration_aliases = [kubernetes.eks_cluster_1, kubernetes.eks_cluster_2] + } + helm = { + source = "hashicorp/helm" + version = ">= 2.9.0" + configuration_aliases = [helm.eks_cluster_1, helm.eks_cluster_2] + } + kubectl = { + source = "gavinbunney/kubectl" + version = ">= 1.14" + } + grafana = { + source = "grafana/grafana" + version = ">= 1.25.0" + } + } +}