mirror of
https://github.com/storytold/terraform-aws-observability-accelerator.git
synced 2026-10-29 08:56:49 +00:00
Addition of EKS multi-cluster observability example (#155)
* Adding EKS multicluster observability example * EKS multicluster example - precommit fix * Corrected the path to an example * Made the multicluster example simpler * Pre-commit changes * Saved the images to Github and linked them in docs * Comments and language edits * Fix trailing spaces * Non-controversial naming and simpler variables * Added region to the data gathering * Formatting changes --------- Co-authored-by: Rodrigue Koffi <bonclay7@users.noreply.github.com>
This commit is contained in:
Vijay Chintalapati
committed by
GitHub
GitHub
parent
5b52c7b8e1
commit
b84e9b800f
@@ -0,0 +1,96 @@
|
|||||||
|
# AWS EKS Multicluster Observability
|
||||||
|
|
||||||
|
This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with more than one EKS cluster and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
#### 1. EKS clusters
|
||||||
|
|
||||||
|
Using the example [eks-cluster-with-vpc](../../examples/eks-cluster-with-vpc/), create two EKS clusters with the names:
|
||||||
|
1. `eks-cluster-1`
|
||||||
|
2. `eks-cluster-2`
|
||||||
|
|
||||||
|
#### 2. Amazon Managed Serivce for Prometheus (AMP) workspace
|
||||||
|
|
||||||
|
We recommend that you create a new AMP workspace. To do that you can run the following command.
|
||||||
|
|
||||||
|
Ensure you have the following necessary IAM permissions
|
||||||
|
* `aps.CreateWorkspace`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export TF_VAR_managed_prometheus_workspace_id=$(aws amp create-workspace --alias observability-accelerator --query='workspaceId' --output text)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. Amazon Managed Grafana (AMG) workspace
|
||||||
|
|
||||||
|
To run this example you need an AMG workspace. If you have
|
||||||
|
an existing workspace, create an environment variable as described below.
|
||||||
|
To create a new workspace, visit our supporting example for managed Grafana.
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4. Grafana API Key
|
||||||
|
|
||||||
|
AMG provides a control plane API for generating Grafana API keys.
|
||||||
|
As a security best practice, we will provide to Terraform a short lived API key to
|
||||||
|
run the `apply` or `destroy` command.
|
||||||
|
|
||||||
|
Ensure you have the following necessary IAM permissions
|
||||||
|
* `grafana.CreateWorkspaceApiKey`
|
||||||
|
* `grafana.DeleteWorkspaceApiKey`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||||
|
```
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
#### 1. Download sources and initialize Terraform
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
|
||||||
|
cd terraform-aws-observability-accelerator/examples/eks-multicluster
|
||||||
|
terraform init
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. Deploy
|
||||||
|
|
||||||
|
Verify by looking at the file `variables.tf` that there are two EKS clusters targeted for deployment by the names/ids:
|
||||||
|
1. `eks-cluster-1`
|
||||||
|
2. `eks-cluster-2`
|
||||||
|
|
||||||
|
The difference in deployment between these clusters is that Terraform, when setting up the EKS cluster behind variable `eks_cluster_1_id` for observability, also sets up:
|
||||||
|
* Dashboard folder and files in `AMG`
|
||||||
|
* Prometheus and Java, alerting and recording rules in `AMP`
|
||||||
|
|
||||||
|
!!! warning
|
||||||
|
To override the defaults, create a `terraform.tfvars` and change the default values of the variables.
|
||||||
|
|
||||||
|
Run the following command to deploy
|
||||||
|
|
||||||
|
```sh
|
||||||
|
terraform apply --auto-approve
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verifying Multicluster Observability
|
||||||
|
|
||||||
|
One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace.
|
||||||
|
|
||||||
|
Note how you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster.
|
||||||
|
|
||||||
|
<img width="2557" alt="eks-multicluster-1" src="https://user-images.githubusercontent.com/4762573/233949110-ce275d06-7ad8-494c-b527-d9c2a0fb6645.png">
|
||||||
|
|
||||||
|
<img width="2560" alt="eks-multicluster-2" src="https://user-images.githubusercontent.com/4762573/233949227-f401f81e-e0d6-4242-96ad-0bcd39ad4e2d.png">
|
||||||
|
|
||||||
|
## Cleanup
|
||||||
|
|
||||||
|
To clean up entirely, run the following command:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
terraform destroy --auto-approve
|
||||||
|
```
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# AWS EKS Multicluster Observability
|
||||||
|
|
||||||
|
This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with more than one EKS cluster and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
#### 1. EKS clusters
|
||||||
|
|
||||||
|
Using the example [eks-cluster-with-vpc](../../examples/eks-cluster-with-vpc/), create two EKS clusters with the names:
|
||||||
|
1. `eks-cluster-1`
|
||||||
|
2. `eks-cluster-2`
|
||||||
|
|
||||||
|
#### 2. Amazon Managed Serivce for Prometheus (AMP) workspace
|
||||||
|
|
||||||
|
We recommend that you create a new AMP workspace. To do that you can run the following command.
|
||||||
|
|
||||||
|
Ensure you have the following necessary IAM permissions
|
||||||
|
* `aps.CreateWorkspace`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export TF_VAR_managed_prometheus_workspace_id=$(aws amp create-workspace --alias observability-accelerator --query='workspaceId' --output text)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. Amazon Managed Grafana (AMG) workspace
|
||||||
|
|
||||||
|
To run this example you need an AMG workspace. If you have
|
||||||
|
an existing workspace, create an environment variable as described below.
|
||||||
|
To create a new workspace, visit our supporting example for managed Grafana.
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4. Grafana API Key
|
||||||
|
|
||||||
|
AMG provides a control plane API for generating Grafana API keys.
|
||||||
|
As a security best practice, we will provide to Terraform a short lived API key to
|
||||||
|
run the `apply` or `destroy` command.
|
||||||
|
|
||||||
|
Ensure you have the following necessary IAM permissions
|
||||||
|
* `grafana.CreateWorkspaceApiKey`
|
||||||
|
* `grafana.DeleteWorkspaceApiKey`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||||
|
```
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
#### 1. Download sources and initialize Terraform
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
|
||||||
|
cd terraform-aws-observability-accelerator/examples/eks-multicluster
|
||||||
|
terraform init
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. Deploy
|
||||||
|
|
||||||
|
Verify by looking at the file `variables.tf` that there are two EKS clusters targeted for deployment by the names/ids:
|
||||||
|
1. `eks-cluster-1`
|
||||||
|
2. `eks-cluster-2`
|
||||||
|
|
||||||
|
The difference in deployment between these clusters is that Terraform, when setting up the EKS cluster behind variable `eks_cluster_1_id` for observability, also sets up:
|
||||||
|
* Dashboard folder and files in `AMG`
|
||||||
|
* Prometheus and Java, alerting and recording rules in `AMP`
|
||||||
|
|
||||||
|
!!! warning
|
||||||
|
To override the defaults, create a `terraform.tfvars` and change the default values of the variables.
|
||||||
|
|
||||||
|
Run the following command to deploy
|
||||||
|
|
||||||
|
```sh
|
||||||
|
terraform apply --auto-approve
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verifying Multicluster Observability
|
||||||
|
|
||||||
|
One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace.
|
||||||
|
|
||||||
|
Note how you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster.
|
||||||
|
|
||||||
|
<img width="2557" alt="eks-multicluster-1" src="https://user-images.githubusercontent.com/4762573/233949110-ce275d06-7ad8-494c-b527-d9c2a0fb6645.png">
|
||||||
|
|
||||||
|
<img width="2560" alt="eks-multicluster-2" src="https://user-images.githubusercontent.com/4762573/233949227-f401f81e-e0d6-4242-96ad-0bcd39ad4e2d.png">
|
||||||
|
|
||||||
|
## Cleanup
|
||||||
|
|
||||||
|
To clean up entirely, run the following command:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
terraform destroy --auto-approve
|
||||||
|
```
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
data "aws_eks_cluster_auth" "eks_cluster_1" {
|
||||||
|
name = var.eks_cluster_1_id
|
||||||
|
provider = aws.eks_cluster_1
|
||||||
|
}
|
||||||
|
|
||||||
|
data "aws_eks_cluster_auth" "eks_cluster_2" {
|
||||||
|
name = var.eks_cluster_2_id
|
||||||
|
provider = aws.eks_cluster_2
|
||||||
|
}
|
||||||
|
|
||||||
|
data "aws_eks_cluster" "eks_cluster_1" {
|
||||||
|
name = var.eks_cluster_1_id
|
||||||
|
provider = aws.eks_cluster_1
|
||||||
|
}
|
||||||
|
|
||||||
|
data "aws_eks_cluster" "eks_cluster_2" {
|
||||||
|
name = var.eks_cluster_2_id
|
||||||
|
provider = aws.eks_cluster_2
|
||||||
|
}
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
module "aws_observability_accelerator" {
|
||||||
|
source = "../../../terraform-aws-observability-accelerator"
|
||||||
|
aws_region = var.eks_cluster_1_region
|
||||||
|
enable_managed_prometheus = false
|
||||||
|
enable_alertmanager = true
|
||||||
|
create_dashboard_folder = true
|
||||||
|
create_prometheus_data_source = true
|
||||||
|
grafana_api_key = var.grafana_api_key
|
||||||
|
managed_prometheus_workspace_region = null
|
||||||
|
managed_prometheus_workspace_id = var.managed_prometheus_workspace_id
|
||||||
|
managed_grafana_workspace_id = var.managed_grafana_workspace_id
|
||||||
|
|
||||||
|
providers = {
|
||||||
|
aws = aws.eks_cluster_1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module "eks_cluster_1_monitoring" {
|
||||||
|
source = "../../../terraform-aws-observability-accelerator//modules/eks-monitoring"
|
||||||
|
eks_cluster_id = var.eks_cluster_1_id
|
||||||
|
enable_amazon_eks_adot = true
|
||||||
|
enable_cert_manager = true
|
||||||
|
enable_java = true
|
||||||
|
|
||||||
|
# This configuration section results in actions performed on AMG and AMP; and it needs to be done just once
|
||||||
|
# And hence, this in performed in conjunction with the setup of the eks_cluster_1 EKS cluster
|
||||||
|
enable_dashboards = true
|
||||||
|
enable_alerting_rules = true
|
||||||
|
enable_recording_rules = true
|
||||||
|
|
||||||
|
grafana_api_key = var.grafana_api_key
|
||||||
|
dashboards_folder_id = module.aws_observability_accelerator.grafana_dashboards_folder_id
|
||||||
|
managed_prometheus_workspace_id = module.aws_observability_accelerator.managed_prometheus_workspace_id
|
||||||
|
managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint
|
||||||
|
managed_prometheus_workspace_region = module.aws_observability_accelerator.managed_prometheus_workspace_region
|
||||||
|
|
||||||
|
java_config = {
|
||||||
|
enable_alerting_rules = true
|
||||||
|
enable_recording_rules = true
|
||||||
|
scrape_sample_limit = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
prometheus_config = {
|
||||||
|
global_scrape_interval = "60s"
|
||||||
|
global_scrape_timeout = "15s"
|
||||||
|
scrape_sample_limit = 2000
|
||||||
|
}
|
||||||
|
|
||||||
|
providers = {
|
||||||
|
aws = aws.eks_cluster_1
|
||||||
|
kubernetes = kubernetes.eks_cluster_1
|
||||||
|
helm = helm.eks_cluster_1
|
||||||
|
grafana = grafana
|
||||||
|
}
|
||||||
|
|
||||||
|
depends_on = [
|
||||||
|
module.aws_observability_accelerator
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
module "eks_cluster_2_monitoring" {
|
||||||
|
source = "../../../terraform-aws-observability-accelerator//modules/eks-monitoring"
|
||||||
|
eks_cluster_id = var.eks_cluster_2_id
|
||||||
|
enable_amazon_eks_adot = true
|
||||||
|
enable_cert_manager = true
|
||||||
|
enable_java = true
|
||||||
|
|
||||||
|
# Since the following were enabled in conjunction with the set up of the eks_cluster_1 EKS cluster, we will skip
|
||||||
|
# them with the eks_cluster_2 EKS cluster
|
||||||
|
enable_dashboards = false
|
||||||
|
enable_alerting_rules = false
|
||||||
|
enable_recording_rules = false
|
||||||
|
|
||||||
|
grafana_api_key = var.grafana_api_key
|
||||||
|
dashboards_folder_id = module.aws_observability_accelerator.grafana_dashboards_folder_id
|
||||||
|
managed_prometheus_workspace_id = module.aws_observability_accelerator.managed_prometheus_workspace_id
|
||||||
|
managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint
|
||||||
|
managed_prometheus_workspace_region = module.aws_observability_accelerator.managed_prometheus_workspace_region
|
||||||
|
|
||||||
|
java_config = {
|
||||||
|
enable_alerting_rules = false # addressed while setting up the eks_cluster_1 EKS cluster
|
||||||
|
enable_recording_rules = false # addressed while setting up the eks_cluster_1 EKS cluster
|
||||||
|
scrape_sample_limit = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
prometheus_config = {
|
||||||
|
global_scrape_interval = "60s"
|
||||||
|
global_scrape_timeout = "15s"
|
||||||
|
scrape_sample_limit = 2000
|
||||||
|
}
|
||||||
|
|
||||||
|
providers = {
|
||||||
|
aws = aws.eks_cluster_2
|
||||||
|
kubernetes = kubernetes.eks_cluster_2
|
||||||
|
helm = helm.eks_cluster_2
|
||||||
|
grafana = grafana
|
||||||
|
}
|
||||||
|
|
||||||
|
depends_on = [
|
||||||
|
module.aws_observability_accelerator
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
provider "kubernetes" {
|
||||||
|
host = data.aws_eks_cluster.eks_cluster_1.endpoint
|
||||||
|
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_1.certificate_authority[0].data)
|
||||||
|
token = data.aws_eks_cluster_auth.eks_cluster_1.token
|
||||||
|
alias = "eks_cluster_1"
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "kubernetes" {
|
||||||
|
host = data.aws_eks_cluster.eks_cluster_2.endpoint
|
||||||
|
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_2.certificate_authority[0].data)
|
||||||
|
token = data.aws_eks_cluster_auth.eks_cluster_2.token
|
||||||
|
alias = "eks_cluster_2"
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "helm" {
|
||||||
|
kubernetes {
|
||||||
|
host = data.aws_eks_cluster.eks_cluster_1.endpoint
|
||||||
|
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_1.certificate_authority[0].data)
|
||||||
|
token = data.aws_eks_cluster_auth.eks_cluster_1.token
|
||||||
|
}
|
||||||
|
alias = "eks_cluster_1"
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "helm" {
|
||||||
|
kubernetes {
|
||||||
|
host = data.aws_eks_cluster.eks_cluster_2.endpoint
|
||||||
|
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_2.certificate_authority[0].data)
|
||||||
|
token = data.aws_eks_cluster_auth.eks_cluster_2.token
|
||||||
|
}
|
||||||
|
alias = "eks_cluster_2"
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "aws" {
|
||||||
|
region = var.eks_cluster_1_region
|
||||||
|
alias = "eks_cluster_1"
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "aws" {
|
||||||
|
region = var.eks_cluster_2_region
|
||||||
|
alias = "eks_cluster_2"
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "grafana" {
|
||||||
|
url = module.aws_observability_accelerator.managed_grafana_workspace_endpoint
|
||||||
|
auth = var.grafana_api_key
|
||||||
|
}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
variable "eks_cluster_1_id" {
|
||||||
|
description = "Name or ID of the EKS cluster 1"
|
||||||
|
type = string
|
||||||
|
default = "eks-cluster-1"
|
||||||
|
nullable = false
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "eks_cluster_1_region" {
|
||||||
|
description = "AWS region of the EKS cluster 1"
|
||||||
|
type = string
|
||||||
|
default = "us-west-2"
|
||||||
|
nullable = false
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "eks_cluster_2_id" {
|
||||||
|
description = "Name or ID of the EKS cluster 2"
|
||||||
|
type = string
|
||||||
|
default = "eks-cluster-2"
|
||||||
|
nullable = true
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "eks_cluster_2_region" {
|
||||||
|
description = "AWS region of the EKS cluster 2"
|
||||||
|
type = string
|
||||||
|
default = "us-west-2"
|
||||||
|
nullable = true
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "managed_prometheus_workspace_id" {
|
||||||
|
description = "Amazon Managed Service for Prometheus Workspace ID"
|
||||||
|
type = string
|
||||||
|
default = ""
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "managed_grafana_workspace_id" {
|
||||||
|
description = "Amazon Managed Grafana Workspace ID"
|
||||||
|
type = string
|
||||||
|
default = ""
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "grafana_api_key" {
|
||||||
|
description = "API key for authorizing the Grafana provider to make changes to Amazon Managed Grafana"
|
||||||
|
type = string
|
||||||
|
default = ""
|
||||||
|
sensitive = true
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
terraform {
|
||||||
|
required_version = ">= 1.3.9"
|
||||||
|
|
||||||
|
required_providers {
|
||||||
|
aws = {
|
||||||
|
source = "hashicorp/aws"
|
||||||
|
version = ">= 4.55.0"
|
||||||
|
configuration_aliases = [aws.eks_cluster_1, aws.eks_cluster_2]
|
||||||
|
}
|
||||||
|
kubernetes = {
|
||||||
|
source = "hashicorp/kubernetes"
|
||||||
|
version = ">= 2.18.0"
|
||||||
|
configuration_aliases = [kubernetes.eks_cluster_1, kubernetes.eks_cluster_2]
|
||||||
|
}
|
||||||
|
helm = {
|
||||||
|
source = "hashicorp/helm"
|
||||||
|
version = ">= 2.9.0"
|
||||||
|
configuration_aliases = [helm.eks_cluster_1, helm.eks_cluster_2]
|
||||||
|
}
|
||||||
|
kubectl = {
|
||||||
|
source = "gavinbunney/kubectl"
|
||||||
|
version = ">= 1.14"
|
||||||
|
}
|
||||||
|
grafana = {
|
||||||
|
source = "grafana/grafana"
|
||||||
|
version = ">= 1.25.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user