mirror of
https://github.com/storytold/terraform-aws-observability-accelerator.git
synced 2026-10-09 00:09:43 +00:00
Addition of EKS multi-cluster observability example (#155)
* Adding EKS multicluster observability example * EKS multicluster example - precommit fix * Corrected the path to an example * Made the multicluster example simpler * Pre-commit changes * Saved the images to Github and linked them in docs * Comments and language edits * Fix trailing spaces * Non-controversial naming and simpler variables * Added region to the data gathering * Formatting changes --------- Co-authored-by: Rodrigue Koffi <bonclay7@users.noreply.github.com>
This commit is contained in:
Vijay Chintalapati
committed by
GitHub
GitHub
parent
5b52c7b8e1
commit
b84e9b800f
@@ -0,0 +1,96 @@
|
||||
# AWS EKS Multicluster Observability
|
||||
|
||||
This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with more than one EKS cluster and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
#### 1. EKS clusters
|
||||
|
||||
Using the example [eks-cluster-with-vpc](../../examples/eks-cluster-with-vpc/), create two EKS clusters with the names:
|
||||
1. `eks-cluster-1`
|
||||
2. `eks-cluster-2`
|
||||
|
||||
#### 2. Amazon Managed Serivce for Prometheus (AMP) workspace
|
||||
|
||||
We recommend that you create a new AMP workspace. To do that you can run the following command.
|
||||
|
||||
Ensure you have the following necessary IAM permissions
|
||||
* `aps.CreateWorkspace`
|
||||
|
||||
```sh
|
||||
export TF_VAR_managed_prometheus_workspace_id=$(aws amp create-workspace --alias observability-accelerator --query='workspaceId' --output text)
|
||||
```
|
||||
|
||||
#### 3. Amazon Managed Grafana (AMG) workspace
|
||||
|
||||
To run this example you need an AMG workspace. If you have
|
||||
an existing workspace, create an environment variable as described below.
|
||||
To create a new workspace, visit our supporting example for managed Grafana.
|
||||
|
||||
!!! note
|
||||
For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
|
||||
|
||||
```sh
|
||||
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||
```
|
||||
|
||||
#### 4. Grafana API Key
|
||||
|
||||
AMG provides a control plane API for generating Grafana API keys.
|
||||
As a security best practice, we will provide to Terraform a short lived API key to
|
||||
run the `apply` or `destroy` command.
|
||||
|
||||
Ensure you have the following necessary IAM permissions
|
||||
* `grafana.CreateWorkspaceApiKey`
|
||||
* `grafana.DeleteWorkspaceApiKey`
|
||||
|
||||
```sh
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
#### 1. Download sources and initialize Terraform
|
||||
|
||||
```sh
|
||||
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
|
||||
cd terraform-aws-observability-accelerator/examples/eks-multicluster
|
||||
terraform init
|
||||
```
|
||||
|
||||
#### 2. Deploy
|
||||
|
||||
Verify by looking at the file `variables.tf` that there are two EKS clusters targeted for deployment by the names/ids:
|
||||
1. `eks-cluster-1`
|
||||
2. `eks-cluster-2`
|
||||
|
||||
The difference in deployment between these clusters is that Terraform, when setting up the EKS cluster behind variable `eks_cluster_1_id` for observability, also sets up:
|
||||
* Dashboard folder and files in `AMG`
|
||||
* Prometheus and Java, alerting and recording rules in `AMP`
|
||||
|
||||
!!! warning
|
||||
To override the defaults, create a `terraform.tfvars` and change the default values of the variables.
|
||||
|
||||
Run the following command to deploy
|
||||
|
||||
```sh
|
||||
terraform apply --auto-approve
|
||||
```
|
||||
|
||||
## Verifying Multicluster Observability
|
||||
|
||||
One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace.
|
||||
|
||||
Note how you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster.
|
||||
|
||||
<img width="2557" alt="eks-multicluster-1" src="https://user-images.githubusercontent.com/4762573/233949110-ce275d06-7ad8-494c-b527-d9c2a0fb6645.png">
|
||||
|
||||
<img width="2560" alt="eks-multicluster-2" src="https://user-images.githubusercontent.com/4762573/233949227-f401f81e-e0d6-4242-96ad-0bcd39ad4e2d.png">
|
||||
|
||||
## Cleanup
|
||||
|
||||
To clean up entirely, run the following command:
|
||||
|
||||
```sh
|
||||
terraform destroy --auto-approve
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
data "aws_eks_cluster_auth" "eks_cluster_1" {
|
||||
name = var.eks_cluster_1_id
|
||||
provider = aws.eks_cluster_1
|
||||
}
|
||||
|
||||
data "aws_eks_cluster_auth" "eks_cluster_2" {
|
||||
name = var.eks_cluster_2_id
|
||||
provider = aws.eks_cluster_2
|
||||
}
|
||||
|
||||
data "aws_eks_cluster" "eks_cluster_1" {
|
||||
name = var.eks_cluster_1_id
|
||||
provider = aws.eks_cluster_1
|
||||
}
|
||||
|
||||
data "aws_eks_cluster" "eks_cluster_2" {
|
||||
name = var.eks_cluster_2_id
|
||||
provider = aws.eks_cluster_2
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
module "aws_observability_accelerator" {
|
||||
source = "../../../terraform-aws-observability-accelerator"
|
||||
aws_region = var.eks_cluster_1_region
|
||||
enable_managed_prometheus = false
|
||||
enable_alertmanager = true
|
||||
create_dashboard_folder = true
|
||||
create_prometheus_data_source = true
|
||||
grafana_api_key = var.grafana_api_key
|
||||
managed_prometheus_workspace_region = null
|
||||
managed_prometheus_workspace_id = var.managed_prometheus_workspace_id
|
||||
managed_grafana_workspace_id = var.managed_grafana_workspace_id
|
||||
|
||||
providers = {
|
||||
aws = aws.eks_cluster_1
|
||||
}
|
||||
}
|
||||
|
||||
module "eks_cluster_1_monitoring" {
|
||||
source = "../../../terraform-aws-observability-accelerator//modules/eks-monitoring"
|
||||
eks_cluster_id = var.eks_cluster_1_id
|
||||
enable_amazon_eks_adot = true
|
||||
enable_cert_manager = true
|
||||
enable_java = true
|
||||
|
||||
# This configuration section results in actions performed on AMG and AMP; and it needs to be done just once
|
||||
# And hence, this in performed in conjunction with the setup of the eks_cluster_1 EKS cluster
|
||||
enable_dashboards = true
|
||||
enable_alerting_rules = true
|
||||
enable_recording_rules = true
|
||||
|
||||
grafana_api_key = var.grafana_api_key
|
||||
dashboards_folder_id = module.aws_observability_accelerator.grafana_dashboards_folder_id
|
||||
managed_prometheus_workspace_id = module.aws_observability_accelerator.managed_prometheus_workspace_id
|
||||
managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint
|
||||
managed_prometheus_workspace_region = module.aws_observability_accelerator.managed_prometheus_workspace_region
|
||||
|
||||
java_config = {
|
||||
enable_alerting_rules = true
|
||||
enable_recording_rules = true
|
||||
scrape_sample_limit = 1
|
||||
}
|
||||
|
||||
prometheus_config = {
|
||||
global_scrape_interval = "60s"
|
||||
global_scrape_timeout = "15s"
|
||||
scrape_sample_limit = 2000
|
||||
}
|
||||
|
||||
providers = {
|
||||
aws = aws.eks_cluster_1
|
||||
kubernetes = kubernetes.eks_cluster_1
|
||||
helm = helm.eks_cluster_1
|
||||
grafana = grafana
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
module.aws_observability_accelerator
|
||||
]
|
||||
}
|
||||
|
||||
module "eks_cluster_2_monitoring" {
|
||||
source = "../../../terraform-aws-observability-accelerator//modules/eks-monitoring"
|
||||
eks_cluster_id = var.eks_cluster_2_id
|
||||
enable_amazon_eks_adot = true
|
||||
enable_cert_manager = true
|
||||
enable_java = true
|
||||
|
||||
# Since the following were enabled in conjunction with the set up of the eks_cluster_1 EKS cluster, we will skip
|
||||
# them with the eks_cluster_2 EKS cluster
|
||||
enable_dashboards = false
|
||||
enable_alerting_rules = false
|
||||
enable_recording_rules = false
|
||||
|
||||
grafana_api_key = var.grafana_api_key
|
||||
dashboards_folder_id = module.aws_observability_accelerator.grafana_dashboards_folder_id
|
||||
managed_prometheus_workspace_id = module.aws_observability_accelerator.managed_prometheus_workspace_id
|
||||
managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint
|
||||
managed_prometheus_workspace_region = module.aws_observability_accelerator.managed_prometheus_workspace_region
|
||||
|
||||
java_config = {
|
||||
enable_alerting_rules = false # addressed while setting up the eks_cluster_1 EKS cluster
|
||||
enable_recording_rules = false # addressed while setting up the eks_cluster_1 EKS cluster
|
||||
scrape_sample_limit = 1
|
||||
}
|
||||
|
||||
prometheus_config = {
|
||||
global_scrape_interval = "60s"
|
||||
global_scrape_timeout = "15s"
|
||||
scrape_sample_limit = 2000
|
||||
}
|
||||
|
||||
providers = {
|
||||
aws = aws.eks_cluster_2
|
||||
kubernetes = kubernetes.eks_cluster_2
|
||||
helm = helm.eks_cluster_2
|
||||
grafana = grafana
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
module.aws_observability_accelerator
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
provider "kubernetes" {
|
||||
host = data.aws_eks_cluster.eks_cluster_1.endpoint
|
||||
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_1.certificate_authority[0].data)
|
||||
token = data.aws_eks_cluster_auth.eks_cluster_1.token
|
||||
alias = "eks_cluster_1"
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
host = data.aws_eks_cluster.eks_cluster_2.endpoint
|
||||
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_2.certificate_authority[0].data)
|
||||
token = data.aws_eks_cluster_auth.eks_cluster_2.token
|
||||
alias = "eks_cluster_2"
|
||||
}
|
||||
|
||||
provider "helm" {
|
||||
kubernetes {
|
||||
host = data.aws_eks_cluster.eks_cluster_1.endpoint
|
||||
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_1.certificate_authority[0].data)
|
||||
token = data.aws_eks_cluster_auth.eks_cluster_1.token
|
||||
}
|
||||
alias = "eks_cluster_1"
|
||||
}
|
||||
|
||||
provider "helm" {
|
||||
kubernetes {
|
||||
host = data.aws_eks_cluster.eks_cluster_2.endpoint
|
||||
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_2.certificate_authority[0].data)
|
||||
token = data.aws_eks_cluster_auth.eks_cluster_2.token
|
||||
}
|
||||
alias = "eks_cluster_2"
|
||||
}
|
||||
|
||||
provider "aws" {
|
||||
region = var.eks_cluster_1_region
|
||||
alias = "eks_cluster_1"
|
||||
}
|
||||
|
||||
provider "aws" {
|
||||
region = var.eks_cluster_2_region
|
||||
alias = "eks_cluster_2"
|
||||
}
|
||||
|
||||
provider "grafana" {
|
||||
url = module.aws_observability_accelerator.managed_grafana_workspace_endpoint
|
||||
auth = var.grafana_api_key
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
variable "eks_cluster_1_id" {
|
||||
description = "Name or ID of the EKS cluster 1"
|
||||
type = string
|
||||
default = "eks-cluster-1"
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "eks_cluster_1_region" {
|
||||
description = "AWS region of the EKS cluster 1"
|
||||
type = string
|
||||
default = "us-west-2"
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "eks_cluster_2_id" {
|
||||
description = "Name or ID of the EKS cluster 2"
|
||||
type = string
|
||||
default = "eks-cluster-2"
|
||||
nullable = true
|
||||
}
|
||||
|
||||
variable "eks_cluster_2_region" {
|
||||
description = "AWS region of the EKS cluster 2"
|
||||
type = string
|
||||
default = "us-west-2"
|
||||
nullable = true
|
||||
}
|
||||
|
||||
variable "managed_prometheus_workspace_id" {
|
||||
description = "Amazon Managed Service for Prometheus Workspace ID"
|
||||
type = string
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "managed_grafana_workspace_id" {
|
||||
description = "Amazon Managed Grafana Workspace ID"
|
||||
type = string
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "grafana_api_key" {
|
||||
description = "API key for authorizing the Grafana provider to make changes to Amazon Managed Grafana"
|
||||
type = string
|
||||
default = ""
|
||||
sensitive = true
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
terraform {
|
||||
required_version = ">= 1.3.9"
|
||||
|
||||
required_providers {
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = ">= 4.55.0"
|
||||
configuration_aliases = [aws.eks_cluster_1, aws.eks_cluster_2]
|
||||
}
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = ">= 2.18.0"
|
||||
configuration_aliases = [kubernetes.eks_cluster_1, kubernetes.eks_cluster_2]
|
||||
}
|
||||
helm = {
|
||||
source = "hashicorp/helm"
|
||||
version = ">= 2.9.0"
|
||||
configuration_aliases = [helm.eks_cluster_1, helm.eks_cluster_2]
|
||||
}
|
||||
kubectl = {
|
||||
source = "gavinbunney/kubectl"
|
||||
version = ">= 1.14"
|
||||
}
|
||||
grafana = {
|
||||
source = "grafana/grafana"
|
||||
version = ">= 1.25.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user