mirror of
https://github.com/storytold/terraform-aws-observability-accelerator.git
synced 2026-10-09 00:09:43 +00:00
Compose EKS monitoring modules (#115)
* Move modules around * Update amp billing source * Merge Java monitoring to EKS * Update docs * Merge nginx pattern * Pre-commit * Add save and test URL output * Move EKS dependencies to EKS monitoring module * update docs * Update examples and docs * Add java doc * Add NGINX doc * Update nginx doc * Fix amp monitoring example path * Fix pre-commit * Todo: move to main after merge * Update docs, fix tags
This commit is contained in:
Rodrigue Koffi
+2
-4
@@ -15,11 +15,9 @@ terraform destroy
|
||||
To remove resources from your Terraform state, run
|
||||
|
||||
```bash
|
||||
# grafana workspace
|
||||
terraform state rm "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
|
||||
|
||||
# prometheus workspace
|
||||
terraform state rm "module.eks_observability_accelerator.aws_prometheus_workspace.this[0]"
|
||||
```
|
||||
|
||||
> **Note:** To view all the features proposed by this module, visit the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
|
||||
!!! note
|
||||
To view all the features proposed by this module, visit the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
|
||||
|
||||
+15
-16
@@ -2,7 +2,7 @@
|
||||
|
||||
This example demonstrates how to monitor your Amazon Elastic Kubernetes Service
|
||||
(Amazon EKS) cluster with the Observability Accelerator's EKS
|
||||
[infrastructure module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
|
||||
[infrastructure module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring).
|
||||
|
||||
Monitoring Amazon Elastic Kubernetes Service (Amazon EKS) for metrics has two categories:
|
||||
the control plane and the Amazon EKS nodes (with Kubernetes objects).
|
||||
@@ -72,24 +72,17 @@ aws amp create-workspace --alias observability-accelerator --query '.workspaceId
|
||||
|
||||
### 5. Amazon Managed Grafana workspace
|
||||
|
||||
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, edit and run:
|
||||
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, create an environment variable as described below.
|
||||
To create a new workspace, visit our Amazon Managed Grafana [documentation](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html).
|
||||
Make sure to provide the workspace with Amazon Managed Service for Prometheus read permissions.
|
||||
|
||||
!!! note
|
||||
For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
|
||||
|
||||
```bash
|
||||
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||
```
|
||||
|
||||
To create a new one, within this example's Terraform state (sharing the same lifecycle with all the
|
||||
other resources created by Terraform):
|
||||
|
||||
- Edit main.tf and set `enable_managed_grafana = true`
|
||||
- Run
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform apply -target "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
|
||||
export TF_VAR_managed_grafana_workspace_id=$(terraform output --raw managed_grafana_workspace_id)
|
||||
```
|
||||
|
||||
### 6. Grafana API Key
|
||||
|
||||
Amazon Managed Grafana provides a control plane API for generating Grafana API keys.
|
||||
@@ -114,7 +107,13 @@ terraform apply
|
||||
|
||||
1. Prometheus datasource on Grafana
|
||||
|
||||
Open your Grafana workspace and under Configuration -> Data sources, you should see `aws-observability-accelerator`. Open and click `Save & test`. You should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
|
||||
Make sure to open the link in the output. After a successful deployment, this will open
|
||||
the Prometheus datasource configuration on Grafana.
|
||||
Click `Save & test` and you should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
|
||||
|
||||
```bash
|
||||
terraform output grafana_prometheus_datasource_test
|
||||
```
|
||||
|
||||
2. Grafana dashboards
|
||||
|
||||
@@ -126,7 +125,7 @@ Open a specific dashboard and you should be able to view its visualization
|
||||
|
||||
<img width="2056" alt="cluster headlines" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">
|
||||
|
||||
2. Amazon Managed Service for Prometheus rules and alerts
|
||||
3. Amazon Managed Service for Prometheus rules and alerts
|
||||
|
||||
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you should find new rules deployed.
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# Monitor Java/JMX applications running on Amazon EKS
|
||||
|
||||
!!! note
|
||||
Since v2.x, Java based applications monitoring on EKS has been merged within
|
||||
the [eks-monitoring module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring)
|
||||
to allow visibility both on the cluster and the workloads, [#59](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues/59).
|
||||
|
||||
In addition to EKS infrastructure monitoring, the current example provides
|
||||
curated Grafana dashboards, Prometheus alerting and recording rules with multiple
|
||||
configuration options for Java based workloads on EKS.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Add Java metrics, dashboards and alerts
|
||||
|
||||
From the [previous example's](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/) configuration,
|
||||
simply enable the Java pattern's flag.
|
||||
|
||||
```hcl
|
||||
|
||||
module "eks_monitoring" {
|
||||
...
|
||||
enable_java = true
|
||||
}
|
||||
```
|
||||
|
||||
You can further customize the Java pattern by providing `java_config` [options](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/feat/modules-composition/modules/eks-monitoring/README.md#input_java_config).
|
||||
|
||||
### 2. Grafana API key
|
||||
|
||||
Make sure to refresh your temporary Grafana API key
|
||||
|
||||
```bash
|
||||
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
Simply run this command to deploy.
|
||||
|
||||
```bash
|
||||
terraform apply
|
||||
```
|
||||
|
||||
!!! note
|
||||
To see the complete Java example, open the [example on the repository](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples/existing-cluster-java)
|
||||
|
||||
## Visualization
|
||||
|
||||
1. Grafana dashboards
|
||||
|
||||
Go to the Dashboards panel of your Grafana workspace. There will be a folder called `Observability Accelerator Dashboards`
|
||||
|
||||
<img width="832" alt="image" src="https://user-images.githubusercontent.com/97046295/194903648-57c55d30-6f90-4b03-9eb6-577aaba7dc22.png">
|
||||
|
||||
Open the "Java/JMX" dashboard to view its visualization
|
||||
|
||||
<img width="2560" alt="Grafana Java dashboard" src="https://user-images.githubusercontent.com/10175027/217821001-2119c81f-94bd-4811-8bbb-caaf1ae5a77a.png">
|
||||
|
||||
2. Amazon Managed Service for Prometheus rules and alerts
|
||||
|
||||
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed.
|
||||
|
||||
<img width="1314" alt="image" src="https://user-images.githubusercontent.com/97046295/194904104-09a28577-d149-478e-b0a1-dc21cb7effc1.png">
|
||||
|
||||
!!! note
|
||||
To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html)
|
||||
|
||||
|
||||
## Deploy an example Java application
|
||||
|
||||
In this section we will reuse an example from the AWS OpenTelemetry collector [repository](https://github.com/aws-observability/aws-otel-collector/blob/main/docs/developers/container-insights-eks-jmx.md). For convenience, the steps can be found below.
|
||||
|
||||
1. Clone [this repository](https://github.com/aws-observability/aws-otel-test-framework) and navigate to the `sample-apps/jmx/` directory.
|
||||
|
||||
2. Authenticate to Amazon ECR
|
||||
|
||||
```sh
|
||||
export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
|
||||
export AWS_REGION={region}
|
||||
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
|
||||
```
|
||||
|
||||
3. Create an Amazon ECR repository
|
||||
|
||||
```sh
|
||||
aws ecr create-repository --repository-name prometheus-sample-tomcat-jmx \
|
||||
--image-scanning-configuration scanOnPush=true \
|
||||
--region $AWS_REGION
|
||||
```
|
||||
|
||||
4. Build Docker image and push to ECR.
|
||||
|
||||
```sh
|
||||
docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest .
|
||||
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest
|
||||
```
|
||||
|
||||
5. Install sample application
|
||||
|
||||
```sh
|
||||
export SAMPLE_TRAFFIC_NAMESPACE=javajmx-sample
|
||||
curl https://raw.githubusercontent.com/aws-observability/aws-otel-test-framework/terraform/sample-apps/jmx/examples/prometheus-metrics-sample.yaml > metrics-sample.yaml
|
||||
sed -i "s/{{aws_account_id}}/$AWS_ACCOUNT_ID/g" metrics-sample.yaml
|
||||
sed -i "s/{{region}}/$AWS_REGION/g" metrics-sample.yaml
|
||||
sed -i "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" metrics-sample.yaml
|
||||
kubectl apply -f metrics-sample.yaml
|
||||
```
|
||||
|
||||
Verify that the sample application is running:
|
||||
|
||||
```sh
|
||||
kubectl get pods -n $SAMPLE_TRAFFIC_NAMESPACE
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tomcat-bad-traffic-generator 1/1 Running 0 11s
|
||||
tomcat-example-7958666589-2q755 0/1 ContainerCreating 0 11s
|
||||
tomcat-traffic-generator 1/1 Running 0 11s
|
||||
```
|
||||
@@ -0,0 +1,120 @@
|
||||
# Monitor Nginx applications running on Amazon EKS
|
||||
|
||||
!!! note
|
||||
Since v2.x, NGINX based applications monitoring on EKS has been merged within
|
||||
the [eks-monitoring module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring)
|
||||
to allow visibility both on the cluster and the workloads, [#59](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues/59).
|
||||
|
||||
In addition to EKS infrastructure monitoring, the current example provides
|
||||
curated Grafana dashboards, Prometheus alerting and recording rules with multiple
|
||||
configuration options for NGINX based workloads on EKS.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Add NGINX metrics, dashboards and alerts
|
||||
|
||||
From the [EKS cluster monitoring example's](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/) configuration,
|
||||
simply enable the NGINX pattern's flag.
|
||||
|
||||
```hcl
|
||||
|
||||
module "eks_monitoring" {
|
||||
...
|
||||
enable_nginx = true
|
||||
}
|
||||
```
|
||||
|
||||
You can further customize the NGINX pattern by providing `nginx_config` [options](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/feat/modules-composition/modules/eks-monitoring/README.md#input_nginx_config).
|
||||
|
||||
### 2. Grafana API key
|
||||
|
||||
Make sure to refresh your temporary Grafana API key
|
||||
|
||||
```bash
|
||||
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
Simply run this command to deploy.
|
||||
|
||||
```bash
|
||||
terraform apply
|
||||
```
|
||||
|
||||
!!! note
|
||||
To see the complete NGINX example, open the [example on the repository](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples/existing-cluster-nginx)
|
||||
|
||||
## Visualization
|
||||
|
||||
1. Grafana dashboards
|
||||
|
||||
Go to the Dashboards panel of your Grafana workspace. You will see a list of dashboards under the `Observability Accelerator Dashboards`
|
||||
|
||||
<img width="1208" alt="image" src="https://user-images.githubusercontent.com/97046295/190665211-60faef71-d83d-4d59-ac80-bf4309d8c082.png">
|
||||
|
||||
Open the NGINX dashboard and you will be able to view its visualization
|
||||
|
||||
<img width="1850" alt="image" src="https://user-images.githubusercontent.com/97046295/196226043-e49afeb9-7828-467f-9199-5707cdc69aa9.png">
|
||||
|
||||
2. Amazon Managed Service for Prometheus rules and alerts
|
||||
|
||||
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed.
|
||||
|
||||
<img width="1054" alt="image" src="https://user-images.githubusercontent.com/97046295/190665728-ae8bb709-ad93-4629-b845-85c158dd1925.png">
|
||||
|
||||
!!! note
|
||||
To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html)
|
||||
|
||||
## Deploy an example application to visualize metrics
|
||||
|
||||
In this section we will deploy sample application and extract metrics using AWS OpenTelemetry collector
|
||||
|
||||
### 1. Add the helm incubator repo:
|
||||
|
||||
```sh
|
||||
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
|
||||
```
|
||||
|
||||
### 2. Enter the following command to create a new namespace:
|
||||
|
||||
```sh
|
||||
kubectl create namespace nginx-ingress-sample
|
||||
```
|
||||
|
||||
### 3. Enter the following commands to install NGINX:
|
||||
|
||||
```sh
|
||||
helm install my-nginx ingress-nginx/ingress-nginx \
|
||||
--namespace nginx-ingress-sample \
|
||||
--set controller.metrics.enabled=true \
|
||||
--set-string controller.metrics.service.annotations."prometheus\.io/port"="10254" \
|
||||
--set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true"
|
||||
```
|
||||
|
||||
### 4. Set an EXTERNAL-IP variable to the value of the EXTERNAL-IP column in the row of the NGINX ingress controller.
|
||||
|
||||
```sh
|
||||
EXTERNAL_IP=your-nginx-controller-external-ip
|
||||
```
|
||||
|
||||
### 5. Start some sample NGINX traffic by entering the following command.
|
||||
|
||||
```sh
|
||||
SAMPLE_TRAFFIC_NAMESPACE=nginx-sample-traffic
|
||||
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_traffic/nginx-traffic/nginx-traffic-sample.yaml |
|
||||
sed "s/{{external_ip}}/$EXTERNAL_IP/g" |
|
||||
sed "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" |
|
||||
kubectl apply -f -
|
||||
```
|
||||
|
||||
### 6. Verify if the application is running
|
||||
|
||||
```sh
|
||||
kubectl get pods -n nginx-ingress-sample
|
||||
```
|
||||
|
||||
### 7. Visualize the Application's dashboard
|
||||
|
||||
Log back into your Managed Grafana Workspace and navigate to the dashboard side panel, click on `Observability Accelerator Dashboards` Folder and open the `NGINX` Dashboard.
|
||||
Reference in New Issue
Block a user