storytold/terraform-aws-observability-accelerator
2
0
Fork 0
mirror of https://github.com/storytold/terraform-aws-observability-accelerator.git synced 2026-10-09 00:09:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compose EKS monitoring modules (#115)

Browse Source 
* Move modules around

* Update amp billing source

* Merge Java monitoring to EKS

* Update docs

* Merge nginx pattern

* Pre-commit

* Add save and test URL output

* Move EKS dependencies to EKS monitoring module

* update docs

* Update examples and docs

* Add java doc

* Add NGINX doc

* Update nginx doc

* Fix amp monitoring example path

* Fix pre-commit

* Todo: move to main after merge

* Update docs, fix tags
This commit is contained in:
Rodrigue Koffi
2023-02-20 18:36:08 +01:00
committed by GitHub
parent fe83579997
commit daed34db80
99 changed files with 887 additions and 1477 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concepts.md
+16 -4
View File
@@ -31,6 +31,18 @@ you need to track changes as part of a Git repository or CI/CD pipeline.
!!! warning
When using `tfvars` files, always be careful to not store and commit any secrets (keys, passwords, ...)
## v2.x changes
v2.x [releases](https://github.com/aws-observability/terraform-aws-observability-accelerator/releases) introduce
couple of breaking changes compared to previous versions:
- `modules/workloads/infra` module moves to `modules/eks-monitoring`
- EKS configuration options moves from the base module to the `eks-monitoring` module
- EKS workload modules **java,nginx** merge into `eks-monitoring` as configuration options (patterns),
see [examples](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples)
- Examples have been updated to reflect these changes
## Base module
The base module allows you to configure the AWS Observability services for your cluster and
@@ -41,7 +53,7 @@ and ADOT Operator deployed for you and ready to receive your data.
The base module serve as an anchor to the workload modules and cannot run on its own.
```hcl
module "eks_observability_accelerator" {
module "aws_observability_accelerator" {
# use release tags and check for the latest versions
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
@@ -49,7 +61,7 @@ module "eks_observability_accelerator" {
aws_region = "eu-west-1"
eks_cluster_id = "my-eks-cluster"
# As Grafana shares a different lifecycle, it's best to use an existing workspace.
# As Grafana shares a different lifecycle, we recommend using an existing workspace.
managed_grafana_workspace_id = var.managed_grafana_workspace_id
grafana_api_key = var.grafana_api_key
}
@@ -58,7 +70,7 @@ module "eks_observability_accelerator" {
You can optionally reuse an existing Amazon Managed Service for Prometheus Workspace:
```hcl
module "eks_observability_accelerator" {
module "aws_observability_accelerator" {
# use release tags and check for the latest versions
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
@@ -83,7 +95,7 @@ View all the configuration options in the [module's documentation](https://githu
Workloads modules are focused Terraform modules provided in this repository. They essentially provide curated metrics collection, alerts and Grafana dashboards according to the use case. Most of those modules require the base module.
You can check the full workload modules list and their documentation [here](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads).
You can check the full workload modules list and their documentation [here](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/).
All the modules come with end-to-end deployable examples.
docs/eks/destroy.md
+2 -4
View File
@@ -15,11 +15,9 @@ terraform destroy
To remove resources from your Terraform state, run
```bash
# grafana workspace
terraform state rm "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
# prometheus workspace
terraform state rm "module.eks_observability_accelerator.aws_prometheus_workspace.this[0]"
```
> **Note:** To view all the features proposed by this module, visit the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
!!! note
To view all the features proposed by this module, visit the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
docs/eks/index.md
+15 -16
View File
@@ -2,7 +2,7 @@
This example demonstrates how to monitor your Amazon Elastic Kubernetes Service
(Amazon EKS) cluster with the Observability Accelerator's EKS
[infrastructure module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
[infrastructure module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring).
Monitoring Amazon Elastic Kubernetes Service (Amazon EKS) for metrics has two categories:
the control plane and the Amazon EKS nodes (with Kubernetes objects).
@@ -72,24 +72,17 @@ aws amp create-workspace --alias observability-accelerator --query '.workspaceId
### 5. Amazon Managed Grafana workspace
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, edit and run:
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, create an environment variable as described below.
To create a new workspace, visit our Amazon Managed Grafana [documentation](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html).
Make sure to provide the workspace with Amazon Managed Service for Prometheus read permissions.
!!! note
For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
```bash
export TF_VAR_managed_grafana_workspace_id=g-xxx
```
To create a new one, within this example's Terraform state (sharing the same lifecycle with all the
other resources created by Terraform):
- Edit main.tf and set `enable_managed_grafana = true`
- Run
```bash
terraform init
terraform apply -target "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
export TF_VAR_managed_grafana_workspace_id=$(terraform output --raw managed_grafana_workspace_id)
```
### 6. Grafana API Key
Amazon Managed Grafana provides a control plane API for generating Grafana API keys.
@@ -114,7 +107,13 @@ terraform apply
1. Prometheus datasource on Grafana
Open your Grafana workspace and under Configuration -> Data sources, you should see `aws-observability-accelerator`. Open and click `Save & test`. You should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
Make sure to open the link in the output. After a successful deployment, this will open
the Prometheus datasource configuration on Grafana.
Click `Save & test` and you should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
```bash
terraform output grafana_prometheus_datasource_test
```
2. Grafana dashboards
@@ -126,7 +125,7 @@ Open a specific dashboard and you should be able to view its visualization
<img width="2056" alt="cluster headlines" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">
2. Amazon Managed Service for Prometheus rules and alerts
3. Amazon Managed Service for Prometheus rules and alerts
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you should find new rules deployed.
docs/eks/java.md
+120
View File
@@ -0,0 +1,120 @@
# Monitor Java/JMX applications running on Amazon EKS
!!! note
Since v2.x, Java based applications monitoring on EKS has been merged within
the [eks-monitoring module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring)
to allow visibility both on the cluster and the workloads, [#59](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues/59).
In addition to EKS infrastructure monitoring, the current example provides
curated Grafana dashboards, Prometheus alerting and recording rules with multiple
configuration options for Java based workloads on EKS.
## Setup
### 1. Add Java metrics, dashboards and alerts
From the [previous example's](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/) configuration,
simply enable the Java pattern's flag.
```hcl
module "eks_monitoring" {
...
enable_java = true
}
```
You can further customize the Java pattern by providing `java_config` [options](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/feat/modules-composition/modules/eks-monitoring/README.md#input_java_config).
### 2. Grafana API key
Make sure to refresh your temporary Grafana API key
```bash
export TF_VAR_managed_grafana_workspace_id=g-xxx
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
```
## Deploy
Simply run this command to deploy.
```bash
terraform apply
```
!!! note
To see the complete Java example, open the [example on the repository](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples/existing-cluster-java)
## Visualization
1. Grafana dashboards
Go to the Dashboards panel of your Grafana workspace. There will be a folder called `Observability Accelerator Dashboards`
<img width="832" alt="image" src="https://user-images.githubusercontent.com/97046295/194903648-57c55d30-6f90-4b03-9eb6-577aaba7dc22.png">
Open the "Java/JMX" dashboard to view its visualization
<img width="2560" alt="Grafana Java dashboard" src="https://user-images.githubusercontent.com/10175027/217821001-2119c81f-94bd-4811-8bbb-caaf1ae5a77a.png">
2. Amazon Managed Service for Prometheus rules and alerts
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed.
<img width="1314" alt="image" src="https://user-images.githubusercontent.com/97046295/194904104-09a28577-d149-478e-b0a1-dc21cb7effc1.png">
!!! note
To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html)
## Deploy an example Java application
In this section we will reuse an example from the AWS OpenTelemetry collector [repository](https://github.com/aws-observability/aws-otel-collector/blob/main/docs/developers/container-insights-eks-jmx.md). For convenience, the steps can be found below.
1. Clone [this repository](https://github.com/aws-observability/aws-otel-test-framework) and navigate to the `sample-apps/jmx/` directory.
2. Authenticate to Amazon ECR
```sh
export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
export AWS_REGION={region}
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
```
3. Create an Amazon ECR repository
```sh
aws ecr create-repository --repository-name prometheus-sample-tomcat-jmx \
--image-scanning-configuration scanOnPush=true \
--region $AWS_REGION
```
4. Build Docker image and push to ECR.
```sh
docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest .
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest
```
5. Install sample application
```sh
export SAMPLE_TRAFFIC_NAMESPACE=javajmx-sample
curl https://raw.githubusercontent.com/aws-observability/aws-otel-test-framework/terraform/sample-apps/jmx/examples/prometheus-metrics-sample.yaml > metrics-sample.yaml
sed -i "s/{{aws_account_id}}/$AWS_ACCOUNT_ID/g" metrics-sample.yaml
sed -i "s/{{region}}/$AWS_REGION/g" metrics-sample.yaml
sed -i "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" metrics-sample.yaml
kubectl apply -f metrics-sample.yaml
```
Verify that the sample application is running:
```sh
kubectl get pods -n $SAMPLE_TRAFFIC_NAMESPACE
NAME READY STATUS RESTARTS AGE
tomcat-bad-traffic-generator 1/1 Running 0 11s
tomcat-example-7958666589-2q755 0/1 ContainerCreating 0 11s
tomcat-traffic-generator 1/1 Running 0 11s
```
docs/eks/nginx.md
+120
View File
@@ -0,0 +1,120 @@
# Monitor Nginx applications running on Amazon EKS
!!! note
Since v2.x, NGINX based applications monitoring on EKS has been merged within
the [eks-monitoring module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring)
to allow visibility both on the cluster and the workloads, [#59](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues/59).
In addition to EKS infrastructure monitoring, the current example provides
curated Grafana dashboards, Prometheus alerting and recording rules with multiple
configuration options for NGINX based workloads on EKS.
## Setup
### 1. Add NGINX metrics, dashboards and alerts
From the [EKS cluster monitoring example's](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/) configuration,
simply enable the NGINX pattern's flag.
```hcl
module "eks_monitoring" {
...
enable_nginx = true
}
```
You can further customize the NGINX pattern by providing `nginx_config` [options](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/feat/modules-composition/modules/eks-monitoring/README.md#input_nginx_config).
### 2. Grafana API key
Make sure to refresh your temporary Grafana API key
```bash
export TF_VAR_managed_grafana_workspace_id=g-xxx
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
```
## Deploy
Simply run this command to deploy.
```bash
terraform apply
```
!!! note
To see the complete NGINX example, open the [example on the repository](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples/existing-cluster-nginx)
## Visualization
1. Grafana dashboards
Go to the Dashboards panel of your Grafana workspace. You will see a list of dashboards under the `Observability Accelerator Dashboards`
<img width="1208" alt="image" src="https://user-images.githubusercontent.com/97046295/190665211-60faef71-d83d-4d59-ac80-bf4309d8c082.png">
Open the NGINX dashboard and you will be able to view its visualization
<img width="1850" alt="image" src="https://user-images.githubusercontent.com/97046295/196226043-e49afeb9-7828-467f-9199-5707cdc69aa9.png">
2. Amazon Managed Service for Prometheus rules and alerts
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed.
<img width="1054" alt="image" src="https://user-images.githubusercontent.com/97046295/190665728-ae8bb709-ad93-4629-b845-85c158dd1925.png">
!!! note
To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html)
## Deploy an example application to visualize metrics
In this section we will deploy sample application and extract metrics using AWS OpenTelemetry collector
### 1. Add the helm incubator repo:
```sh
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
```
### 2. Enter the following command to create a new namespace:
```sh
kubectl create namespace nginx-ingress-sample
```
### 3. Enter the following commands to install NGINX:
```sh
helm install my-nginx ingress-nginx/ingress-nginx \
--namespace nginx-ingress-sample \
--set controller.metrics.enabled=true \
--set-string controller.metrics.service.annotations."prometheus\.io/port"="10254" \
--set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true"
```
### 4. Set an EXTERNAL-IP variable to the value of the EXTERNAL-IP column in the row of the NGINX ingress controller.
```sh
EXTERNAL_IP=your-nginx-controller-external-ip
```
### 5. Start some sample NGINX traffic by entering the following command.
```sh
SAMPLE_TRAFFIC_NAMESPACE=nginx-sample-traffic
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_traffic/nginx-traffic/nginx-traffic-sample.yaml |
sed "s/{{external_ip}}/$EXTERNAL_IP/g" |
sed "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" |
kubectl apply -f -
```
### 6. Verify if the application is running
```sh
kubectl get pods -n nginx-ingress-sample
```
### 7. Visualize the Application's dashboard
Log back into your Managed Grafana Workspace and navigate to the dashboard side panel, click on `Observability Accelerator Dashboards` Folder and open the `NGINX` Dashboard.
docs/images/example-diagram.svg
+1 -1
View File
File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 10 KiB

After

Width:  |  Height:  |  Size: 10 KiB

docs/index.md
+1 -1
View File
@@ -31,7 +31,7 @@ to be deployed in our packaged
We have supporting examples for quick setup such as:
- Creating an empty Amazon EKS cluster and a VPC
- Creating and configure an Amazon Managed Grafana workspace with SSO
- Creating and configure an Amazon Managed Grafana workspace with SSO (coming soon)
## Motivation
docs/support.md
+9
View File
@@ -0,0 +1,9 @@
# Support & Feedback
AWS Observability Accelerator for Terraform is maintained by AWS Solution Architects.
It is not part of an AWS service and support is provided best-effort by the
AWS Observability Accelerator community.
To post feedback, submit feature ideas, or report bugs, please use the [issues](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues) section of this GitHub repo.
If you are interested in contributing, see the [contribution guide](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/main/CONTRIBUTING.md).
docs/workloads/java.md
-196
View File
@@ -1,196 +0,0 @@
# Monitor Java/JMX applications running on Amazon EKS
The current example deploys the [java workload module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/java),
to provide to an existing EKS cluster with an OpenTelemetry collector,
curated Grafana dashboards, Prometheus alerting and recording rules with multiple
configuration options on the cluster infrastructure.
## Prerequisites
!!! note
Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.
## Setup
### 1. Download sources and initialize Terraform
```bash
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
cd examples/existing-cluster-java
terraform init
```
### 2. AWS Region
Specify the AWS Region where the resources will be deployed:
```bash
export TF_VAR_aws_region=xxx
```
### 3. Amazon EKS Cluster
To run this example, you need to provide your EKS cluster name. If you don't
have a cluster ready, visit [this example](https://aws-observability.github.io/terraform-aws-observability-accelerator/helpers/new-eks-cluster/)
first to create a new one.
Specify your cluster name:
```bash
export TF_VAR_eks_cluster_id=xxx
```
### 4. Amazon Managed Service for Prometheus workspace (optional)
By default, we create an Amazon Managed Service for Prometheus workspace for you.
However, if you have an existing workspace you want to reuse, edit and run:
```bash
export TF_VAR_managed_prometheus_workspace_id=ws-xxx
```
To create a workspace outside of Terraform's state, simply run:
```bash
aws amp create-workspace --alias observability-accelerator --query '.workspaceId' --output text
```
### 5. Amazon Managed Grafana workspace
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, edit and run:
```bash
export TF_VAR_managed_grafana_workspace_id=g-xxx
```
To create a new one, within this example's Terraform state (sharing the same lifecycle with all the other resources):
- Edit main.tf and set `enable_managed_grafana = true`
- Run
```bash
terraform init
terraform apply -target "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
export TF_VAR_managed_grafana_workspace_id=$(terraform output --raw managed_grafana_workspace_id)
```
### 6. Grafana API Key
Amazon Managed Grafana provides a control plane API for generating Grafana API keys.
As a security best practice, we will provide to Terraform a short lived API key to
run the `apply` or `destroy` command.
Ensure you have necessary IAM permissions (`CreateWorkspaceApiKey, DeleteWorkspaceApiKey`)
```bash
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
```
## Deploy
Simply run this command to deploy.
```bash
terraform apply
```
## Visualization
1. Prometheus datasource on Grafana
Open your Grafana workspace and under Configuration -> Data sources, you will see `aws-observability-accelerator`. Open and click `Save & test`. You will then see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
2. Grafana dashboards
Go to the Dashboards panel of your Grafana workspace. There will be a folder called `Observability Accelerator Dashboards`
<img width="832" alt="image" src="https://user-images.githubusercontent.com/97046295/194903648-57c55d30-6f90-4b03-9eb6-577aaba7dc22.png">
Open the "Java/JMX" dashboard to view its visualization
<img width="2560" alt="Grafana Java dashboard" src="https://user-images.githubusercontent.com/10175027/217821001-2119c81f-94bd-4811-8bbb-caaf1ae5a77a.png">
2. Amazon Managed Service for Prometheus rules and alerts
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed.
<img width="1314" alt="image" src="https://user-images.githubusercontent.com/97046295/194904104-09a28577-d149-478e-b0a1-dc21cb7effc1.png">
!!! note
To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html)
## Deploy an Example Java Application
In this section we will reuse an example from the AWS OpenTelemetry collector [repository](https://github.com/aws-observability/aws-otel-collector/blob/main/docs/developers/container-insights-eks-jmx.md). For convenience, the steps can be found below.
1. Clone [this repository](https://github.com/aws-observability/aws-otel-test-framework) and navigate to the `sample-apps/jmx/` directory.
2. Authenticate to Amazon ECR
```sh
export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
export AWS_REGION={region}
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
```
3. Create an Amazon ECR repository
```sh
aws ecr create-repository --repository-name prometheus-sample-tomcat-jmx \
--image-scanning-configuration scanOnPush=true \
--region $AWS_REGION
```
4. Build Docker image and push to ECR.
```sh
docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest .
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest
```
5. Install sample application
```sh
export SAMPLE_TRAFFIC_NAMESPACE=javajmx-sample
curl https://raw.githubusercontent.com/aws-observability/aws-otel-test-framework/terraform/sample-apps/jmx/examples/prometheus-metrics-sample.yaml > metrics-sample.yaml
sed -i "s/{{aws_account_id}}/$AWS_ACCOUNT_ID/g" metrics-sample.yaml
sed -i "s/{{region}}/$AWS_REGION/g" metrics-sample.yaml
sed -i "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" metrics-sample.yaml
kubectl apply -f metrics-sample.yaml
```
Verify that the sample application is running:
```sh
kubectl get pods -n $SAMPLE_TRAFFIC_NAMESPACE
NAME READY STATUS RESTARTS AGE
tomcat-bad-traffic-generator 1/1 Running 0 11s
tomcat-example-7958666589-2q755 0/1 ContainerCreating 0 11s
tomcat-traffic-generator 1/1 Running 0 11s
```
## Destroy resources
If you leave this stack running, you will continue to incur charges. To remove all resources
created by Terraform, [refresh your Grafana API key](#6-grafana-api-key) and run the command below.
!!! warning
Be careful, this command will removing everything created by Terraform. If you wish
to keep your Amazon Managed Grafana or Amazon Managed Service for Prometheus workspaces. Remove them
from your terraform state before running the destroy command.
```bash
terraform destroy
```
To remove resources from your Terraform state, run
```bash
# grafana workspace
terraform state rm "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
# prometheus workspace
terraform state rm "module.eks_observability_accelerator.aws_prometheus_workspace.this[0]"
```
docs/workloads/nginx.md
-198
View File
@@ -1,198 +0,0 @@
# Monitor Nginx applications running on Amazon EKS
The current example deploys the [nginx workload module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/nginx),
to provide an existing EKS cluster with an OpenTelemetry collector,
curated Grafana dashboards, Prometheus alerting and recording rules with multiple
configuration options on the cluster infrastructure.
## Prerequisites
!!! note
Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.
## Setup
### 1. Download sources and initialize Terraform
```bash
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
cd examples/existing-cluster-nginx
terraform init
```
### 2. AWS Region
Specify the AWS Region where the resources will be deployed:
```bash
export TF_VAR_aws_region=xxx
```
### 3. Amazon EKS Cluster
To run this example, you need to provide your EKS cluster name. If you don't
have a cluster ready, visit [this example](https://aws-observability.github.io/terraform-aws-observability-accelerator/helpers/new-eks-cluster/)
first to create a new one.
Specify your cluster name:
```bash
export TF_VAR_eks_cluster_id=xxx
```
### 4. Amazon Managed Service for Prometheus workspace (optional)
By default, we create an Amazon Managed Service for Prometheus workspace for you.
However, if you have an existing workspace you want to reuse, edit and run:
```bash
export TF_VAR_managed_prometheus_workspace_id=ws-xxx
```
To create a workspace outside of Terraform's state, simply run:
```bash
aws amp create-workspace --alias observability-accelerator --query '.workspaceId' --output text
```
### 5. Amazon Managed Grafana workspace
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, edit and run:
```bash
export TF_VAR_managed_grafana_workspace_id=g-xxx
```
To create a new one, within this example's Terraform state (sharing the same lifecycle with all the other resources):
- Edit main.tf and set `enable_managed_grafana = true`
- Run
```bash
terraform init
terraform apply -target "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
export TF_VAR_managed_grafana_workspace_id=$(terraform output --raw managed_grafana_workspace_id)
```
### 6. Grafana API Key
Amazon Managed Grafana provides a control plane API for generating Grafana API keys.
As a security best practice, we will provide to Terraform a short lived API key to
run the `apply` or `destroy` command.
Ensure you have necessary IAM permissions (`CreateWorkspaceApiKey, DeleteWorkspaceApiKey`)
```bash
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
```
## Deploy
Simply run this command to deploy.
```bash
terraform apply
```
## Visualization
### 1. Prometheus datasource on Grafana
Open your Grafana workspace and under Configuration -> Data sources, you will see `aws-observability-accelerator`. Open and click `Save & test`. You will see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
### 2. Grafana dashboards
Go to the Dashboards panel of your Grafana workspace. You will see a list of dashboards under the `Observability Accelerator Dashboards`
<img width="1208" alt="image" src="https://user-images.githubusercontent.com/97046295/190665211-60faef71-d83d-4d59-ac80-bf4309d8c082.png">
Open the NGINX dashboard and you will be able to view its visualization
<img width="1850" alt="image" src="https://user-images.githubusercontent.com/97046295/196226043-e49afeb9-7828-467f-9199-5707cdc69aa9.png">
### 3. Amazon Managed Service for Prometheus rules and alerts
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed.
<img width="1054" alt="image" src="https://user-images.githubusercontent.com/97046295/190665728-ae8bb709-ad93-4629-b845-85c158dd1925.png">
!!! note
To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html)
## Deploy an Example Application to Visualize
In this section we will deploy sample application and extract metrics using AWS OpenTelemetry collector
### 1. Add the helm incubator repo:
```sh
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
```
### 2. Enter the following command to create a new namespace:
```sh
kubectl create namespace nginx-ingress-sample
```
### 3. Enter the following commands to install NGINX:
```sh
helm install my-nginx ingress-nginx/ingress-nginx \
--namespace nginx-ingress-sample \
--set controller.metrics.enabled=true \
--set-string controller.metrics.service.annotations."prometheus\.io/port"="10254" \
--set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true"
```
### 4. Set an EXTERNAL-IP variable to the value of the EXTERNAL-IP column in the row of the NGINX ingress controller.
```sh
EXTERNAL_IP=your-nginx-controller-external-ip
```
### 5. Start some sample NGINX traffic by entering the following command.
```sh
SAMPLE_TRAFFIC_NAMESPACE=nginx-sample-traffic
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/master/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_traffic/nginx-traffic/nginx-traffic-sample.yaml |
sed "s/{{external_ip}}/$EXTERNAL_IP/g" |
sed "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" |
kubectl apply -f -
```
### 6. Verify if the application is running
```sh
kubectl get pods -n nginx-ingress-sample
```
### 7. Visualize the Application's dashboard
Log back into your Managed Grafana Workspace and navigate to the dashboard side panel, click on `Observability Accelerator Dashboards` Folder and open the `NGINX` Dashboard.
## Destroy resources
If you leave this stack running, you will continue to incur charges. To remove all resources
created by Terraform, [refresh your Grafana API key](#6-grafana-api-key) and run the command below.
!!! warning
Be careful, this command will removing everything created by Terraform. If you wish
to keep your Amazon Managed Grafana or Amazon Managed Service for Prometheus workspaces. Remove them
from your terraform state before running the destroy command.
```bash
terraform destroy
```
To remove resources from your Terraform state, run
```bash
# grafana workspace
terraform state rm "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
# prometheus workspace
terraform state rm "module.eks_observability_accelerator.aws_prometheus_workspace.this[0]"
```