mirror of
https://github.com/storytold/terraform-aws-observability-accelerator.git
synced 2026-10-09 00:09:43 +00:00
Kevin Lewin
116 lines
4.3 KiB
Markdown
116 lines
4.3 KiB
Markdown
# Monitoring Amazon Managed Service for Prometheus workspaces
|
|
|
|
This example allows you to monitor your Amazon Managed Service for Prometheus workspaces
|
|
using Amazon CloudWatch vended metrics and logs. It also creates configurable CloudWatch
|
|
alarms for service usage limits. Those informations are displayed in a Managed Grafana
|
|
workspace dashboard.
|
|
|
|
## Prerequisites
|
|
|
|
!!! note
|
|
Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites)
|
|
before proceeding.
|
|
This example doesn't require an Amazon EKS cluster and Kubernetes tools (ex. `kubectl`).
|
|
|
|
!!! note
|
|
This example requires [CloudWatch Billing Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html#turning_on_billing_metrics) to be enabled in order to create some of the alarms in this module.
|
|
|
|
## Setup
|
|
|
|
|
|
### 1. Download sources and initialize Terraform
|
|
|
|
```bash
|
|
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
|
|
cd examples/managed-prometheus-monitoring
|
|
terraform init
|
|
```
|
|
|
|
### 2. AWS Region
|
|
|
|
Specify the AWS Region where the resources will be deployed:
|
|
|
|
```bash
|
|
export TF_VAR_aws_region=xxx
|
|
```
|
|
|
|
### 3. Amazon Managed Service for Prometheus workspace
|
|
|
|
Specify one or more workspaces in the same Region separated with a comma seperated string.
|
|
|
|
```bash
|
|
export TF_VAR_managed_prometheus_workspace_id="ws-xxx"
|
|
```
|
|
|
|
You can use the following command to create alarms for all of the workspaces in a region.
|
|
|
|
```sh
|
|
export TF_VAR_managed_prometheus_workspace_id=$(aws amp list-workspaces --query 'workspaces[].workspaceId' --output text | sed -E 's/\t/,/g')
|
|
```
|
|
|
|
### 4. Amazon Managed Grafana workspace
|
|
|
|
To run this example you need an Amazon Managed Grafana workspace.
|
|
|
|
```bash
|
|
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
|
```
|
|
|
|
### 5. Grafana API Key
|
|
|
|
Amazon Managed Grafana provides a control plane API for generating Grafana API keys.
|
|
As a security best practice, we will provide to Terraform a short lived API key to
|
|
run the `apply` or `destroy` command.
|
|
|
|
Ensure you have necessary IAM permissions (`CreateWorkspaceApiKey, DeleteWorkspaceApiKey`)
|
|
|
|
```bash
|
|
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
|
```
|
|
|
|
## Deploy
|
|
|
|
Simply run this command to deploy the example
|
|
|
|
```sh
|
|
terraform apply
|
|
```
|
|
|
|
## Visualization
|
|
|
|
### 1. Cloudwatch datasource on Grafana
|
|
|
|
Open your Grafana workspace and under Configuration -> Data sources, you should see `aws-observability-accelerator-cloudwatch`. Open and click `Save & test`. You should see a notification confirming that the CloudWatch datasource is ready to be used on Grafana.
|
|
|
|
### 2. Grafana dashboards
|
|
|
|
Go to the Dashboards panel of your Grafana workspace. You should see a list of dashboards under the `AMP Monitoring Dashboards` folder.
|
|
|
|
Open the `AMP Accelerator Dashboard` to see a visualization of the AMP workspace.
|
|
|
|
<img width="1786" alt="Screen Shot 2022-10-11 at 2 16 17 PM" src="https://user-images.githubusercontent.com/97046295/196742772-fba1a5fb-dd38-445c-88a9-607f38994713.png">
|
|
|
|
### 3. Amazon Managed Service for Prometheus CloudWatch Alarms.
|
|
|
|
Open the CloudWatch console and click `Alarms` > `All Alarms` to review the service limit alarms.
|
|
|
|
<img width="1525" alt="image" src="https://user-images.githubusercontent.com/97046295/196742923-876e3b1c-6f2a-419d-ad39-9c057a0f7650.png">
|
|
|
|
In us-east-1 region an alarm is created for billing. This alarm utilizes anomaly detection to detect anomalies in the Estimated Charges billing metric.
|
|
|
|
<img width="1346" alt="image" src="https://user-images.githubusercontent.com/97046295/197042518-a98d69df-8f53-4a4a-afb8-f424d91da56f.png">
|
|
|
|
## Destroy resources
|
|
|
|
If you leave this stack running, you will continue to incur charges. To remove all resources
|
|
created by Terraform, [refresh your Grafana API key](#5-grafana-api-key) and run the command below.
|
|
|
|
!!! warning
|
|
Be careful, this command will remove everything created by Terraform. If you wish
|
|
to keep your Amazon Managed Grafana Dashboards or CloudWatch Alarms. Remove them
|
|
from your terraform state before running the destroy command.
|
|
|
|
```bash
|
|
terraform destroy
|
|
```
|