mirror of
https://github.com/storytold/terraform-aws-observability-accelerator.git
synced 2026-10-09 00:09:43 +00:00
Move all dashboards to GitOps (#175)
* Typo * Remove Grafana provider * Temp: move dashbaords to gitOps * Move external labels to resource attributes * Avoid DDoS with using 0.0.0.0 * Pre-commit * Transition in two steps Will need to remove provider in a separate version to provide a transition path as removing this will break terraform and leave orphans in the state * Move patterns' dashboards creation to gitOps Standardize config objects for patterns as well * Pre-commit * Create AMP dashboard from external source with Grafana provider * Fix deprecated option * Fix Flux requirements * Run pre-commit * Update example with operator * Cleanup examples * Update multicluster example * Update multicluster example * Drop dead variable * Update docs * Change GitOps branch name * Update docs * Replacing Secrets Manager to SSM to store Grafana API Key (#178) * Fixing SSM * Fixing SSM * Replacing Secrets Manager with SSM * Replacing Secrets Manager with SSM * Update architecture diagram * Update architecture diagram * Update README.md * Update index.md * Fixing Grafana Operator Version * Fix multicluster example * Update docs --------- Co-authored-by: Ela AWS <51791117+elamaran11@users.noreply.github.com> Co-authored-by: Elamaran Shanmugam <elamaran.shan@gmail.com>
This commit is contained in:
Rodrigue Koffi
+38
-22
@@ -111,26 +111,41 @@ terraform apply
|
||||
|
||||
## Visualization
|
||||
|
||||
#### 1. Prometheus data source on Grafana
|
||||
|
||||
Make sure to open the link in the output. After a successful deployment, this will open
|
||||
the Prometheus data source configuration on Grafana.
|
||||
Click `Save & test` and you should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
|
||||
|
||||
```bash
|
||||
terraform output grafana_prometheus_datasource_test
|
||||
```
|
||||
|
||||
#### 2. Grafana dashboards
|
||||
|
||||
Go to the Dashboards panel of your Grafana workspace. You should see a list of dashboards under the `Observability Accelerator Dashboards`
|
||||
#### 1. Grafana dashboards
|
||||
|
||||
Login to your Grafana workspace and navigate to the Dashboards panel. You should see a list of dashboards under the `Observability Accelerator Dashboards`
|
||||
<img width="1540" alt="image" src="https://user-images.githubusercontent.com/10175027/190000716-29e16698-7c90-49d6-8c37-79ca1790e2cc.png">
|
||||
|
||||
Open a specific dashboard and you should be able to view its visualization
|
||||
|
||||
<img width="2056" alt="cluster headlines" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">
|
||||
|
||||
With v2.5 and above, the dashboards are managed with a Grafana Operator running in your cluster.
|
||||
From the cluster to view all dashboards as Kubernetes objects, run
|
||||
|
||||
```console
|
||||
kubectl get grafanadashboards -A
|
||||
NAMESPACE NAME AGE
|
||||
grafana-operator cluster-grafanadashboard 138m
|
||||
grafana-operator java-grafanadashboard 143m
|
||||
grafana-operator kubelet-grafanadashboard 13h
|
||||
grafana-operator namespace-workloads-grafanadashboard 13h
|
||||
grafana-operator nginx-grafanadashboard 134m
|
||||
grafana-operator node-exporter-grafanadashboard 13h
|
||||
grafana-operator nodes-grafanadashboard 13h
|
||||
grafana-operator workloads-grafanadashboard 13h
|
||||
```
|
||||
|
||||
You can inspect more details per dashboard using this command
|
||||
|
||||
```console
|
||||
kubectl describe grafanadashboards cluster-grafanadashboard -n grafana-operator
|
||||
```
|
||||
|
||||
Grafana Operator and Flux always work together to synchronize your dashboards with Git.
|
||||
If you delete your dashboards by accident, they will be re-provisioned automatically.
|
||||
|
||||
|
||||
#### 3. Amazon Managed Service for Prometheus rules and alerts
|
||||
|
||||
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you should find new rules deployed.
|
||||
@@ -216,21 +231,22 @@ export GO_AMG_API_KEY=$(aws grafana create-workspace-api-key \
|
||||
--output text)
|
||||
```
|
||||
|
||||
- Next, lets grab the Grafana API key secret name from AWS Secrets Manager. The keyname should start with `terraform-..`
|
||||
|
||||
```bash
|
||||
aws secretsmanager list-secrets
|
||||
```
|
||||
|
||||
- Finally, update the Grafana API key secret in AWS Secrets Manager using the above new Grafana API key:
|
||||
|
||||
```bash
|
||||
aws secretsmanager update-secret \
|
||||
--secret-id <Your Secret Name> \
|
||||
--secret-string "{\"GF_SECURITY_ADMIN_APIKEY\": \"${GO_AMG_API_KEY}\"}" \
|
||||
aws aws ssm put-parameter \
|
||||
--name "/terraform-accelerator/grafana-api-key" \
|
||||
--type "SecureString" \
|
||||
--value "{\"GF_SECURITY_ADMIN_APIKEY\": \"${GO_AMG_API_KEY}\"}" \
|
||||
--region <Your AWS Region>
|
||||
```
|
||||
|
||||
- If the issue persists, you can force the synchronization by deleting the `externalsecret` Kubernetes object.
|
||||
|
||||
```bash
|
||||
kubectl delete externalsecret/external-secrets-sm -n grafana-operator
|
||||
```
|
||||
|
||||
### 2. Upgrade from 2.1.0 or earlier
|
||||
|
||||
When you upgrade the eks-monitoring module from v2.1.0 or earlier, the following error may occur.
|
||||
|
||||
+1
-1
@@ -32,7 +32,7 @@ Make sure to refresh your temporary Grafana API key
|
||||
|
||||
```bash
|
||||
export TF_VAR_managed_grafana_workspace_id=g-xxx
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 7200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
@@ -11,7 +11,7 @@ Using the example [eks-cluster-with-vpc](https://aws-observability.github.io/ter
|
||||
1. `eks-cluster-1`
|
||||
2. `eks-cluster-2`
|
||||
|
||||
#### 2. Amazon Managed Serivce for Prometheus (AMP) workspace
|
||||
#### 2. Amazon Managed Service for Prometheus (AMP) workspace
|
||||
|
||||
We recommend that you create a new AMP workspace. To do that you can run the following command.
|
||||
|
||||
@@ -48,7 +48,7 @@ Ensure you have the following necessary IAM permissions
|
||||
* `grafana.DeleteWorkspaceApiKey`
|
||||
|
||||
```sh
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 7200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
|
||||
```
|
||||
|
||||
## Setup
|
||||
@@ -70,8 +70,8 @@ Verify by looking at the file `variables.tf` that there are two EKS clusters tar
|
||||
|
||||
The difference in deployment between these clusters is that Terraform, when setting up the EKS cluster behind variable `eks_cluster_1_id` for observability, also sets up:
|
||||
|
||||
* Dashboard folder and files in `AMG`
|
||||
* Prometheus and Java, alerting and recording rules in `AMP`
|
||||
* Dashboard folder and files in Amazon Managed Grafana
|
||||
* Prometheus and Java, alerting and recording rules in Amazon Managed Service for Prometheus
|
||||
|
||||
!!! warning
|
||||
To override the defaults, create a `terraform.tfvars` and change the default values of the variables.
|
||||
|
||||
Reference in New Issue
Block a user