storytold/terraform-aws-observability-accelerator
2
0
Fork 0
mirror of https://github.com/storytold/terraform-aws-observability-accelerator.git synced 2026-10-17 00:23:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: Update troubleshooting and navigation (#228)

Browse Source 
* Update docs structure with regrouped patterns

* Rename multi cluster to avoid ambiguity with cross account

* Separate troubleshooting for eks-monitoring
This commit is contained in:
Rodrigue Koffi
2023-09-06 17:41:47 +02:00
committed by GitHub
parent 9498a06d3e
commit f65afffd58
4 changed files with 272 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/eks/index.md
-73
View File
@@ -191,76 +191,3 @@ sum(up{job="custom-metrics"}) by (container_name, cluster, nodename)
```
<img width="2560" alt="Screenshot 2023-01-31 at 11 16 21" src="https://user-images.githubusercontent.com/10175027/215869004-e05f557d-c81a-41fb-a452-ede9f986cb27.png">
## Troubleshooting
### 1. Grafana dashboards missing or Grafana API key expired
In case you don't see the grafana dashboards in your Amazon Managed Grafana console, check on the logs on your grafana operator pod using the below command :
```bash
kubectl get pods -n grafana-operator
```
Output:
```console
NAME READY STATUS RESTARTS AGE
grafana-operator-866d4446bb-nqq5c 1/1 Running 0 3h17m
```
```bash
kubectl logs grafana-operator-866d4446bb-nqq5c -n grafana-operator
```
Output:
```console
1.6857285045556655e+09 ERROR error reconciling datasource {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "datasource": "grafanadatasource-sample-amp", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDatasourceReconciler).Reconcile
```
If you observe, the the above `grafana-api-key error` in the logs, your grafana API key is expired. Please use the operational procedure to update your `grafana-api-key` :
- First, lets create a new Grafana API key.
```bash
export GO_AMG_API_KEY=$(aws grafana create-workspace-api-key \
--key-name "grafana-operator-key-new" \
--key-role "ADMIN" \
--seconds-to-live 432000 \
--workspace-id <YOUR_WORKSPACE_ID> \
--query key \
--output text)
```
- Finally, update the Grafana API key secret in AWS SSM Parameter Store using the above new Grafana API key:
```bash
aws aws ssm put-parameter \
--name "/terraform-accelerator/grafana-api-key" \
--type "SecureString" \
--value "{\"GF_SECURITY_ADMIN_APIKEY\": \"${GO_AMG_API_KEY}\"}" \
--region <Your AWS Region>
```
- If the issue persists, you can force the synchronization by deleting the `externalsecret` Kubernetes object.
```bash
kubectl delete externalsecret/external-secrets-sm -n grafana-operator
```
### 2. Upgrade from 2.1.0 or earlier
When you upgrade the eks-monitoring module from v2.1.0 or earlier, the following error may occur.
```bash
Error: cannot patch "prometheus-node-exporter" with kind DaemonSet: DaemonSet.apps "prometheus-node-exporter" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"prometheus-node-exporter", "app.kubernetes.io/name":"prometheus-node-exporter"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
```
This is due to the upgrade of the node-exporter chart from v2 to v4. Manually delete the node-exporter's DaemonSet as described in [the link here](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter#3x-to-4x), and then apply.
```bash
kubectl -n prometheus-node-exporter delete daemonset -l app=prometheus-node-exporter
terraform apply
```