terraform-aws-observability-accelerator/modules/eks-monitoring/README.md

Infrastructure monitoring

This module provides EKS cluster monitoring with the following resources:

• AWS Distro For OpenTelemetry Operator and Collector for Metrics and Traces
• Logs with AWS for FluentBit
• Installs Grafana Operator to add AWS data sources and create Grafana Dashboards to Amazon Managed Grafana.
• Installs FluxCD to perform GitOps sync of a Git Repo to EKS Cluster. We will use this later for creating Grafana Dashboards and AWS datasources to Amazon Managed Grafana.
• Installs External Secrets Operator to retrieve and Sync the Grafana API keys.
• Amazon Managed Grafana Dashboard and data source
• Alerts and recording rules with AWS Managed Service for Prometheus

This module makes use of the open source kube-prometheus-stack

Requirements

Name	Version
terraform	>= 1.1.0
aws	>= 4.0.0
grafana	>= 1.25.0
helm	>= 2.4.1
kubectl	>= 1.14
kubernetes	>= 2.10

Providers

Name	Version
aws	>= 4.0.0
helm	>= 2.4.1
kubectl	>= 1.14

Modules

Name	Source	Version
external_secrets	./add-ons/external-secrets	n/a
fluentbit_logs	./add-ons/aws-for-fluentbit	n/a
helm_addon	github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons/helm-addon	v4.32.0
java_monitoring	./patterns/java	n/a
nginx_monitoring	./patterns/nginx	n/a
operator	./add-ons/adot-operator	n/a

Resources

Name	Type
aws_prometheus_rule_group_namespace.alerting_rules	resource
aws_prometheus_rule_group_namespace.recording_rules	resource
helm_release.fluxcd	resource
helm_release.grafana_operator	resource
helm_release.kube_state_metrics	resource
helm_release.prometheus_node_exporter	resource
kubectl_manifest.flux_gitrepository	resource
kubectl_manifest.flux_kustomization	resource
aws_caller_identity.current	data source
aws_eks_cluster.eks_cluster	data source
aws_partition.current	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
custom_metrics_config	Configuration object to enable custom metrics collection	object({ ports = list(number) # paths = optional(list(string), ["/metrics"]) # list of samples to be dropped by label prefix, ex: go_ -> discards go_.* dropped_series_prefixes = list(string) })	{ "dropped_series_prefixes": [ "unspecified" ], "ports": [] }	no
dashboards_folder_id	Grafana folder ID for automatic dashboards	string	n/a	yes
eks_cluster_id	EKS Cluster Id	string	n/a	yes
enable_alerting_rules	Enables or disables Managed Prometheus alerting rules	bool	true	no
enable_amazon_eks_adot	Enables the ADOT Operator on the EKS Cluster	bool	true	no
enable_cert_manager	Allow reusing an existing installation of cert-manager	bool	true	no
enable_custom_metrics	Allows additional metrics collection for config elements in the custom_metrics_config config object. Automatic dashboards are not included	bool	false	no
enable_dashboards	Enables or disables curated dashboards	bool	true	no
enable_external_secrets	Installs External Secrets to EKS Cluster	bool	true	no
enable_fluxcd	Enables or disables FluxCD. Disabling this might affect some data in the dashboards	bool	true	no
enable_grafana_operator	Deploys Grafana Operator to EKS Cluster	bool	true	no
enable_java	Enable Java workloads monitoring, alerting and default dashboards	bool	false	no
enable_kube_state_metrics	Enables or disables Kube State metrics exporter. Disabling this might affect some data in the dashboards	bool	true	no
enable_logs	Using AWS For FluentBit to collect cluster and application logs to Amazon CloudWatch	bool	true	no
enable_nginx	Enable NGINX workloads monitoring, alerting and default dashboards	bool	false	no
enable_node_exporter	Enables or disables Node exporter. Disabling this might affect some data in the dashboards	bool	true	no
enable_recording_rules	Enables or disables Managed Prometheus recording rules	bool	true	no
enable_tracing	(Experimental) Enables tracing with AWS X-Ray. This changes the deploy mode of the collector to daemon set. Requirement: adot add-on <= 0.58-build.0	bool	false	no
flux_config	FluxCD configuration	object({ create_namespace = bool k8s_namespace = string helm_chart_name = string helm_chart_version = string helm_release_name = string helm_repo_url = string helm_settings = map(string) helm_values = map(any) })	{ "create_namespace": true, "helm_chart_name": "flux2", "helm_chart_version": "2.7.0", "helm_release_name": "observability-fluxcd-addon", "helm_repo_url": "https://fluxcd-community.github.io/helm-charts", "helm_settings": {}, "helm_values": {}, "k8s_namespace": "flux-system" }	no
flux_gitrepository_branch	Flux GitRepository Branch	string	"feature/allDashboards"	no
flux_gitrepository_url	Flux GitRepository URL	string	"https://github.com/aws-observability/aws-observability-accelerator"	no
flux_kustomization_path	Flux Kustomization Path	string	"./artifacts/grafana-operator-manifests"	no
flux_name	Flux GitRepository and Kustomization Name	string	"grafana-dashboards"	no
go_config	Grafana Operator configuration	object({ create_namespace = bool helm_chart = string helm_name = string k8s_namespace = string helm_release_name = string helm_chart_version = string })	{ "create_namespace": true, "helm_chart": "oci://ghcr.io/grafana-operator/helm-charts/grafana-operator", "helm_chart_version": "v5.0.0-rc1", "helm_name": "grafana-operator", "helm_release_name": "grafana-operator", "k8s_namespace": "grafana-operator" }	no
grafana_api_key	Grafana API key for the Amazon Managed Grafana workspace	string	n/a	yes
grafana_cluster_dashboard_url	Dashboard URL for Cluster Grafana Dashboard JSON	string	"https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/1d731aca31cdeb26e9fe9d017e609a5ba1621a30/artifacts/grafana-dashboards/cluster.json"	no
grafana_kubelet_dashboard_url	Dashboard URL for Kubelet Grafana Dashboard JSON	string	"https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/1d731aca31cdeb26e9fe9d017e609a5ba1621a30/artifacts/grafana-dashboards/kubelet.json"	no
grafana_namespace_workloads_dashboard_url	Dashboard URL for Namespace Workloads Grafana Dashboard JSON	string	"https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/1d731aca31cdeb26e9fe9d017e609a5ba1621a30/artifacts/grafana-dashboards/namespace-workloads.json"	no
grafana_node_exporter_dashboard_url	Dashboard URL for Node Exporter Grafana Dashboard JSON	string	"https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/nodeexporter-nodes.json"	no
grafana_nodes_dashboard_url	Dashboard URL for Nodes Grafana Dashboard JSON	string	"https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/1d731aca31cdeb26e9fe9d017e609a5ba1621a30/artifacts/grafana-dashboards/nodes.json"	no
grafana_url	Endpoint URL of Amazon Managed Grafana workspace	string	n/a	yes
grafana_workloads_dashboard_url	Dashboard URL for Workloads Grafana Dashboard JSON	string	"https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/1d731aca31cdeb26e9fe9d017e609a5ba1621a30/artifacts/grafana-dashboards/workloads.json"	no
helm_config	Helm Config for Prometheus	any	{}	no
irsa_iam_permissions_boundary	IAM permissions boundary for IRSA roles	string	null	no
irsa_iam_role_path	IAM role path for IRSA roles	string	"/"	no
java_config	Configuration object for Java/JMX monitoring	object({ enable_alerting_rules = bool enable_recording_rules = bool scrape_sample_limit = number })	{ "enable_alerting_rules": true, "enable_recording_rules": true, "scrape_sample_limit": 1000 }	no
ksm_config	Kube State metrics configuration	object({ create_namespace = bool k8s_namespace = string helm_chart_name = string helm_chart_version = string helm_release_name = string helm_repo_url = string helm_settings = map(string) helm_values = map(any) scrape_interval = string scrape_timeout = string })	{ "create_namespace": true, "helm_chart_name": "kube-state-metrics", "helm_chart_version": "4.24.0", "helm_release_name": "kube-state-metrics", "helm_repo_url": "https://prometheus-community.github.io/helm-charts", "helm_settings": {}, "helm_values": {}, "k8s_namespace": "kube-system", "scrape_interval": "60s", "scrape_timeout": "15s" }	no
logs_config	Configuration object for logs collection	object({ cw_log_retention_days = number })	{ "cw_log_retention_days": 90 }	no
managed_prometheus_workspace_endpoint	Amazon Managed Prometheus Workspace Endpoint	string	""	no
managed_prometheus_workspace_id	Amazon Managed Prometheus Workspace ID	string	null	no
managed_prometheus_workspace_region	Amazon Managed Prometheus Workspace's Region	string	null	no
ne_config	Node exporter configuration	object({ create_namespace = bool k8s_namespace = string helm_chart_name = string helm_chart_version = string helm_release_name = string helm_repo_url = string helm_settings = map(string) helm_values = map(any) scrape_interval = string scrape_timeout = string })	{ "create_namespace": true, "helm_chart_name": "prometheus-node-exporter", "helm_chart_version": "4.14.0", "helm_release_name": "prometheus-node-exporter", "helm_repo_url": "https://prometheus-community.github.io/helm-charts", "helm_settings": {}, "helm_values": {}, "k8s_namespace": "prometheus-node-exporter", "scrape_interval": "60s", "scrape_timeout": "60s" }	no
nginx_config	Configuration object for NGINX monitoring	object({ enable_alerting_rules = bool scrape_sample_limit = number prometheus_metrics_endpoint = string })	{ "enable_alerting_rules": true, "prometheus_metrics_endpoint": "metrics", "scrape_sample_limit": 1000 }	no
prometheus_config	Controls default values such as scrape interval, timeouts and ports globally	object({ global_scrape_interval = string global_scrape_timeout = string })	{ "global_scrape_interval": "60s", "global_scrape_timeout": "15s" }	no
tags	Additional tags (e.g. map('BusinessUnit,XYZ)	map(string)	{}	no
target_secret_name	Target secret in Kubernetes to store the Grafana API Key Secret	string	"grafana-admin-credentials"	no
target_secret_namespace	Target namespace of secret in Kubernetes to store the Grafana API Key Secret	string	"grafana-operator"	no
tracing_config	Configuration object for traces collection to AWS X-Ray	object({ otlp_grpc_endpoint = string otlp_http_endpoint = string send_batch_size = number timeout = string })	{ "otlp_grpc_endpoint": "0.0.0.0:4317", "otlp_http_endpoint": "0.0.0.0:4318", "send_batch_size": 50, "timeout": "30s" }	no

Outputs

Name	Description
eks_cluster_id	EKS Cluster Id
eks_cluster_version	EKS Cluster version
grafana_dashboard_urls	URLs for dashboards created

Troubleshooting

1. When you upgrade the eks-monitoring module from v2.1.0 or earlier, the following error may occur.

Error: cannot patch "prometheus-node-exporter" with kind DaemonSet: DaemonSet.apps "prometheus-node-exporter" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"prometheus-node-exporter", "app.kubernetes.io/name":"prometheus-node-exporter"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

This is due to the upgrade of the node-exporter chart from v2 to v4. Manually delete the node-exporter's DaemonSet as described in the link here, and then apply.

kubectl -n prometheus-node-exporter delete daemonset -l app=prometheus-node-exporter
terraform apply

2. In case you dont see the grafana dashboards in your Amazon Managed Grafana console, check on the logs on your grafana operator pod using the below command :

kubectl get pods -n grafana-operator

Output:

NAME                                READY   STATUS    RESTARTS   AGE
grafana-operator-866d4446bb-nqq5c   1/1     Running   0          3h17m

kubectl logs grafana-operator-866d4446bb-nqq5c -n grafana-operator

Output:

1.6857285045556655e+09	ERROR	error reconciling datasource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "datasource": "grafanadatasource-sample-amp", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDatasourceReconciler).Reconcile

If you observe, the the above grafana-api-key error in the logs, your grafana API key is expired. Please use the operational procedure to update your grafana-api-key :

• First, lets create a new Grafana API key.

export GO_AMG_API_KEY=$(aws grafana create-workspace-api-key \
  --key-name "grafana-operator-key-new" \
  --key-role "ADMIN" \
  --seconds-to-live 432000 \
  --workspace-id <YOUR_WORKSPACE_ID> \
  --query key \
  --output text)

• Next, lets grab the Grafana API key secret name from AWS Secrets Manager. The keyname should start with terraform-..

aws secretsmanager list-secrets

• Finally, update the Grafana API key secret in AWS Secrets Manager using the above new Grafana API key:

aws secretsmanager update-secret \
    --secret-id  <Your Secret Name> \
    --secret-string "${GO_AMG_API_KEY}" \
    --region <Your AWS Region>

Grafana datasources and dashboards via GitOps on Amazon Managed Grafana

We have upgraded out solution to use grafana-operator and Flux to create Grafana datasources, folder and dashboards via GitOps on Amazon Managed Grafana.

The grafana-operator is a Kubernetes operator built to help you manage your Grafana instances inside and outside Kubernetes. Grafana Operator makes it possible for you to manage and create Grafana dashboards, datasources etc. declaratively between multiple instances in an easy and scalable way. Using grafana-operator it will be possible to add AWS data sources such as Amazon Managed Service for Prometheus, Amazon CloudWatch, AWS X-Ray to Amazon Managed Grafana and create Grafana dashboards on Amazon Managed Grafana from your Amazon EKS cluster. This enables us to use our Kubernetes cluster to create and manage the lifecycle of resources in Amazon Managed Grafana in a Kubernetes native way. This ultimately enables us to use GitOps mechanisms using CNCF projects such as Flux to create and manage the lifecycle of resources in Amazon Managed Grafana.

GitOps is a way of managing application and infrastructure deployment so that the whole system is described declaratively in a Git repository. It is an operational model that offers you the ability to manage the state of multiple Kubernetes clusters leveraging the best practices of version control, immutable artifacts, and automation. Flux is a declarative, GitOps-based continuous delivery tool that can be integrated into any CI/CD pipeline. It gives users the flexibility of choosing their Git provider (GitHub, GitLab, BitBucket). Now, with grafana-operator supporting the management of external Grafana instances such as Amazon Managed Grafana, operations personas can use GitOps mechanisms using CNCF projects such as Flux to create and manage the lifecycle of resources in Amazon Managed Grafana.

We have setup a GitRepository and Kustomization using flux to sync our GitHub Repository to add Grafana Datasources, folder and Dashboards to Amazon Managed Grafana using Grafana Operator. GitRepository defines a Source to produce an Artifact for a Git repository revision. Kustomization defines a pipeline for fetching, decrypting, building, validating and applying Kustomize overlays or plain Kubernetes manifests. we are also using Flux Post build variable substitution to dynamically render variables such as AMG_AWS_REGION, AMP_ENDPOINT_URL, AMG_ENDPOINT_URL,GRAFANA_NODEEXP_DASH_URL on the YAML manifests during deployment time to avoid hardcoding on the YAML manifests stored in Git repo.

We have placed our declarative code snippet to create an Amazon Managed Service For Promethes datasource and Grafana Dashboard in Amazon Managed Grafana in our AWS Observabiity Accelerator GitHub Repository. We have setup a GitRepository to point to the AWS Observabiity Accelerator GitHub Repository and Kustomization for flux to sync Git Repository with artifacts in ./artifacts/grafana-operator-manifests path in the AWS Observabiity Accelerator GitHub Repository. You can use this extension of our solution to point your own Kubernetes manifests to create Grafana Datasources and personified Grafana Dashboards of your choice using GitOps with Grafana Operator and Flux in Kubernetes native way with altering and redeploying this solution for changes to Grafana resources.