terraform-aws-observability-accelerator/examples/existing-cluster-java/README.md

Monitor Java applications running on Amazon EKS

This example demonstrates how to use the AWS Observability Accelerator Terraform modules to monitor EKS infrastructure and Java based workloads. The current example deploys the AWS Distro for OpenTelemetry Operator for Amazon EKS with its requirements and make use of an existing Amazon Managed Grafana workspace. It creates a new Amazon Managed Service for Prometheus workspace unless provided with an existing one to reuse.

Since v2.x releases, it uses the EKS monitoring module to provide an existing EKS cluster with an OpenTelemetry collector, curated Grafana dashboards, Prometheus alerting and recording rules with multiple configuration options on the cluster infrastructure. You will gain both visibility on the cluster and Java based applications.

Prerequisites

Ensure that you have the following tools installed locally:

1. aws cli v2
2. kubectl
3. terraform

Setup

This example uses a local terraform state. If you need states to be saved remotely, on Amazon S3 for example, visit the terraform remote states documentation

1. Clone the repo using the command below

git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git

2. Initialize terraform

cd examples/existing-cluster-java
terraform init

3. Amazon EKS Cluster

To run this example, you need to provide your EKS cluster name. If you don't have a cluster ready, visit this example first to create a new one.

Add your cluster name for eks_cluster_id="..." to the terraform.tfvars or use an environment variable export TF_VAR_eks_cluster_id=xxx.

4. Amazon Managed Grafana workspace

To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, create an environment variable export TF_VAR_managed_grafana_workspace_id=g-xxx.

To create a new one, visit this example.

In the URL https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com, the workspace ID would be g-xyz

5. Grafana API Key

Amazon Managed Service for Grafana provides a control plane API for generating Grafana API keys. We will provide to Terraform a short lived API key to run the apply or destroy command. Ensure you have necessary IAM permissions (CreateWorkspaceApiKey, DeleteWorkspaceApiKey)

export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`

Deploy

terraform apply -var-file=terraform.tfvars

or if you had only setup environment variables, run

terraform apply

Additional configuration

For the purpose of the example, we have provided default values for some of the variables.

1. AWS Region

Specify the AWS Region where the resources will be deployed. Edit the terraform.tfvars file and modify aws_region="...". You can also use environement variables export TF_VAR_aws_region=xxx.

2. Amazon Managed Service for Prometheus workspace

If you have an existing workspace, add managed_prometheus_workspace_id=ws-xxx or use an environment variable export TF_VAR_managed_prometheus_workspace_id=ws-xxx.

Visualization

1. Prometheus datasource on Grafana

Make sure to open the link in the output. After a successful deployment, this will open the Prometheus datasource configuration on Grafana. Click Save & test and you should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.

terraform output grafana_prometheus_datasource_test

2. Grafana dashboards

Go to the Dashboards panel of your Grafana workspace. There will be a folder called Observability Accelerator Dashboards

Open the "Java/JMX" dashboard to view its visualization

2. Amazon Managed Service for Prometheus rules and alerts

Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the Rules management tab, you will find new rules deployed.

To setup your alert receiver, with Amazon SNS, follow this documentation

Deploy an example Java application

In this section we will reuse an example from the AWS OpenTelemetry collector repository. For convenience, the steps can be found below.

1. Clone this repository and navigate to the sample-apps/jmx/ directory.

2. Authenticate to Amazon ECR

export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
export AWS_REGION={region}
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

3. Create an Amazon ECR repository

aws ecr create-repository --repository-name prometheus-sample-tomcat-jmx \
 --image-scanning-configuration scanOnPush=true \
 --region $AWS_REGION

4. Build Docker image and push to ECR.

docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest .
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/prometheus-sample-tomcat-jmx:latest

5. Install sample application

export SAMPLE_TRAFFIC_NAMESPACE=javajmx-sample
curl https://raw.githubusercontent.com/aws-observability/aws-otel-test-framework/terraform/sample-apps/jmx/examples/prometheus-metrics-sample.yaml > metrics-sample.yaml
sed -i "s/{{aws_account_id}}/$AWS_ACCOUNT_ID/g" metrics-sample.yaml
sed -i "s/{{region}}/$AWS_REGION/g" metrics-sample.yaml
sed -i "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" metrics-sample.yaml
kubectl apply -f metrics-sample.yaml

Verify that the sample application is running:

kubectl get pods -n $SAMPLE_TRAFFIC_NAMESPACE

NAME                              READY   STATUS              RESTARTS   AGE
tomcat-bad-traffic-generator      1/1     Running             0          11s
tomcat-example-7958666589-2q755   0/1     ContainerCreating   0          11s
tomcat-traffic-generator          1/1     Running             0          11s

Destroy resources

If you leave this stack running, you will continue to incur charges. To remove all resources created by Terraform, refresh your Grafana API key and run:

terraform destroy -var-file=terraform.tfvars

Requirements

Name	Version
terraform	>= 1.1.0
aws	>= 4.0.0
helm	>= 2.4.1
kubectl	>= 1.14
kubernetes	>= 2.10

Providers

Name	Version
aws	>= 4.0.0

Modules

Name	Source	Version
aws_observability_accelerator	../../	n/a
eks_monitoring	../../modules/eks-monitoring	n/a

Resources

Name	Type
aws_eks_cluster.this	data source
aws_eks_cluster_auth.this	data source

Inputs

Name	Description	Type	Default	Required
aws_region	AWS Region	string	n/a	yes
eks_cluster_id	Name of the EKS cluster	string	n/a	yes
enable_dashboards	Enables or disables curated dashboards	bool	true	no
grafana_api_key	API key for external-secrets to create secrets for grafana-operator	string	n/a	yes
managed_grafana_workspace_id	Amazon Managed Grafana Workspace ID	string	n/a	yes
managed_prometheus_workspace_id	Amazon Managed Service for Prometheus Workspace ID	string	""	no

Outputs

Name	Description
aws_region	AWS Region
eks_cluster_id	EKS Cluster Id
eks_cluster_version	EKS Cluster version
managed_prometheus_workspace_endpoint	Amazon Managed Prometheus workspace endpoint
managed_prometheus_workspace_id	Amazon Managed Prometheus workspace ID