terraform-aws-observability-accelerator/docs/eks/tracing.md

Tracing on Amazon EKS

Distributed tracing helps you have end-to-end visibility between transactions in distributed nodes. The eks-monitoring module is configured by default to collect traces into AWS X-Ray.

The AWS Distro for OpenTelemetry collector is configured to receive traces in the OTLP format (OTLP receiver), using the OpenTelemetry SDK or auto-instrumentation agents.

!!! note To disable the tracing configuration, set up enable_tracing = false in the module configuration

Instrumentation

Let's take a sample application that is already instrumented with the OpenTelemetry SDK.

!!! note To learn more about instrumenting with OpenTelemetry, please visit the OpenTelemetry documentation for your programming language.

Cloning the repo

git clone https://github.com/aws-observability/aws-otel-community.git
cd aws-otel-community/sample-apps/go-sample-app

Deploying on Amazon EKS

Using the sample application, we will build a container image, create and push an image on Amazon ECR. We will use a Kubernetes manifest to deploy to an EKS cluster.

!!! warning The following steps require that you have an EKS cluster ready. To deploy an EKS cluster, please visit our example.

Building container image

=== "amd64 linux"

```console
docker build -t go-sample-app .
```

=== "cross platform build"

```bash
docker buildx build -t go-sample-app . --platform=linux/amd64
```

Publishing on Amazon ECR

=== "using docker"

```console
export ECR_REPOSITORY_URI=$(aws ecr create-repository --repository go-sample-app --query repository.repositoryUri --output text)
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
docker tag go-sample-app:latest "${ECR_REPOSITORY_URI}:latest"
docker push "${ECR_REPOSITORY_URI}:latest"
```

Deploying on Amazon EKS

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-sample-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-sample-app
  template:
    metadata:
      labels:
        app: go-sample-app
    spec:
      containers:
        - name: go-sample-app
          image: "${ECR_REPOSITORY_URI}:latest" # make sure to replace this variable
          imagePullPolicy: Always
          env:
          - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
            value: adot-collector.adot-collector-kubeprometheus.svc.cluster.local:4317
          resources:
            limits:
              cpu:  300m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 180Mi
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: go-sample-app
  namespace: default
  labels:
    app: go-sample-app
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: go-sample-app
---
apiVersion: v1
kind: Service
metadata:
  name: go-sample-app
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: go-sample-app
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

Deploying and testing

With the Kubernetes manifest ready, run:

kubectl apply -f eks.yaml

You should see the pods running with the command:

kubectl get pods
NAME                              READY   STATUS    RESTARTS        AGE
go-sample-app-67c48ff8c6-bdw74    1/1     Running   0               4s
go-sample-app-67c48ff8c6-t6k2j    1/1     Running   0               4s

To simulate some traffic you can forward the service port to your local host and test a few queries

kubectl port-forward deployment/go-sample-app 8080:8080

Test a few endpoints

curl http://localhost:8080/
curl http://localhost:8080/outgoing-http-call
curl http://localhost:8080/aws-sdk-call
curl http://localhost:8080/outgoing-sampleapp

Visualizing traces

As this is a basic example, the service map doesn't have a lot of nodes, but this shows you how to setup tracing in your application and deploying it on Amazon EKS using the eks-monitoring module.

With Flux and Grafana Operator, the eks-monitoring module configures an AWS X-Ray data source on your provided Grafana workspace. Open the Grafana explorer view and select the X-Ray data source. If you type the query below, and select Trace List for Query Type, you should see the list of traces occured in the selected timeframe.

You can add the service map to a dashboard, for example a service focused dashboard. You can click on any of the traces to view a node map and the traces details.

There is a button that can take you the CloudWatch console to view the same data. If your logs are stored on CloudWatch Logs, this page can present all the logs in the trace details page. The CloudWatch Log Group name should be added to the trace as an attribute. Read more about this in our One Observability Workshop

Resoures

• AWS Observability Best Practices
• One Observability Workshop
• AWS Distro for OpenTelemetry documentation
• AWS X-Ray user guide
• OpenTelemetry documentation