storytold/terraform-aws-observability-accelerator
2
0
Fork 0
mirror of https://github.com/storytold/terraform-aws-observability-accelerator.git synced 2026-10-09 00:09:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Improve tracing (#195)

Browse Source 
* Dropping daemonset option

* Update adot loglevel config

* Enable tracing by default

* Init tracing docs

* Add tracing docs

* Update tracing.md

* Update collector endpoint

* Fix tracing instructions
This commit is contained in:
Rodrigue Koffi
2023-08-07 20:59:18 +02:00
committed by GitHub
parent e6083b163f
commit 26371d5210
5 changed files with 218 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/eks/tracing.md
+208
View File
@@ -0,0 +1,208 @@
# Tracing on Amazon EKS
[Distributed tracing](https://aws-observability.github.io/observability-best-practices/signals/traces/)
helps you have end-to-end visibility between transactions in distributed nodes.
The `eks-monitoring` module is configured by default to collect traces into
[AWS X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html).
The AWS Distro for OpenTelemetry collector is configured to receive traces
in the OTLP format (OTLP receiver), using the OpenTelemetry SDK or
auto-instrumentation agents.
!!! note
To disable the tracing configuration, set up `enable_tracing = false` in
the [module configuration](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/eks-monitoring#input_enable_tracing)
## Instrumentation
Let's take a [sample application](https://github.com/aws-observability/aws-otel-community/tree/master/sample-apps/go-sample-app)
that is already instrumented with the OpenTelemetry SDK.
!!! note
To learn more about instrumenting with OpenTelemetry, please visit the
[OpenTelemetry documentation](https://opentelemetry.io/docs/instrumentation/)
for your programming language.
Cloning the repo
```console
git clone https://github.com/aws-observability/aws-otel-community.git
cd aws-otel-community/sample-apps/go-sample-app
```
Highlighting code sections
## Deploying on Amazon EKS
Using the sample application, we will build a container image, create and push
an image on Amazon ECR. We will use a Kubernetes manifest to deploy to an EKS
cluster.
!!! warning
The following steps require that you have an EKS cluster ready. To deploy
an EKS cluster, please visit [our example](https://aws-observability.github.io/terraform-aws-observability-accelerator/helpers/new-eks-cluster/).
### Building container image
=== "amd64 linux"
``` console
docker build -t go-sample-app .
```
=== "cross platform build"
``` bash
docker buildx build -t go-sample-app . --platform=linux/amd64
```
### Publishing on Amazon ECR
=== "using docker"
``` console
export ECR_REPOSITORY_URI=$(aws ecr create-repository --repository go-sample-app --query repository.repositoryUri --output text)
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
docker tag go-sample-app:latest "${ECR_REPOSITORY_URI}:latest"
docker push "${ECR_REPOSITORY_URI}:latest"
```
## Deploying on Amazon EKS
``` yaml title="eks.yaml" linenums="1"
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-sample-app
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: go-sample-app
template:
metadata:
labels:
app: go-sample-app
spec:
containers:
- name: go-sample-app
image: "${ECR_REPOSITORY_URI}:latest" # make sure to replace this variable
imagePullPolicy: Always
env:
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: adot-collector.adot-collector-kubeprometheus.svc.cluster.local:4317
resources:
limits:
cpu: 300m
memory: 300Mi
requests:
cpu: 100m
memory: 180Mi
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: go-sample-app
namespace: default
labels:
app: go-sample-app
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: go-sample-app
---
apiVersion: v1
kind: Service
metadata:
name: go-sample-app
namespace: default
spec:
type: ClusterIP
selector:
app: go-sample-app
ports:
- protocol: TCP
port: 8080
targetPort: 8080
```
### Deploying and testing
With the Kubernetes manifest ready, run:
```bash
kubectl apply -f eks.yaml
```
You should see the pods running with the command:
```console
kubectl get pods
NAME READY STATUS RESTARTS AGE
go-sample-app-67c48ff8c6-bdw74 1/1 Running 0 4s
go-sample-app-67c48ff8c6-t6k2j 1/1 Running 0 4s
```
To simulate some traffic you can forward the service port to your local host
and test a few queries
```console
kubectl port-forward deployment/go-sample-app 8080:8080
```
Test a few endpoints
```
curl http://localhost:8080/
curl http://localhost:8080/outgoing-http-call
curl http://localhost:8080/aws-sdk-call
curl http://localhost:8080/outgoing-sampleapp
```
## Visualizing traces
As this is a basic example, the service map doesn't have a lot of nodes,
but this shows you how to setup tracing in your application and deploying
it on Amazon EKS using the `eks-monitoring` module.
With Flux and Grafana Operator, the `eks-monitoring` module configures
an AWS X-Ray data source on your provided Grafana workspace. Open the
Grafana explorer view and select the X-Ray data source. If you type the query
below, and select `Trace List` for **Query Type**, you should see the list
of traces occured in the selected timeframe.
<img width="1721" alt="Screenshot 2023-07-20 at 21 42 30" src="https://github.com/aws-observability/terraform-aws-observability-accelerator/assets/10175027/bd992a77-05fb-47d2-8ed4-af05d96e951d">
You can add the service map to a dashbaord, for example a service focused
dashbaord. You can click on any of the traces to view a node map and the traces
details.
There is a button that can take you the CloudWatch console to view the same
data. If your logs are stored on CloudWatch Logs, this page can present
all the logs in the trace details page. The CloudWatch Log Group name should
be added to the trace as an attribute.
Read more about this in our [One Observability Workshop](https://catalog.workshops.aws/observability/en-US/use-cases/trace-to-logs-java-instrumentation/concepts)
![CloudWatch service map](https://user-images.githubusercontent.com/10175027/254973349-1028f428-c2ef-4bd2-8114-0d0961d7cdd8.png)
## Resoures
- [AWS Observability Best Practices](https://aws-observability.github.io/observability-best-practices/)
- [One Observability Workshop](https://catalog.workshops.aws/observability/en-US/)
- [AWS Distro for OpenTelemetry documentation](https://aws-otel.github.io/docs/introduction)
- [AWS X-Ray user guide](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html)
- [OpenTelemetry documentation](https://opentelemetry.io/docs/what-is-opentelemetry/)
mkdocs.yml
+3
View File
@@ -32,6 +32,7 @@ nav:
- Nginx: eks/nginx.md
- Istio: eks/istio.md
- Viewing logs: eks/logs.md
- Tracing: eks/tracing.md
- Teardown: eks/destroy.md
- Monitoring Managed Service for Prometheus Workspaces: workloads/managed-prometheus.md
- Supporting Examples:
@@ -47,6 +48,8 @@ markdown_extensions:
- codehilite
- footnotes
- pymdownx.critic
- pymdownx.tabbed:
alternate_style: true
- pymdownx.superfences:
custom_fences:
- name: mermaid
modules/eks-monitoring/README.md
+2 -2
View File
@@ -66,7 +66,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_adot_loglevel"></a> [adot\_loglevel](#input\_adot\_loglevel) | Verbosity level for ADOT collector logs | `string` | `"warn"` | no |
| <a name="input_adot_loglevel"></a> [adot\_loglevel](#input\_adot\_loglevel) | Verbosity level for ADOT collector logs. This accepts (detailed\|normal\|basic), see https://aws-otel.github.io/docs/components/misc-exporters for mor infos. | `string` | `"normal"` | no |
| <a name="input_custom_metrics_config"></a> [custom\_metrics\_config](#input\_custom\_metrics\_config) | Configuration object to enable custom metrics collection | <pre>map(object({<br> enableBasicAuth = bool<br> path = string<br> basicAuthUsername = string<br> basicAuthPassword = string<br> ports = string<br> droppedSeriesPrefixes = string<br> }))</pre> | `null` | no |
| <a name="input_eks_cluster_id"></a> [eks\_cluster\_id](#input\_eks\_cluster\_id) | EKS Cluster Id | `string` | n/a | yes |
| <a name="input_enable_alerting_rules"></a> [enable\_alerting\_rules](#input\_enable\_alerting\_rules) | Enables or disables Managed Prometheus alerting rules | `bool` | `true` | no |
@@ -84,7 +84,7 @@ See examples using this Terraform modules in the **Amazon EKS** section of [this
| <a name="input_enable_nginx"></a> [enable\_nginx](#input\_enable\_nginx) | Enable NGINX workloads monitoring, alerting and default dashboards | `bool` | `false` | no |
| <a name="input_enable_node_exporter"></a> [enable\_node\_exporter](#input\_enable\_node\_exporter) | Enables or disables Node exporter. Disabling this might affect some data in the dashboards | `bool` | `true` | no |
| <a name="input_enable_recording_rules"></a> [enable\_recording\_rules](#input\_enable\_recording\_rules) | Enables or disables Managed Prometheus recording rules | `bool` | `true` | no |
| <a name="input_enable_tracing"></a> [enable\_tracing](#input\_enable\_tracing) | (Experimental) Enables tracing with AWS X-Ray. This changes the deploy mode of the collector to daemon set. Requirement: adot add-on <= 0.58-build.0 | `bool` | `false` | no |
| <a name="input_enable_tracing"></a> [enable\_tracing](#input\_enable\_tracing) | Enables tracing with OTLP traces receiver to X-Ray | `bool` | `true` | no |
| <a name="input_flux_config"></a> [flux\_config](#input\_flux\_config) | FluxCD configuration | <pre>object({<br> create_namespace = bool<br> k8s_namespace = string<br> helm_chart_name = string<br> helm_chart_version = string<br> helm_release_name = string<br> helm_repo_url = string<br> helm_settings = map(string)<br> helm_values = map(any)<br> })</pre> | <pre>{<br> "create_namespace": true,<br> "helm_chart_name": "flux2",<br> "helm_chart_version": "2.7.0",<br> "helm_release_name": "observability-fluxcd-addon",<br> "helm_repo_url": "https://fluxcd-community.github.io/helm-charts",<br> "helm_settings": {},<br> "helm_values": {},<br> "k8s_namespace": "flux-system"<br>}</pre> | no |
| <a name="input_flux_gitrepository_branch"></a> [flux\_gitrepository\_branch](#input\_flux\_gitrepository\_branch) | Flux GitRepository Branch | `string` | `"main"` | no |
| <a name="input_flux_gitrepository_name"></a> [flux\_gitrepository\_name](#input\_flux\_gitrepository\_name) | Flux GitRepository name | `string` | `"aws-observability-accelerator"` | no |
modules/eks-monitoring/otel-config/templates/opentelemetrycollector.yaml
+1 -86
View File
@@ -3,12 +3,7 @@ kind: OpenTelemetryCollector
metadata:
name: adot
spec:
{{ if .Values.enableTracing }}
mode: daemonset
hostNetwork: true
{{ else }}
mode: deployment
{{ end }}
serviceAccount: adot-collector-kubeprometheus
env:
- name: "K8S_NODE_NAME"
@@ -56,11 +51,6 @@ spec:
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$${1}/proxy/metrics
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_node_name]
{{ end }}
- job_name: 'kubelet'
scheme: https
tls_config:
@@ -78,11 +68,6 @@ spec:
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_node_name]
{{ end }}
- job_name: 'kube-admin'
scheme: https
tls_config:
@@ -195,11 +180,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -302,11 +282,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -408,11 +383,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -530,11 +500,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -652,11 +617,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -773,11 +733,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -882,11 +837,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -993,11 +943,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -1104,11 +1049,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -1215,11 +1155,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -1326,11 +1261,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -1431,11 +1361,6 @@ spec:
regex: "0"
replacement: $$1
action: keep
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
@@ -1459,11 +1384,6 @@ spec:
- action: replace
source_labels: [__meta_kubernetes_endpoint_node_name]
target_label: nodename
{{ if .Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_endpoint_node_name]
{{ end }}
{{ if .Values.enableCustomMetrics }}
{{- range $k, $v := fromYaml .Values.customMetrics }}
- job_name: "{{ $k }}"
@@ -1494,11 +1414,6 @@ spec:
- action: replace
source_labels: [__meta_kubernetes_pod_controller_kind]
target_label: pod_controller_kind
{{ if $.Values.enableTracing }}
- action: keep
regex: $K8S_NODE_NAME
source_labels: [__meta_kubernetes_pod_node_name]
{{ end }}
metric_relabel_configs:
- source_labels: [ __name__ ]
regex: '{{ $v.droppedSeriesPrefixes }}'
@@ -1644,7 +1559,7 @@ spec:
resource_to_telemetry_conversion:
enabled: true
logging:
loglevel: {{ .Values.adotLoglevel }}
verbosity: {{ .Values.adotLoglevel }}
extensions:
sigv4auth:
region: {{ .Values.region }}
modules/eks-monitoring/variables.tf
+4 -4
View File
@@ -34,9 +34,9 @@ variable "irsa_iam_permissions_boundary" {
}
variable "adot_loglevel" {
description = "Verbosity level for ADOT collector logs"
description = "Verbosity level for ADOT collector logs. This accepts (detailed|normal|basic), see https://aws-otel.github.io/docs/components/misc-exporters for mor infos."
type = string
default = "warn"
default = "normal"
}
variable "managed_prometheus_workspace_endpoint" {
@@ -202,9 +202,9 @@ variable "prometheus_config" {
}
variable "enable_tracing" {
description = "(Experimental) Enables tracing with AWS X-Ray. This changes the deploy mode of the collector to daemon set. Requirement: adot add-on <= 0.58-build.0"
description = "Enables tracing with OTLP traces receiver to X-Ray"
type = bool
default = false
default = true
}
variable "tracing_config" {