OpenTelemetry를 Kubernetes 클러스터에 설정하여 애플리케이션의 트레이스를 수집하고, Grafana Tempo로 전송해 시각화합니다.

Cert-Manager 설치 (선택 사항)

Kubernetes OpenTelemetry Operator 설치

kubectl create ns opentelemetry-operator

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

#helm show values open-telemetry/opentelemetry-operator > opentelemetry-operator_values.yaml

helm upgrade --debug --install --reuse-values \
  opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace opentelemetry-operator \
  --create-namespace \
  --values "opentelemetry-operator_values.yaml" \
  --version 0.29.2 \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.certManager.autoGenerateCert=true

Kubernetes Tempo 설치

kubectl create ns tempo
#helm show values grafana/tempo-distributed > tempo_values.yaml

helm upgrade --debug --install \
  tempo grafana/tempo-distributed \
  --create-namespace \
  --namespace tempo \
  --values "tempo_values.yaml" \
  --version 1.4.2

Grafana 데이터 소스에 Tempo 추가

kubectl get svc tempo-distributor-discovery -n tempo

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                                AGE
tempo-distributor-discovery   ClusterIP   None         <none>        3100/TCP,4318/TCP,4317/TCP,55680/TCP   9m9s

Grafana URL 예시: http://tempo-query-frontend-discovery.tempo:3100

OpenTelemetry Collector 설치

#helm show values open-telemetry/opentelemetry-collector > opentelemetry-collector_values.yaml

helm upgrade --debug --install --reuse-values \
  opentelemetry-collector open-telemetry/opentelemetry-collector \
  --values "opentelemetry-collector_values.yaml" \
  --namespace opentelemetry-operator

자동 계측 설정

kubectl create ns nlp
kubectl delete -f opentelemetry-instrumentation.yaml -n nlp
kubectl apply -f opentelemetry-instrumentation.yaml -n nlp

kubectl get instrumentations.opentelemetry.io -n nlp

마스터 노드가 워커 노드의 9443/tcp 포트에 접근할 수 있도록 방화벽 규칙을 추가하거나 기존 규칙을 수정하여 9443/tcp 접근을 허용해야 합니다.

테스트 애플리케이션 설정

kubectl apply -f test2.yaml -n nlp

kubectl logs -l app.kubernetes.io/name=opentelemetry-operator \
  --container manager -n opentelemetry-operator --follow

각 네임스페이스에 계측 설치 필요:

PROJECTS=(devops devops-dev common common-dev)
for item in "${PROJECTS[@]}"; do
  echo "===================== ${item}"
  kubectl delete -f opentelemetry-instrumentation.yaml -n ${item}
  kubectl apply -f opentelemetry-instrumentation.yaml -n ${item}
done

Ingress-Nginx 생성

kubectl -n opentelemetry-operator apply -f collector-ingress.yaml

curl을 이용한 테스트

curl -i http://collector.opentelemetry-operator.topzone.io/v1/traces -X POST -H "Content-Type: application/json" -d @span.json

자동 계측 (Node.js)

1. 패키지 추가

test-application/package.json

"dependencies": {
  "@opentelemetry/api": "^1.7.0",
  "@opentelemetry/auto-instrumentations-node": "^0.40.0"
}

2. 사이드카 추가

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-application
spec:
  selector:
    matchLabels:
      app: test-application
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-nodejs: "true"

수동 계측 (Python)

from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

resource = Resource(attributes={
    SERVICE_NAME: "tz-devops-admin"
})

traceProvider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector.opentelemetry-operator.topzone.io/v1/traces"))
traceProvider.add_span_processor(processor)
trace.set_tracer_provider(traceProvider)

def do_GET(self, httpd):
    tracer = trace.get_tracer("do_GET")
    with tracer.start_as_current_span("ri_cal") as span:
        span.set_attribute("printed_string", "done")
        with tracer.start_as_current_span("ri_usage") as span:
            span.set_attribute("printed_string", "done")

참고: https://github.com/doohee323/tz-eks-main/tree/eks-main-t/tz-local/resource/opentelemetry

원문)
https://www.linkedin.com/pulse/opentelemetry-k8s-dewey-hong-jdnzc/?trackingId=UGXm85DYTRaCjn%2F0wugyVw%3D%3D