Skip to content

Send traces and APM data

Your services send OpenTelemetry traces (OTLP) to the collector the seylo-observability chart runs, seylo-otel. From every span it receives:

  • Traces: the spans are stored in VictoriaTraces in your cluster, and Observability → Traces & APM shows them: APM lists your services, and each one opens its traces.
  • APM: calls, errors and latency are counted per service into VictoriaMetrics, and the APM view shows them.

APM is counted from all spans, not estimated from a sample.

The same steps, with your install namespace filled in, are in Seylo under Admin → Integrations → Observability stores → Send traces and APM data. The Traces & APM page links there when no service sends traces yet.

Let Seylo find the gaps. Once the operator below is installed, the Coverage tab of Traces & APM lists which workloads serve traffic and send no spans, and proposes the one annotation that instruments each, for your approval. See APM coverage.

  • The collector receives OTLP only.

    • gRPC: seylo-otel.seylo-observability.svc:4317
    • HTTP: http://seylo-otel.seylo-observability.svc:4318
  • It takes traces, not metrics or logs. Every option below switches the SDK’s OTLP metrics and logs export off (OTEL_METRICS_EXPORTER=none, OTEL_LOGS_EXPORTER=none). Seylo already has both:

    • container logs from stdout and stderr, through vlagent;
    • metrics from scraping (see configure).

    Sent to the collector anyway, they are refused, and the refusals fill your application’s logs.

  • Each span needs its namespace. A person with access to only some namespaces sees a service only if its spans carry the k8s.namespace.name resource attribute.

    • Sent straight from the pod: the collector adds it from the address the span came from. From chart 0.2.19 it also deletes any k8s.* attribute the application set, so no service can choose its namespace.
    • Relayed through another collector or an eBPF agent: the span must carry it already. From chart 0.2.19 a relay sends to the forwarder ports, 4327 and 4328, which exist only for the namespaces you list.

    See which namespace a span belongs to.

The OpenTelemetry Operator adds the language agent to a pod when it starts, from one annotation. You don’t rebuild the image or change code. Supported: Java, Node.js, Python and .NET. Go is covered further down.

  1. Install the operator once per cluster. The two --set lines let it make its own webhook certificate; drop them if cert-manager is already installed. Chart 0.124.1 is operator 0.160.0 (released 2026-09-28).

    Terminal window
    helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
    helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
    --version 0.124.1 \
    --namespace opentelemetry-operator-system --create-namespace \
    --set admissionWebhooks.certManager.enabled=false \
    --set admissionWebhooks.autoGenerateCert.enabled=true
  2. Say where spans go: one Instrumentation, beside the stores. Every namespace uses it. Save the file as seylo-instrumentation.yaml, then run kubectl apply -f seylo-instrumentation.yaml once the operator is ready.

    apiVersion: opentelemetry.io/v1alpha1
    kind: Instrumentation
    metadata:
    name: seylo
    namespace: seylo-observability
    spec:
    exporter:
    endpoint: http://seylo-otel.seylo-observability.svc:4318
    env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: http/protobuf
    - name: OTEL_METRICS_EXPORTER
    value: "none"
    - name: OTEL_LOGS_EXPORTER
    value: "none"
    propagators: [tracecontext, baggage]
    sampler:
    type: parentbased_always_on

    Sampling is off on purpose: APM counts the spans it receives, so sampling would make every request rate a fraction of the real one.

  3. Annotate the workload’s pod template. For the fictional Java service checkout-api in shop:

    spec:
    template:
    metadata:
    annotations:
    instrumentation.opentelemetry.io/inject-java: "seylo-observability/seylo"

    For another language, replace inject-java with inject-nodejs, inject-python or inject-dotnet. Alpine (musl) images need one more annotation:

    • Python: instrumentation.opentelemetry.io/otel-python-platform: "musl"
    • .NET: instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"

    If Argo CD, Flux or Helm manages the workload, make this change in Git or in the chart values. A change made only in the cluster is reverted at the next sync. For a Deployment nobody reconciles, run (bash or zsh):

    Terminal window
    kubectl patch deployment checkout-api -n shop --type merge \
    -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"seylo-observability/seylo"}}}}}'
  4. Let the Deployment roll. Changing the pod template rolls the Deployment, and the new pods start with the agent. There is nothing else to restart. kubectl describe pod -n shop shows an added init container named after the language.

  1. Send a few requests to the service.

  2. Observability → Traces & APM lists the service within about a minute. APM counts server and consumer spans, so a worker that only makes outgoing calls appears among the traces, not in the APM service list.

  3. From a coding agent connected to Seylo, traces_services lists the same services, and traces_search finds their requests.

  4. Nothing after a few minutes? The collector says what it refused:

    Terminal window
    kubectl logs -n seylo-observability deploy/seylo-otel --since=10m

    If networkPolicy.otelIngressFrom is set in the chart, the service’s namespace must be in it. A collector or eBPF agent sending to the forwarder port must have its namespace in traces.trustedForwarders.namespaces.

Seylo reads spans through the agent when a page asks, and keeps none of them. They stay in VictoriaTraces in your cluster for the chart’s retention.