Send traces and APM data
Your services send OpenTelemetry traces (OTLP) to the collector the
seylo-observability chart runs, seylo-otel. From every span it receives:
- Traces: the spans are stored in VictoriaTraces in your cluster, and Observability → Traces & APM shows them: APM lists your services, and each one opens its traces.
- APM: calls, errors and latency are counted per service into VictoriaMetrics, and the APM view shows them.
APM is counted from all spans, not estimated from a sample.
The same steps, with your install namespace filled in, are in Seylo under Admin → Integrations → Observability stores → Send traces and APM data. The Traces & APM page links there when no service sends traces yet.
Let Seylo find the gaps. Once the operator below is installed, the Coverage tab of Traces & APM lists which workloads serve traffic and send no spans, and proposes the one annotation that instruments each, for your approval. See APM coverage.
Three things true of every option
Section titled “Three things true of every option”-
The collector receives OTLP only.
- gRPC:
seylo-otel.seylo-observability.svc:4317 - HTTP:
http://seylo-otel.seylo-observability.svc:4318
- gRPC:
-
It takes traces, not metrics or logs. Every option below switches the SDK’s OTLP metrics and logs export off (
OTEL_METRICS_EXPORTER=none,OTEL_LOGS_EXPORTER=none). Seylo already has both:- container logs from stdout and stderr, through vlagent;
- metrics from scraping (see configure).
Sent to the collector anyway, they are refused, and the refusals fill your application’s logs.
-
Each span needs its namespace. A person with access to only some namespaces sees a service only if its spans carry the
k8s.namespace.nameresource attribute.- Sent straight from the pod: the collector adds it from the address the
span came from. From chart 0.2.19 it also deletes any
k8s.*attribute the application set, so no service can choose its namespace. - Relayed through another collector or an eBPF agent: the span must carry it already. From chart 0.2.19 a relay sends to the forwarder ports, 4327 and 4328, which exist only for the namespaces you list.
- Sent straight from the pod: the collector adds it from the address the
span came from. From chart 0.2.19 it also deletes any
Choose a way in
Section titled “Choose a way in”The OpenTelemetry Operator adds the language agent to a pod when it starts, from one annotation. You don’t rebuild the image or change code. Supported: Java, Node.js, Python and .NET. Go is covered further down.
-
Install the operator once per cluster. The two
--setlines let it make its own webhook certificate; drop them if cert-manager is already installed. Chart 0.124.1 is operator 0.160.0 (released 2026-09-28).Terminal window helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-chartshelm install opentelemetry-operator open-telemetry/opentelemetry-operator \--version 0.124.1 \--namespace opentelemetry-operator-system --create-namespace \--set admissionWebhooks.certManager.enabled=false \--set admissionWebhooks.autoGenerateCert.enabled=true -
Say where spans go: one Instrumentation, beside the stores. Every namespace uses it. Save the file as
seylo-instrumentation.yaml, then runkubectl apply -f seylo-instrumentation.yamlonce the operator is ready.apiVersion: opentelemetry.io/v1alpha1kind: Instrumentationmetadata:name: seylonamespace: seylo-observabilityspec:exporter:endpoint: http://seylo-otel.seylo-observability.svc:4318env:- name: OTEL_EXPORTER_OTLP_PROTOCOLvalue: http/protobuf- name: OTEL_METRICS_EXPORTERvalue: "none"- name: OTEL_LOGS_EXPORTERvalue: "none"propagators: [tracecontext, baggage]sampler:type: parentbased_always_onSampling is off on purpose: APM counts the spans it receives, so sampling would make every request rate a fraction of the real one.
-
Annotate the workload’s pod template. For the fictional Java service
checkout-apiinshop:spec:template:metadata:annotations:instrumentation.opentelemetry.io/inject-java: "seylo-observability/seylo"For another language, replace
inject-javawithinject-nodejs,inject-pythonorinject-dotnet. Alpine (musl) images need one more annotation:- Python:
instrumentation.opentelemetry.io/otel-python-platform: "musl" - .NET:
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"
If Argo CD, Flux or Helm manages the workload, make this change in Git or in the chart values. A change made only in the cluster is reverted at the next sync. For a Deployment nobody reconciles, run (bash or zsh):
Terminal window kubectl patch deployment checkout-api -n shop --type merge \-p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"seylo-observability/seylo"}}}}}' - Python:
-
Let the Deployment roll. Changing the pod template rolls the Deployment, and the new pods start with the agent. There is nothing else to restart.
kubectl describe pod -n shopshows an added init container named after the language.
Add OpenTelemetry to the application yourself:
| Language | How |
|---|---|
| Java | The OpenTelemetry Java agent, -javaagent:/path/opentelemetry-javaagent.jar (JAVA_TOOL_OPTIONS works). No code change |
| Node.js | npm install @opentelemetry/auto-instrumentations-node, start with node --require @opentelemetry/auto-instrumentations-node/register app.js |
| Python | pip install opentelemetry-distro opentelemetry-exporter-otlp, run opentelemetry-bootstrap -a install, start with opentelemetry-instrument python app.py |
| .NET | OpenTelemetry .NET automatic instrumentation, or OpenTelemetry.Extensions.Hosting with AddOtlpExporter() |
| Go | The go.opentelemetry.io/otel SDK with the otlptracehttp exporter and the otelhttp / otelgrpc middleware |
Then point it at the collector with environment variables on the container:
env: - name: OTEL_SERVICE_NAME value: checkout-api - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://seylo-otel.seylo-observability.svc:4318 - name: OTEL_EXPORTER_OTLP_PROTOCOL value: http/protobuf - name: OTEL_METRICS_EXPORTER value: "none" - name: OTEL_LOGS_EXPORTER value: "none"Nothing to set for the namespace: the collector works out the namespace, pod
and Deployment from the pod that sends. From chart 0.2.19 it ignores any
k8s.* attribute the application sets, so setting k8s.namespace.name
yourself changes nothing. For gRPC, use port 4317 with
OTEL_EXPORTER_OTLP_PROTOCOL=grpc.
These are not received by the collector:
| Not received | Instead |
|---|---|
Datadog tracers (dd-trace) and the Datadog Agent protocol |
Replace dd-trace with auto-instrumentation or an SDK for the services you want here |
| New Relic, Dynatrace, AppDynamics and Elastic APM agents | Use their OpenTelemetry mode if they have one, or auto-instrumentation or an SDK |
| Jaeger Thrift and the jaeger-agent UDP ports | Jaeger’s own client libraries are retired in favour of OpenTelemetry. Switch to auto-instrumentation or an SDK |
| Zipkin JSON | An OpenTelemetry SDK |
If you already run an OpenTelemetry Collector that exports to a vendor, keep it and add Seylo as a second destination, in a traces pipeline of its own beside yours. Your pipeline stays as it is, and both receive every span while you compare:
processors: resource/seylo: # an application does not choose its own namespace attributes: - { action: delete, pattern: '^k8s\.' } k8sattributes/seylo: # names the pod from the sender's address; needs pods/namespaces/replicasets read pod_association: - sources: - from: connection batch/seylo: {}exporters: otlp_http/seylo: # collectors before the component rename call it otlphttp/seylo endpoint: http://seylo-otel.seylo-observability.svc:4328service: pipelines: traces/seylo: # beside your traces pipeline, which stays as it is receivers: [otlp] # the receiver your applications send to processors: [resource/seylo, k8sattributes/seylo, batch/seylo] exporters: [otlp_http/seylo]Spans now reach Seylo from your collector’s pod, so they go to the
forwarder port, 4328 (HTTP) or 4327 (gRPC), which keeps the
k8s.namespace.name your collector sets. That port is closed until you list
your collector’s namespace, here the fictional otel-gateway, on the
seylo-observability chart (0.2.19 or later):
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \ --namespace seylo-observability --reset-then-reuse-values \ --set 'traces.trustedForwarders.namespaces={otel-gateway}'The list replaces the previous one: name every forwarder’s namespace in one
command, comma-separated inside the braces. Read
what listing a namespace means
before you do. On chart 0.2.18 or earlier the forwarder ports do not exist:
send to port 4318 instead, and skip the helm upgrade.
An eBPF agent on each node watches HTTP and gRPC traffic in the kernel and creates a server span per request. It works for any language, with no annotation, no restart and no code. What you give up:
- Spans stop at the process edge: no database calls and no internal functions.
- The agent needs root-level access to every node.
Grafana Beyla is packaged as a Helm chart. This sends traces only to the collector. Chart 1.16.14 is Beyla 3.38.0, released 2026-10-05.
helm repo add grafana https://grafana.github.io/helm-chartshelm install beyla grafana/beyla \ --version 1.16.14 \ --namespace beyla --create-namespace \ --set-string env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://seylo-otel.seylo-observability.svc:4328/v1/traces \ --set-string env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobufBeyla names each span’s pod itself, so it sends to the forwarder port,
4328, which opens only to namespaces you list on the seylo-observability
chart (0.2.19 or later):
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \ --namespace seylo-observability --reset-then-reuse-values \ --set 'traces.trustedForwarders.namespaces={beyla}'Any pod in a listed namespace can label spans with any namespace; see
what listing a namespace means.
On chart 0.2.18 or earlier, use port 4318 in the endpoint and skip the
helm upgrade.
OpenTelemetry eBPF Instrumentation (OBI)
is the same engine, donated to OpenTelemetry. It is at v0.14.0 (2026-10-02), and
its own project warns of breaking changes between minor releases. Follow its
Kubernetes setup, set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT as above, keep it
off the host network, and add its namespace to
traces.trustedForwarders.namespaces the same way.
Check it works
Section titled “Check it works”-
Send a few requests to the service.
-
Observability → Traces & APM lists the service within about a minute. APM counts server and consumer spans, so a worker that only makes outgoing calls appears among the traces, not in the APM service list.
-
From a coding agent connected to Seylo,
traces_serviceslists the same services, andtraces_searchfinds their requests. -
Nothing after a few minutes? The collector says what it refused:
Terminal window kubectl logs -n seylo-observability deploy/seylo-otel --since=10mIf
networkPolicy.otelIngressFromis set in the chart, the service’s namespace must be in it. A collector or eBPF agent sending to the forwarder port must have its namespace intraces.trustedForwarders.namespaces.
Seylo reads spans through the agent when a page asks, and keeps none of them. They stay in VictoriaTraces in your cluster for the chart’s retention.