Skip to content

APM coverage and autonomous tracing

A service that sends no traces is a gap in every picture Seylo draws: APM, the service map, follow a request and Nightshift’s investigations. Coverage finds those gaps, and for a workload that serves traffic and sends no spans, proposes the one change that closes it: a single annotation for the OpenTelemetry Operator you installed.

Nothing is written to your cluster until a person approves it, unless a grant you signed says otherwise.

Observability → Traces & APM, tab Coverage. Pick the cluster and a namespace. For each Deployment and StatefulSet in it, Seylo shows:

Column What it says
Workload The workload, and its kind
Coverage Its status (below), and a proposal’s status once one is filed
Language The language Seylo read, and how sure it is. Hover for the evidence
Spans How many spans arrived from it in the last 60 minutes, counted through the traces store’s namespace filter, so you see only your own namespaces
Why Why it has that status, and what to do
Status Meaning
Sends traces Spans arrived from it in the last hour
Can instrument It serves traffic, sends no spans, its language is clear, and the operator has an Instrumentation to name. Instrument is offered
Which language? It serves traffic and sends no spans, but the evidence for its language is missing, conflicts, or is only a port. Pick the language to propose instrumenting it
Needs the operator The OpenTelemetry Operator, or an Instrumentation for it to use, is not there yet
Annotated, silent It already carries the operator’s annotation and still sends nothing. The reason lists the usual causes
Not proposed Seylo will not instrument it automatically, with the reason: Go, a web server, or Python or .NET on an Alpine image
Serves no traffic No Service selects its pods, so no request reaches it to follow: a worker or a batch job
Unknown The traces store could not be read

A workload serves traffic when a Service selects its pods, or an Ingress reaches that Service. The language is read from the first container, the one the operator instruments, using its image, command, environment variable names (never values), labels and ports.

The same answer is available to your coding agent through the apm_coverage MCP tool, read-only. Other places lead here too: an APM page with no traces links to Coverage; when Nightshift investigates a Deployment that sends no spans, it records the gap and offers Instrument and the workload’s name, a link to this list; and Get request metrics in the assistants opens coverage at the workload.

Above the list, Seylo says whether the operator is installed and which Instrumentation a proposal will name:

  • OpenTelemetry Operator installed; proposals name the Instrumentation …
  • The OpenTelemetry Operator is not installed - Seylo holds no credential that could install it. A person installs it, as in send traces: no code.
  • No Instrumentation to name - create the shared seylo Instrumentation from the same guide (one per cluster).
  • Seylo cannot tell whether the operator is installed - the read failed; nothing is proposed until it can.

A proposal names the Instrumentation called seylo first, else the only one in the workload’s namespace, else the only one in the cluster. When there are several and none of those rules picks one, Seylo asks you rather than choosing where a workload’s spans go.

  1. On a Can instrument row, press Instrument (releaser). The row says first that this restarts the workload’s pods, and that nothing changes until it is approved.
  2. Seylo files one change as a Nightshift fix: the annotation instrumentation.opentelemetry.io/inject-<language>: "<namespace>/<instrumentation>" on the workload’s pod template. It is checked again, dry-run on the cluster, and shown with its verification and its undo.
  3. Approve it on the fix card, like any change. It then appears on Nightshift → Fixes with From: APM coverage.

The guard accepts exactly one annotation key, on a Deployment or StatefulSet pod template, for a language with evidence, naming an Instrumentation that exists. It never annotates a namespace, and never instruments Go.

After the change, the fix waits for proof: the rollout completes, every pod is Ready, and spans from this workload arrive after the change - within 15 minutes. Then it is verified.

If no span arrives in that time, or a pod restarts after the change, the annotation is removed again and read back: an agent that sends nothing costs every pod memory and start-up time and proves nothing. While it waits, the reason names the usual causes - no request reached it, a NetworkPolicy keeps it from the collector (the observability chart’s networkPolicy.otelIngressFrom), or the Instrumentation’s exporter points elsewhere - so whoever tries again knows what to check first.

Language What Seylo does
Java, Node.js, Python, .NET Proposes the annotation
Python or .NET on an Alpine (musl) image Not proposed: it needs a second annotation, which this one-key change does not make. Add both yourself from send traces
Go Never: the operator instruments Go with a privileged eBPF sidecar. Use the SDK
A web server (such as nginx) Not proposed
Unclear Which language? - pick it, and Seylo proposes

A change made only in the cluster would be reverted at the next sync, so an owned workload is not instrumented like the others. The row shows the annotation as a YAML fragment for its Git source, and:

  • Propose as a pull request - for a workload Argo CD or Flux deploys from GitHub, when an admin has turned on Let Seylo open pull requests. Approving opens the pull request; it is verified by spans once merged and synced. See pull requests for workloads Git owns.
  • Instrument anyway - tick the box acknowledging the owner, and Seylo applies it in the cluster, knowing the owner may revert it.

When the pull request cannot be proposed yet, the row says what is missing.

By default, nothing is instrumented without a person. An admin can request, and an approver sign, a grant that covers it, on Operate → Automation: a grant with the Nightshift action Instrument a workload for tracing (one annotation; restarts its pods).

  • The grant must name its namespace, and instruments only there.
  • It names the languages it covers - Java, Node.js, Python, .NET. Go cannot be granted.
  • Workloads instrumented per day: 1 to 20. Each one restarts a workload’s pods.
  • The grant’s window, rate limit, expiry and Pause all automation apply, as for every unattended change.

Under such a grant, Nightshift reads coverage in that namespace only, and instruments at most one workload per namespace each time it looks, through the same checks, verification and undo as an approval. It never instruments a workload Argo CD, Flux or Helm owns. Without a grant, it reads coverage only when you open it.