Skip to content

Follow a request

When a request fails somewhere deep in your services, the error you see is at the top: the front end returned a 502. Follow a request walks that one request down through every service it touched, and names the span where it dropped - the first failing span that its own children do not explain - with the kind of failure and the attribute or message that shows it.

It needs traces from your services in the observability chart.

  • From a log line on Logs: open the line and press Follow this request. Seylo reads the trace ID in the line - a traceparent, a trace ID field, or a Datadog decimal ID.
  • From a trace on Traces: Follow this request: where it dropped and why.
  • Directly, at /observability/request: paste a Trace id (32 hex digits, a traceparent, or a Datadog ID) and press Follow; or enter a service under Or a service’s failing requests and press Recent failures to pick from its failing requests in the last hour. Seylo picks up to 5, different failures first.

Where it dropped, at the top:

  • the failure, such as connection refused or database connection pool exhausted, on the service and operation where it happened;
  • the evidence: each attribute or message on the span that shows it, and the rule that matched it;
  • which callers above it failed only because this call failed;
  • how many other failing spans are not on this path.

When nothing on the span names a cause, it says so, with the span’s status as recorded, and classes it unclassified. Seylo never guesses a cause.

The slowest hop - the span that spent the most time itself, not waiting on what it called.

The path - every span as a row, the failing path marked and the drop point in red. Click a row for its attributes. Asynchronous hops are followed: a consumer under the producer that sent its message, by span links or by the message’s ID. A hop the read cannot see is shown as a gap, without a name.

If the request did not fail at its root - a failure inside it was handled or retried - the page says so.

Class Class
timeout / deadline exceeded cancelled by the caller
connection refused connection reset / closed mid-request
DNS: the name did not resolve TLS handshake or certificate
upstream unavailable upstream answered with an error (HTTP 5xx or a gRPC server error)
the service’s own error request rejected (4xx)
rate limited circuit breaker open
database connection pool exhausted database refused: too many connections
database deadlock database lock wait timeout
database statement timeout database error
message dead-lettered retries exhausted

Database failures are read from the database’s own error code (SQLSTATE or the vendor’s code) first, then its message.

  • Seylo reads up to 1,000 spans of a trace. A larger trace says that the rest were not analysed.
  • A 64-bit trace ID from a log line is matched by the last 16 digits of the trace ID, and the page says so. Seylo never guesses the missing half.
  • You see only spans from namespaces you may see, through the traces store’s namespace filter, like everywhere else.

During an investigation, Nightshift can follow a request with its trace_path query - from a trace ID it found in the evidence, or by sampling the alerting service’s recent failing requests - and gets the same analysis as numbered evidence. At most 3 per run, each counted as one of its own queries. See queries Nightshift writes itself.