Follow a request
When a request fails somewhere deep in your services, the error you see is at the top: the front end returned a 502. Follow a request walks that one request down through every service it touched, and names the span where it dropped - the first failing span that its own children do not explain - with the kind of failure and the attribute or message that shows it.
It needs traces from your services in the observability chart.
Open it
Section titled “Open it”- From a log line on Logs: open the line and press Follow this
request. Seylo reads the trace ID in the line - a
traceparent, a trace ID field, or a Datadog decimal ID. - From a trace on Traces: Follow this request: where it dropped and why.
- Directly, at
/observability/request: paste a Trace id (32 hex digits, atraceparent, or a Datadog ID) and press Follow; or enter a service under Or a service’s failing requests and press Recent failures to pick from its failing requests in the last hour. Seylo picks up to 5, different failures first.
What it shows
Section titled “What it shows”Where it dropped, at the top:
- the failure, such as
connection refusedordatabase connection pool exhausted, on the service and operation where it happened; - the evidence: each attribute or message on the span that shows it, and the rule that matched it;
- which callers above it failed only because this call failed;
- how many other failing spans are not on this path.
When nothing on the span names a cause, it says so, with the span’s status as recorded, and classes it unclassified. Seylo never guesses a cause.
The slowest hop - the span that spent the most time itself, not waiting on what it called.
The path - every span as a row, the failing path marked and the drop point in red. Click a row for its attributes. Asynchronous hops are followed: a consumer under the producer that sent its message, by span links or by the message’s ID. A hop the read cannot see is shown as a gap, without a name.
If the request did not fail at its root - a failure inside it was handled or retried - the page says so.
The kinds of failure
Section titled “The kinds of failure”| Class | Class |
|---|---|
| timeout / deadline exceeded | cancelled by the caller |
| connection refused | connection reset / closed mid-request |
| DNS: the name did not resolve | TLS handshake or certificate |
| upstream unavailable | upstream answered with an error (HTTP 5xx or a gRPC server error) |
| the service’s own error | request rejected (4xx) |
| rate limited | circuit breaker open |
| database connection pool exhausted | database refused: too many connections |
| database deadlock | database lock wait timeout |
| database statement timeout | database error |
| message dead-lettered | retries exhausted |
Database failures are read from the database’s own error code (SQLSTATE or the vendor’s code) first, then its message.
Limits
Section titled “Limits”- Seylo reads up to 1,000 spans of a trace. A larger trace says that the rest were not analysed.
- A 64-bit trace ID from a log line is matched by the last 16 digits of the trace ID, and the page says so. Seylo never guesses the missing half.
- You see only spans from namespaces you may see, through the traces store’s namespace filter, like everywhere else.
Nightshift follows requests too
Section titled “Nightshift follows requests too”During an investigation, Nightshift can follow a request with its
trace_path query - from a trace ID it found in the evidence, or by sampling
the alerting service’s recent failing requests - and gets the same analysis as
numbered evidence. At most 3 per run, each counted as one of its own queries.
See queries Nightshift writes itself.