Health signals
Some failures never raise an alert on the service that causes them. A
database that has run out of connections keeps running; a consumer that is
stuck stays Ready while its queue fills up. Their callers alert instead.
Health signals close that gap. Seylo reads what the exporters you already run say about pools, connections, lag, backlogs, retries and breakers, judges each one against a stated line, and puts the result on the service map node it is about. A database with no alert of its own can then be named as the root of an incident.
Nothing needs configuring in Seylo. If an exporter’s metrics are in the observability chart’s store, Seylo recognises them by name.
Where you see them
Section titled “Where you see them”- Service map - click a node: Health signals lists each signal with its state, its numbers, its source and how it was placed on that node. A signal Seylo cannot read says not collected and what to install.
- Incidents - the root-cause walk reads the signals across the incident’s namespaces. A failing signal makes its node unhealthy, and each signal line in the incident’s chain carries a badge and its source.
- Nightshift - the
node_healthquery gives an investigation the signals of a workload and of what it depends on, failing first, as numbered evidence. At most 2 per run.
How a signal is judged
Section titled “How a signal is judged”Each signal is read over the last 15 minutes, as the person (or the investigation) asking, so the store’s namespace filter applies as it does everywhere. It ends in one of four states:
| State | Meaning |
|---|---|
| failing | Requests or messages are failing, or not being processed, now. The node is unhealthy for an incident’s walk, with the signal cited |
| degraded | Strain without failure. Cited, never judged |
| healthy | Collected, and inside every line. A store whose exporter says it answers, with nothing failing, is healthy for the walk |
| unknown | Not collected, or not judgeable. Never treated as healthy |
The lines are fixed and stated:
| Signal | Failing | Degraded |
|---|---|---|
| Pool or connections saturation | At 90% or more for 5 minutes | At 90% now |
| Backlog or consumer lag | Rose by 50 or more messages over the window and is still within 10% of its peak | Rose, and is falling |
| Dead letters | 10 or more new in the window | 1 to 9 |
| Oldest message (SQS) | 5 or more minutes older within the window, at its peak | - |
| Deadlocks | 5 or more in the window | 1 to 4 |
| Replication lag | 5 minutes and rising | 30 seconds |
| Long transaction | - | 5 minutes |
| Slow queries (MySQL) | - | The later half of the window has twice the earlier, and 10 or more |
| Retry storm | - | 20% of 20 or more calls retried |
| Retries exhausted, breaker refusals | 5 or more in the window | 1 to 4 |
| Circuit breaker | Open at the end of the window | Opened during the window |
| Matching log lines | 3 or more | 1 or 2 |
Lag and backlog are judged by how they rise, never by their size: a queue of a million messages that is draining is not failing.
What Seylo recognises
Section titled “What Seylo recognises”| Area | Exporter, and what Seylo reads |
|---|---|
| Kafka | kafka_exporter or Strimzi’s Kafka Exporter (consumer group lag, brokers); the OpenTelemetry Kafka metrics receiver; Burrow; KEDA’s Kafka scaler |
| RabbitMQ | The built-in rabbitmq_prometheus plugin, or rabbitmq_exporter: whether it answers, ready and unacknowledged messages per queue, dead-letter queues by name |
| NATS JetStream | prometheus-nats-exporter with JetStream metrics: pending messages per consumer |
| Azure Service Bus, Amazon SQS | An Azure Monitor or CloudWatch exporter you run with your own identity: active and dead-lettered messages, the oldest message’s age |
| PostgreSQL | postgres_exporter: whether it answers, connections against max_connections, deadlocks, long transactions, replication lag |
| MySQL | mysqld_exporter: whether it answers, connections against max_connections, slow queries, replication lag |
| MongoDB | mongodb_exporter: whether it answers, connections in use |
| Redis | redis_exporter: whether it answers, clients against maxclients, memory against maxmemory |
| Connection pools | HikariCP through Micrometer; OpenTelemetry database-client pool metrics |
| Breakers and retries | resilience4j through Micrometer; Envoy’s upstream retries and overflow; Polly v8 |
| Logs | When no metric covers a node, one log search for five patterns: a database pool exhausted, too many connections, dead-lettered, retries exhausted, circuit open |
Azure Service Bus and SQS are recognised by the pattern of their metric names, and the NATS, Burrow, KEDA and Polly metrics have not yet been checked against a live exporter; the source line on each signal says which exporter it came from.
Not recognised, on purpose: gRPC client retry counters (no name is shared across languages), and SQL Server, Oracle, Cassandra, Elasticsearch, DynamoDB and Cosmos DB, which have no exporter Seylo reads yet. Their nodes say nothing, rather than suggesting an exporter Seylo would not read.
How a signal finds its node
Section titled “How a signal finds its node”A signal is placed on a node the map already has; an exporter never adds a node.
- A pool, breaker or retry belongs to the workload whose pods report it.
- A store’s signal belongs to the store its exporter reaches: the one
database, cache or broker the exporter’s workload has an edge to on the map
(for example, its
DATA_SOURCE_URIhost in an environment variable), or the store’s own pod when the exporter runs inside it as a sidecar or plugin. - Consumer lag belongs to the workload named like the consumer group, and the topic shows the same signal, capped at degraded.
A signal Seylo cannot place is listed apart, with what to do. The usual case:
an exporter whose target address is in a Secret, which Seylo never reads.
Declare exporter -> database in an
architecture file instead.
Add an exporter
Section titled “Add an exporter”The seylo-observability chart scrapes every pod annotated
prometheus.io/scrape: "true", with prometheus.io/port (and
prometheus.io/path when it is not /metrics); see
the scrape annotations.
No chart change is needed: run the exporter and annotate its pod.
| For | Run | Port |
|---|---|---|
| PostgreSQL | postgres_exporter (for example the prometheus-postgres-exporter chart). Give it a role with a reserved connection slot, or it cannot report the very moment connections run out |
9187 |
| MySQL | mysqld_exporter |
9104 |
| MongoDB | mongodb_exporter |
9216 |
| Redis | redis_exporter |
9121 |
| RabbitMQ | The rabbitmq_prometheus plugin, with per-object metrics on (prometheus.return_per_object_metrics = true, or the path /metrics/per-object) - otherwise every queue is one summed series |
15692 |
| Kafka | kafka_exporter, or Strimzi’s kafkaExporter. Name consumer groups after their workloads, or declare them |
9308 |
| Azure Service Bus, SQS | An Azure Monitor or CloudWatch exporter, with your own cloud identity. Seylo holds no cloud credential | - |
| Pools, breakers, retries | Your application’s own metrics endpoint (Micrometer, OpenTelemetry, prometheus-net), annotated like any pod | - |
Alert on a signal
Section titled “Alert on a signal”On a signal in a node’s panel, Alert on this (releaser) adds an
alert rule from Seylo’s template for that signal - for
example PostgreSQL connections near max_connections or Kafka consumer
falling behind - which you can then edit under Alerts → Rules. A rule is
only ever created by that click; Seylo does not add rules on its own.