Skip to content

Across clusters

A common migration: the app moves from cluster old to cluster new, and old’s Ingress is kept, pointed at new’s ingress address, so users and DNS need not change. Then users see more 502s and 504s. Looking at new alone, the app may be perfectly healthy: the errors are born at the front proxy.

Seylo links clusters of your organisation when one sends traffic to another, so the service map, incidents and Nightshift see the whole path.

Nothing to configure. Seylo matches an upstream in one cluster to an entry point of another cluster in the same organisation:

An upstream in one cluster An entry point of another
An ExternalName Service’s name An Ingress or HTTPRoute host (*. covers one label)
A selectorless Service’s Endpoints address An Ingress’s published load-balancer address
A host in a workload’s environment, or in a ConfigMap it uses, such as a proxy’s nginx.conf A LoadBalancer Service’s address or externalIPs
  • Exact matches only. Seylo never resolves DNS: what a name resolves to inside your cluster is not something Seylo can know from outside. A front that reaches new only through a name DNS maps to new’s address is not linked.
  • Two ingress layers in one cluster, such as an Ingress of one class proxying to another controller, are linked the same way.
  • Never across organisations.
  • Each person sees a link only as far as their access allows, on each cluster.

On the service map, a cluster with such links shows an Across clusters card: Calls into this cluster, Calls out of this cluster and Two ingress layers in this cluster, each link with the evidence that made it. The other cluster’s end is drawn on the map as a node that names its cluster, or a cluster you cannot see when your access does not reach it, and the edge’s source reads other cluster. A link whose Host no Ingress on the far side serves is marked no rule for Host …, and one that matched more than one entry point, ambiguous.

During an investigation of a workload behind an Ingress whose errors its own logs do not explain, Nightshift reads the ingress controllers’ own access and error lines, hop by hop, and says where the 5xx was first made:

Answer What it means
Born in the app New’s ingress passed on a 5xx the app answered. The investigation continues on the app
Born at the new ingress New’s ingress made the 5xx itself: the app refused, reset, timed out or had no endpoints. Its own error lines say which
Born at the front proxy New answered the proxy without a 5xx, or the proxy gave up first, but the front answered users with a 5xx: a 504 from the proxy’s timeout, a 502 from its connection or TLS to new
Front not connected The front is not connected to Seylo, and new answered the proxy with no 5xx. The answer says what new proved, the likely front causes its lines point at, and that connecting the front cluster lets Nightshift confirm
No 5xx seen Nothing answered 5xx in the window, at any hop read

How a hop tells: a status 499 on new means the client - the front proxy - closed the connection before the answer, and its request time is how long the front waited; a TLS handshake arriving on a plain-HTTP port means the front speaks HTTPS to an HTTP port; a request served by the default backend means no Ingress rule matched the Host.

Every count is of the lines read, from a bounded sample, and said as one. When request ids join the two hops, the answer is per request; otherwise it compares the two hops’ totals over the same window, and says so.

On an incident, the line Where the 5xx is born: shows the answer.

When the 5xx is born at the front, Nightshift names the cause and the exact change, as a kubectl annotate line and as YAML, on the Ingress it belongs on:

Cause What users see
The front’s read timeout is shorter than new takes to answer 504
The front speaks HTTPS to new’s plain-HTTP port 502
The front speaks plain HTTP to an HTTPS port 400
The front passes a Host that no Ingress on new serves 404 from the default backend
TLS verification on, without SNI, so new presents its default certificate 502
The front ends TLS and speaks HTTP, and new’s Ingress redirects HTTP to HTTPS A redirect loop
Old pinned sessions to one pod; new pins nothing Errors for signed-in users

A new timeout is proposed only from the slowest answer new was seen to give, one and a half times it, and never past 300 seconds. When every slow request was cut off by the proxy, the real latency was never measured, and the finding says so instead of picking a number.

A person applies it. Seylo never changes a front proxy on its own. An Ingress that Helm, Argo CD or Flux owns is changed in its source, and the finding names the owner.

When a link shows a migration - old’s front proxies to new - Nightshift also compares the app on both clusters, by name only, never a value:

  • ConfigMap keys, Secret keys and environment variable names the old one had and the new one lacks. A Secret consumed whole is compared by the Secret’s name;
  • replicas or a resource request at less than half, or more than twice, the old value;
  • session affinity.
  • The service map built on each cluster. The links come from the maps.
  • The observability chart on each cluster, with its log store: the hops are read from the ingress controllers’ logs.
  • An NGINX ingress controller (community ingress-nginx, or AKS application routing) logging in its default format, or in JSON with the same fields. Gateway API controllers and other ingress controllers are not read yet.
  • Read across namespaces during an incident on (the default, on Nightshift → Settings → During an incident). With it off, the front is not read.
  • An incident stays one cluster’s. Its walk follows the link into the other cluster; alerts there that the map relates to it are listed under Across clusters as symptoms on another cluster, not made members.
  • No writes. Nothing on this page changes either cluster.