Skip to content

Host and ingress metrics

From chart version 0.2.19, the seylo-observability chart collects two more kinds of metrics, so the Hosts and Ingress dashboard templates have data without anything else to install. Chart 0.2.19 was released on 2 November 2026; with an older chart, these templates need the exporters listed under templates that need an exporter.

The chart runs Prometheus node-exporter (v1.12.1, pinned by digest) on every Linux node. It is on by default. It reports each node’s CPU, memory, filesystem space, load and network interfaces, and nothing else.

What it runs, for your platform team:

  • One pod per Linux node (a DaemonSet). It tolerates every taint, so tainted node pools report too.
  • On the host network, listening on port 19100 of the node’s internal address (19101 from chart 0.2.24; see AKS). Traffic per host network interface can be read only from the host’s network namespace. The port is not 9100, so a node-exporter you already run, such as kube-prometheus-stack’s, keeps its own.
  • Not in the host’s process namespace (no hostPID). The host’s /proc is mounted instead.
  • Not privileged, as user 65534 (non-root), with every Linux capability dropped, a read-only root filesystem and no Kubernetes API token.
  • The host’s /proc, /sys and / mounted read-only.
  • Five node-exporter collectors only: cpu, meminfo, filesystem, loadavg and netdev. Pod volumes, container layers and per-pod network interfaces are not reported.
  • Requests cpu: 10m, memory: 24Mi; limits cpu: 200m, memory: 64Mi (nodeExporter.resources).

Host series carry the chart’s namespace (seylo-observability by default), taken from Kubernetes, so only people who may see that namespace see the Hosts dashboard’s data.

Two of Seylo’s own policy rules look for exactly what node-exporter needs: a workload on the host network, and host path volumes. On Admin → Posture, those findings on node-exporter are still listed, so nothing is hidden, but:

  • each is marked exempt: Seylo’s own, with the reason under it;
  • they are counted on their own row, Exempt (Seylo’s own), and left out of the score;
  • no fix is offered, and they cannot be suppressed.

The exemption holds only while the DaemonSet is as the chart installs it: in the chart’s namespace, with the chart’s labels, one node-exporter container that is not privileged, runs as non-root with its capabilities dropped, does not share the host’s process or IPC namespace, and mounts only /proc, /sys and /, read-only. A copy anywhere else, or one changed to run privileged or to mount anything more, is reported like any other workload. Every other rule applies to node-exporter as usual.

Terminal window
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability \
--namespace seylo-observability --reset-then-reuse-values \
--set nodeExporter.enabled=false

A namespace that enforces the Pod Security baseline or restricted level refuses host-network pods and hostPath volumes; turn node-exporter off there. With the metrics store off (metrics.enabled=false), node-exporter is not installed either.

If the Hosts template says none of its metrics were seen, node-exporter is turned off or the chart is older than 0.2.19. Upgrade with the command above, using --set nodeExporter.enabled=true.

AKS’s node image runs its own node exporter on port 19100 of every node, started with the node. On AKS, chart 0.2.19 to 0.2.23 therefore starts one node-exporter pod per node that cannot bind the port: they crash-loop with listen tcp <node-ip>:19100: bind: address already in use, and helm upgrade --wait times out. The metrics, logs and traces stores are not affected. The Hosts dashboard has no data until this is fixed.

If you upgraded to one of those charts on AKS, move node-exporter to port 19101:

Terminal window
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --namespace seylo-observability --reset-then-reuse-values --set nodeExporter.port=19101

From chart 0.2.24 the default is 19101, so a new install needs nothing. Allow the port in any firewall rule that named 19100 for node-exporter.

If the upgrade that brought node-exporter in failed (--wait timed out on the crash-looping pods), Helm compares the next upgrade with the last release that was deployed, which has no node-exporter. The next upgrade then adds port 19101 beside 19100 instead of replacing it, and Kubernetes refuses it: spec.template.spec.containers[0].ports[1].name: Duplicate value: "metrics". Delete the DaemonSet once, then run the same upgrade again, which recreates it:

Terminal window
kubectl -n seylo-observability delete daemonset seylo-node-exporter

An upgrade from a release that shows as deployed, run without --wait as the upgrade commands in these docs are, compares with the 19100 manifest and replaces the port.

The chart finds ingress-nginx controller pods by their standard labels, app.kubernetes.io/name=ingress-nginx and app.kubernetes.io/component=controller, in any namespace, and reads their metrics port, 10254. You do not need to annotate them. You do need to:

  • turn the controller’s metrics on. With the ingress-nginx Helm chart, --set controller.metrics.enabled=true, which also declares the port. A pod that does not declare port 10254 is not scraped;
  • allow port 10254 from the seylo-observability namespace, if a NetworkPolicy protects the controller’s namespace.

A controller pod already annotated prometheus.io/scrape: "true" is read once, by the annotation, not twice. To stop reading controllers by their labels, set metrics.scrapeIngressNginx=false on the seylo-observability chart. A controller with other labels still needs the scrape annotations.

The series carry the controller’s namespace; each Ingress’s own namespace arrives as exported_namespace, which the Ingress template filters on. Only people who may see the controller’s namespace see its data.

Both scrape jobs keep only the series the Hosts and Ingress templates read. metrics.keepAllMetrics: true keeps everything, as for the kubelet and cAdvisor. To keep a few more, list them in metrics.extraKeepMetrics, for example node_disk_.+. A node-exporter family outside the five collectors also needs its collector turned on, for example:

Terminal window
--set 'nodeExporter.extraArgs={--collector.diskstats}'

See what is stored.