Skip to content

Install the observability chart

Admin → Integrations → Observability stores in Seylo shows whether each store is reachable for the cluster you pick and, while any store is not installed or not answering, the install command. This page covers the same steps in more detail.

Seylo cannot install the chart for you, because it holds no credential that could. Someone with access to the cluster runs one Helm command.

  • A connected Seylo agent. Agent 0.2.1 or later finds the stores without any configuration. Use agent 0.2.5 or later if you can; the latest release is 0.2.22. 0.2.5 is the release that reads usage from the store instead of scraping the kubelet a second time (see how usage is collected). See install the agent.
  • Rights to create cluster-scoped RBAC. The chart creates a ClusterRole and ClusterRoleBinding for each of VictoriaMetrics, kube-state-metrics, vlagent and the OpenTelemetry collector. They are read-only: nodes and the kubelet’s metrics endpoints for scraping, and pods, namespaces and workloads for attaching names. No ClusterRole reads Secrets or ConfigMaps.
  • A StorageClass that provisions ReadWriteOnce volumes. The chart creates three PersistentVolumeClaims, one each for metrics, logs and traces. It uses your cluster’s default StorageClass unless you set storageClassName.
  • Pod Security that allows the log collector. vlagent runs on every node as root. It mounts /var/log read-only and keeps its position in /var/lib/seylo-vlagent, both from the node (hostPath). It has no Linux capabilities and cannot escalate. The baseline and restricted Pod Security levels both refuse hostPath volumes. If your cluster enforces either level on new namespaces, vlagent will not start until the install namespace allows privileged. That is the trade-off for collecting logs from the node. From chart 0.2.19, node-exporter also runs on every Linux node, on the host network and with read-only hostPath mounts, which the same levels refuse. It is not privileged and runs as non-root. See host and ingress metrics, which also says how to turn it off.
  • Image pulls from Docker Hub and registry.k8s.io, or a mirror. See configure.
  • A Helm version that installs from oci:// registries. You need no helm repo add and no registry login.

Minimum Kubernetes version: the chart does not declare one, and Seylo has not published one.

Sizing: none exists yet. The chart’s CPU, memory and disk defaults are starting points, not measurements. Seylo has not measured the stores’ footprint on a real cluster. Watch the first weeks and adjust. See configure.

The chart is signed with cosign keyless by the same release workflow as the agent, so you can check it the same way:

Terminal window
cosign verify --certificate-identity-regexp '^https://github.com/seylo-inc/k8s-deploy/\.github/workflows/release-agent\.yml@refs/tags/agent-v' --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/seylo-inc/charts/seylo-observability:0.2.22

Expect Verified OK. If it fails, do not install. See verify what you install for what the flags mean.

  1. Install the chart into seylo-observability.

    That namespace is where the agent looks by default:

    Terminal window
    helm install seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability --create-namespace

    To change any defaults now, add -f observability-values.yaml. See configure.

  2. Point the agent at it, if it needs pointing.

    • Agent 0.2.1 or later, chart in seylo-observability: there is nothing to do.

    • Agent 0.2.0: upgrade the agent. 0.2.0 needed every store address set by hand.

    • Chart in another namespace: tell the agent where it is:

      Terminal window
      helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values --set observability.enabled=true --set observability.namespace=<your-namespace>
  3. Open Admin → Integrations → Observability stores and pick the cluster. Press Check again if it was open before the install.

In the cluster:

Terminal window
helm list -n seylo-observability
kubectl -n seylo-observability get pods
kubectl -n seylo-observability get pvc

Expect seylo-vm-0, seylo-vl-0 and seylo-vt-0 (the stores), one seylo-ksm pod and one seylo-otel pod, and one seylo-vlagent pod per node. Expect three Bound claims: data-seylo-vm-0, data-seylo-vl-0 and data-seylo-vt-0.

In Seylo, Admin → Integrations → Observability stores. Each line shows a signal, its store and these columns:

Column Healthy What it means
State reachable The agent reached the store’s health endpoint. not installed means the store’s address does not resolve in the cluster
Keeps data for the retention period (retention.period, 30d by default) Read back from the running store’s own flags, not from the chart’s defaults. not reported means the store did not say
Disk cap stops writing below … free for metrics, drops oldest days past … for logs and traces Also read from the running store. The values are the chart’s metrics.minFreeDiskSpace, logs.maxDiskSpaceUsage and traces.maxDiskSpaceUsage. none set means the store has no disk cap
Namespace filter namespace filter enforced Seylo has tested that the store applies its namespace filter. See namespace-scoped installs

The card header says whether the stores are reached through the agent (with its version and any namespace scope) or through the API server’s service proxy.

A brand-new install shows no data yet to test the filter until the store holds its first data. For traces, that means until a service sends some. Then:

  • Observability → Metrics should answer the first query in the query catalogue.
  • Observability → Logs should show your containers’ lines.
  • Observability → Dashboards → Kubernetes cluster overview should draw.

Traces and APM stay empty until a service sends OTLP traces. See send traces from your services.

The chart and the agent share version numbers, but you upgrade them separately. The latest release of both is 0.2.22. The releases that matter for observability:

Upgrade What you get
Agent 0.2.22 The agent reads PodDisruptionBudgets, Jobs, CronJobs, ResourceQuotas and cert-manager’s certificates, for what Nightshift handles. The chart is the same as 0.2.21, so upgrading it is optional
Agent 0.2.21 The agent reads which node autoscaler your cluster runs, for Savings. The chart is the same as 0.2.20, so upgrading it is optional
Chart and agent 0.2.20 The agent reads which Key Vault secret version each pod mounted and the ServiceAccount, for Nightshift and your cloud. The chart drops the network-costs collector: opencost.networkCosts.enabled=true now fails with the reason (network traffic)
Chart and agent 0.2.19 node-exporter on every Linux node for the Hosts dashboard, and ingress-nginx controllers found without annotations (host and ingress metrics). A span’s namespace is the pod it came from: if your own collector or an eBPF agent forwards spans, move it to the forwarder ports (which namespace a span belongs to). The agent reads every served version of an API group
Chart and agent 0.2.18 The agent sends times in what it reads again (0.2.16 and 0.2.17 sent them empty) and reads PriorityClasses. The chart is the same as 0.2.17, so upgrading it is optional
Chart and agent 0.2.17 The agent notices a dropped connection in about 25 seconds and keeps what it produced while disconnected. The chart is the same as 0.2.16, so upgrading it is optional
Chart and agent 0.2.16 A retention period per store, a nightly archive of finished log days to your own bucket, and rehydrate, searched from the Logs page with agent 0.2.16. See keeping data longer. Archive and rehydrate are off by default
Chart and agent 0.2.14 The log store is no longer OOMKilled under query load (2 GiB, bounded caches and queries, a Go soft memory limit on every store); the collector gets 512 MiB and reads more message fields from structured logs. Use 0.2.14 or later, not 0.2.13
Agent 0.2.15 Connect an in-cluster Grafana, Prometheus or Alertmanager by its Service name
Chart and agent 0.2.11 Chargeback by team, label or namespace: the chart lets the agent reach OpenCost’s API, and the agent asks it. Optional billing integrations, all off by default: the Azure price sheet, the Azure cost export, and Datadog and OpenAI spend
Agent 0.2.10 The optional Datadog connector and imports through it
Chart 0.2.10 Fixes --set opencost.enabled=true on an install from before 0.2.8, which failed with nil pointer evaluating opencost.image.repository, and a log collector that crash-looped after upgrading from 0.2.0 or 0.2.1. Nothing in the cluster was changed by either failure. Adds OpenCost’s network-costs collector as an opt-in (opencost.networkCosts.enabled), off by default: it is privileged and uses the host network, and its image is not open source
Chart 0.2.9 Fixes upgrading with --reuse-values from 0.2.7 or earlier, which chart 0.2.8 refused to render. Nothing in the cluster was changed by that failure. Otherwise the same as 0.2.8
Chart 0.2.8 Far fewer disk writes: only the metrics Seylo reads are stored, and the stores save every 30s (see what is stored). Optional OpenCost for cost per namespace and workload
Chart 0.2.5 Pod network traffic and each node’s own CPU and memory are collected. Up to 0.2.4, the chart dropped those series, so network panels and node panels had no data
Agent 0.2.5 The agent reads usage from the store instead of scraping the kubelet again, so metrics are collected once. A namespace-scoped agent gets usage data only with agent 0.2.5 and this chart

Upgrade the chart. --reset-then-reuse-values (Helm 3.14 or later) keeps the values you set and takes the new release’s defaults, including its image versions:

Terminal window
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability --reset-then-reuse-values

On a Helm older than 3.14, save the values you set and pass them back as a file instead:

Terminal window
helm get values seylo-observability -n seylo-observability -o yaml > mine.yaml
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability -f mine.yaml

Each store is a single pod. When an upgrade changes it, that pod restarts, and collection and queries for that signal pause until it is back. Data on the volumes is kept.

Upgrade the agent as described in upgrade the agent, naming --version 0.2.22. Then check that Admin → Integrations → Observability stores shows the new agent version in the Stores card’s header.

  1. Remove the release:

    Terminal window
    helm uninstall seylo-observability -n seylo-observability
  2. Delete the data. Helm does not delete volumes created by a StatefulSet, so the three claims, and everything stored in them, stay until you delete them:

    Terminal window
    kubectl -n seylo-observability delete pvc data-seylo-vm-0 data-seylo-vl-0 data-seylo-vt-0
    kubectl delete namespace seylo-observability

    This deletes the stored metrics, logs and traces permanently. A volume whose reclaim policy is Retain outlives its claim, and the log collector leaves a buffer on each node: leaving Seylo covers both, how to keep the data instead, and a command that checks nothing is left.

  3. Optionally stop the agent looking for the stores:

    Terminal window
    helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values --set observability.enabled=false

    Without this step, the Observability stores tab shows each store as not installed, which is accurate. An agent on the default kubelet source also retries the store every ten minutes, falls back to the kubelet in between, and records a warning each time.

vlagent’s position files stay in /var/lib/seylo-vlagent on each node. They hold read positions in log files, not log content.

After uninstalling: metrics, logs, traces, APM and dashboards have no data for that cluster. Managed vCPU shows as not measured, never as an estimate. Usage falls back to whatever the agent can read itself. See how usage is collected.