Install the observability chart
Admin → Integrations → Observability stores in Seylo shows whether each store is reachable for the cluster you pick and, while any store is not installed or not answering, the install command. This page covers the same steps in more detail.
Seylo cannot install the chart for you, because it holds no credential that could. Someone with access to the cluster runs one Helm command.
Before you start
Section titled “Before you start”- A connected Seylo agent. Agent 0.2.1 or later finds the stores without any configuration. Use agent 0.2.5 or later if you can; the latest release is 0.2.22. 0.2.5 is the release that reads usage from the store instead of scraping the kubelet a second time (see how usage is collected). See install the agent.
- Rights to create cluster-scoped RBAC. The chart creates a ClusterRole and ClusterRoleBinding for each of VictoriaMetrics, kube-state-metrics, vlagent and the OpenTelemetry collector. They are read-only: nodes and the kubelet’s metrics endpoints for scraping, and pods, namespaces and workloads for attaching names. No ClusterRole reads Secrets or ConfigMaps.
- A StorageClass that provisions
ReadWriteOncevolumes. The chart creates three PersistentVolumeClaims, one each for metrics, logs and traces. It uses your cluster’s default StorageClass unless you setstorageClassName. - Pod Security that allows the log collector. vlagent runs on every node as
root. It mounts
/var/logread-only and keeps its position in/var/lib/seylo-vlagent, both from the node (hostPath). It has no Linux capabilities and cannot escalate. ThebaselineandrestrictedPod Security levels both refusehostPathvolumes. If your cluster enforces either level on new namespaces, vlagent will not start until the install namespace allowsprivileged. That is the trade-off for collecting logs from the node. From chart 0.2.19, node-exporter also runs on every Linux node, on the host network and with read-onlyhostPathmounts, which the same levels refuse. It is not privileged and runs as non-root. See host and ingress metrics, which also says how to turn it off. - Image pulls from Docker Hub and
registry.k8s.io, or a mirror. See configure. - A Helm version that installs from
oci://registries. You need nohelm repo addand no registry login.
Minimum Kubernetes version: the chart does not declare one, and Seylo has not published one.
Sizing: none exists yet. The chart’s CPU, memory and disk defaults are starting points, not measurements. Seylo has not measured the stores’ footprint on a real cluster. Watch the first weeks and adjust. See configure.
Verify the chart first (optional)
Section titled “Verify the chart first (optional)”The chart is signed with cosign keyless by the same release workflow as the agent, so you can check it the same way:
cosign verify --certificate-identity-regexp '^https://github.com/seylo-inc/k8s-deploy/\.github/workflows/release-agent\.yml@refs/tags/agent-v' --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/seylo-inc/charts/seylo-observability:0.2.22Expect Verified OK. If it fails, do not install. See
verify what you install for what the
flags mean.
Install
Section titled “Install”-
Install the chart into
seylo-observability.That namespace is where the agent looks by default:
Terminal window helm install seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability --create-namespaceTo change any defaults now, add
-f observability-values.yaml. See configure. -
Point the agent at it, if it needs pointing.
-
Agent 0.2.1 or later, chart in
seylo-observability: there is nothing to do. -
Agent 0.2.0: upgrade the agent. 0.2.0 needed every store address set by hand.
-
Chart in another namespace: tell the agent where it is:
Terminal window helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values --set observability.enabled=true --set observability.namespace=<your-namespace>
-
-
Open Admin → Integrations → Observability stores and pick the cluster. Press Check again if it was open before the install.
Confirm it is working
Section titled “Confirm it is working”In the cluster:
helm list -n seylo-observabilitykubectl -n seylo-observability get podskubectl -n seylo-observability get pvcExpect seylo-vm-0, seylo-vl-0 and seylo-vt-0 (the stores), one
seylo-ksm pod and one seylo-otel pod, and one seylo-vlagent pod per node.
Expect three Bound claims: data-seylo-vm-0, data-seylo-vl-0 and
data-seylo-vt-0.
In Seylo, Admin → Integrations → Observability stores. Each line shows a signal, its store and these columns:
| Column | Healthy | What it means |
|---|---|---|
| State | reachable |
The agent reached the store’s health endpoint. not installed means the store’s address does not resolve in the cluster |
| Keeps data for | the retention period (retention.period, 30d by default) |
Read back from the running store’s own flags, not from the chart’s defaults. not reported means the store did not say |
| Disk cap | stops writing below … free for metrics, drops oldest days past … for logs and traces |
Also read from the running store. The values are the chart’s metrics.minFreeDiskSpace, logs.maxDiskSpaceUsage and traces.maxDiskSpaceUsage. none set means the store has no disk cap |
| Namespace filter | namespace filter enforced |
Seylo has tested that the store applies its namespace filter. See namespace-scoped installs |
The card header says whether the stores are reached through the agent (with its version and any namespace scope) or through the API server’s service proxy.
A brand-new install shows no data yet to test the filter until the store
holds its first data. For traces, that means until a service sends some. Then:
- Observability → Metrics should answer the first query in the query catalogue.
- Observability → Logs should show your containers’ lines.
- Observability → Dashboards → Kubernetes cluster overview should draw.
Traces and APM stay empty until a service sends OTLP traces. See send traces from your services.
Upgrade
Section titled “Upgrade”The chart and the agent share version numbers, but you upgrade them separately. The latest release of both is 0.2.22. The releases that matter for observability:
| Upgrade | What you get |
|---|---|
| Agent 0.2.22 | The agent reads PodDisruptionBudgets, Jobs, CronJobs, ResourceQuotas and cert-manager’s certificates, for what Nightshift handles. The chart is the same as 0.2.21, so upgrading it is optional |
| Agent 0.2.21 | The agent reads which node autoscaler your cluster runs, for Savings. The chart is the same as 0.2.20, so upgrading it is optional |
| Chart and agent 0.2.20 | The agent reads which Key Vault secret version each pod mounted and the ServiceAccount, for Nightshift and your cloud. The chart drops the network-costs collector: opencost.networkCosts.enabled=true now fails with the reason (network traffic) |
| Chart and agent 0.2.19 | node-exporter on every Linux node for the Hosts dashboard, and ingress-nginx controllers found without annotations (host and ingress metrics). A span’s namespace is the pod it came from: if your own collector or an eBPF agent forwards spans, move it to the forwarder ports (which namespace a span belongs to). The agent reads every served version of an API group |
| Chart and agent 0.2.18 | The agent sends times in what it reads again (0.2.16 and 0.2.17 sent them empty) and reads PriorityClasses. The chart is the same as 0.2.17, so upgrading it is optional |
| Chart and agent 0.2.17 | The agent notices a dropped connection in about 25 seconds and keeps what it produced while disconnected. The chart is the same as 0.2.16, so upgrading it is optional |
| Chart and agent 0.2.16 | A retention period per store, a nightly archive of finished log days to your own bucket, and rehydrate, searched from the Logs page with agent 0.2.16. See keeping data longer. Archive and rehydrate are off by default |
| Chart and agent 0.2.14 | The log store is no longer OOMKilled under query load (2 GiB, bounded caches and queries, a Go soft memory limit on every store); the collector gets 512 MiB and reads more message fields from structured logs. Use 0.2.14 or later, not 0.2.13 |
| Agent 0.2.15 | Connect an in-cluster Grafana, Prometheus or Alertmanager by its Service name |
| Chart and agent 0.2.11 | Chargeback by team, label or namespace: the chart lets the agent reach OpenCost’s API, and the agent asks it. Optional billing integrations, all off by default: the Azure price sheet, the Azure cost export, and Datadog and OpenAI spend |
| Agent 0.2.10 | The optional Datadog connector and imports through it |
| Chart 0.2.10 | Fixes --set opencost.enabled=true on an install from before 0.2.8, which failed with nil pointer evaluating opencost.image.repository, and a log collector that crash-looped after upgrading from 0.2.0 or 0.2.1. Nothing in the cluster was changed by either failure. Adds OpenCost’s network-costs collector as an opt-in (opencost.networkCosts.enabled), off by default: it is privileged and uses the host network, and its image is not open source |
| Chart 0.2.9 | Fixes upgrading with --reuse-values from 0.2.7 or earlier, which chart 0.2.8 refused to render. Nothing in the cluster was changed by that failure. Otherwise the same as 0.2.8 |
| Chart 0.2.8 | Far fewer disk writes: only the metrics Seylo reads are stored, and the stores save every 30s (see what is stored). Optional OpenCost for cost per namespace and workload |
| Chart 0.2.5 | Pod network traffic and each node’s own CPU and memory are collected. Up to 0.2.4, the chart dropped those series, so network panels and node panels had no data |
| Agent 0.2.5 | The agent reads usage from the store instead of scraping the kubelet again, so metrics are collected once. A namespace-scoped agent gets usage data only with agent 0.2.5 and this chart |
Upgrade the chart. --reset-then-reuse-values (Helm 3.14 or later) keeps
the values you set and takes the new release’s defaults, including its image
versions:
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability --reset-then-reuse-valuesOn a Helm older than 3.14, save the values you set and pass them back as a file instead:
helm get values seylo-observability -n seylo-observability -o yaml > mine.yamlhelm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability -f mine.yamlEach store is a single pod. When an upgrade changes it, that pod restarts, and collection and queries for that signal pause until it is back. Data on the volumes is kept.
Upgrade the agent as described in
upgrade the agent, naming --version 0.2.22.
Then check that Admin → Integrations → Observability stores shows the new agent
version in the Stores card’s header.
Uninstall
Section titled “Uninstall”-
Remove the release:
Terminal window helm uninstall seylo-observability -n seylo-observability -
Delete the data. Helm does not delete volumes created by a StatefulSet, so the three claims, and everything stored in them, stay until you delete them:
Terminal window kubectl -n seylo-observability delete pvc data-seylo-vm-0 data-seylo-vl-0 data-seylo-vt-0kubectl delete namespace seylo-observabilityThis deletes the stored metrics, logs and traces permanently. A volume whose reclaim policy is
Retainoutlives its claim, and the log collector leaves a buffer on each node: leaving Seylo covers both, how to keep the data instead, and a command that checks nothing is left. -
Optionally stop the agent looking for the stores:
Terminal window helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent --namespace seylo-agent --reset-then-reuse-values --set observability.enabled=falseWithout this step, the Observability stores tab shows each store as
not installed, which is accurate. An agent on the defaultkubeletsource also retries the store every ten minutes, falls back to the kubelet in between, and records a warning each time.
vlagent’s position files stay in /var/lib/seylo-vlagent on each node. They
hold read positions in log files, not log content.
After uninstalling: metrics, logs, traces, APM and dashboards have no data for that cluster. Managed vCPU shows as not measured, never as an estimate. Usage falls back to whatever the agent can read itself. See how usage is collected.