Skip to content

Configure the observability chart

Keep your changes in one values file, for example observability-values.yaml, and apply it with:

Terminal window
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability -f observability-values.yaml

To see every value with its default:

Terminal window
helm show values oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22

Every default quoted on this page comes from the chart’s values.yaml at version 0.2.22.

Every store keeps data for retention.period and has a disk cap. With only a time cap, one noisy workload could fill the volume. With only a disk cap, you could not tell how far back you can look.

Key Default What it does
retention.period 30d How long all three stores keep data
logs.retentionPeriod, traces.retentionPeriod, metrics.retentionPeriod empty From chart 0.2.16, one store’s own period, overriding retention.period. See keeping data longer
metrics.minFreeDiskSpace 2GiB VictoriaMetrics has no setting that drops old data when the disk fills. Instead, it stops accepting writes when free space on its volume falls below this
logs.maxDiskSpaceUsage 16GiB When VictoriaLogs’ data passes this, it drops whole days, oldest first. Keep it below logs.persistence.size
traces.maxDiskSpaceUsage 8GiB The same for VictoriaTraces. Keep it below traces.persistence.size

Whichever limit is reached first applies. Admin → Integrations → Observability stores reads both back from each running store, so it shows what is running, not what you intended.

Example: keep 14 days, and give logs a bigger volume and cap:

retention:
period: 14d
logs:
persistence:
size: 40Gi
maxDiskSpaceUsage: 32GiB

The numbers in this example are illustrations, not recommendations. Seylo has no sizing guidance for the stores yet (see below). If you change logs.persistence.size on an existing install, read growing a volume first.

Key Default
storageClassName "", your cluster’s default StorageClass
metrics.persistence.size 20Gi
logs.persistence.size 20Gi
traces.persistence.size 10Gi

Each store is a StatefulSet with one replica and one ReadWriteOnce claim: data-seylo-vm-0, data-seylo-vl-0 and data-seylo-vt-0. A store has no replica, so while its pod restarts, that signal is neither collected nor queryable.

Kubernetes does not let Helm change a StatefulSet’s volume size or StorageClass after install. Changing *.persistence.size or storageClassName in your values file makes helm upgrade fail with updates to statefulset spec for fields other than ... are forbidden.

You can grow a volume in place if its StorageClass allows expansion. Check the ALLOWVOLUMEEXPANSION column of kubectl get storageclass. The example below grows the logs volume:

  1. Edit the claim and raise spec.resources.requests.storage:

    Terminal window
    kubectl -n seylo-observability edit pvc data-seylo-vl-0
  2. Remove the StatefulSet object but leave its pod and volume running:

    Terminal window
    kubectl -n seylo-observability delete statefulset seylo-vl --cascade=orphan
  3. Put the new size, and a matching logs.maxDiskSpaceUsage, in your values file and run the helm upgrade above. Helm recreates the StatefulSet, which adopts the existing pod and claim.

The StatefulSets are seylo-vm (metrics), seylo-vl (logs) and seylo-vt (traces). Changing the StorageClass of existing data is a migration, and the chart does not do it. The simplest route is to uninstall, delete the claims and install again, which loses the stored data.

Component Key Requests Limits
VictoriaMetrics metrics.resources cpu: 100m, memory: 256Mi cpu: "1", memory: 1Gi
kube-state-metrics kubeStateMetrics.resources cpu: 20m, memory: 64Mi cpu: 200m, memory: 256Mi
VictoriaLogs logs.resources cpu: 100m, memory: 512Mi cpu: "1", memory: 2Gi
vlagent (per node) logs.collector.resources cpu: 50m, memory: 96Mi cpu: 300m, memory: 512Mi
VictoriaTraces traces.resources cpu: 50m, memory: 128Mi cpu: 500m, memory: 512Mi
OpenTelemetry collector traces.collector.resources cpu: 50m, memory: 128Mi cpu: 500m, memory: 512Mi
OpenCost (when enabled) opencost.resources cpu: 10m, memory: 55Mi cpu: 500m, memory: 1Gi

nodeSelector, tolerations and affinity (all empty by default) apply to the stores, kube-state-metrics and the collector. vlagent ignores them. It has its own logs.collector.tolerations, which defaults to operator: Exists so that it runs on every node, tainted ones included. A node without vlagent has no logs in Seylo.

Key Default
metrics.image victoriametrics/victoria-metrics:v1.153.0
kubeStateMetrics.image registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0
logs.image victoriametrics/victoria-logs:v1.52.0
logs.collector.image victoriametrics/vlagent:v1.52.0
traces.image victoriametrics/victoria-traces:v0.12.0
traces.collector.image otel/opentelemetry-collector-contrib:0.161.0
opencost.image ghcr.io/opencost/opencost:1.121.3, pinned by digest

Each has repository and tag. To pull from a mirror, set repository. Keep the tags the chart release names. Seylo tests its queries against those versions.

VictoriaMetrics does its own scraping. There is no separate scraping agent.

Key Default What it does
metrics.scrapeInterval 30s How often every target is scraped
metrics.scrapeAnnotatedPods true Also scrape pods that carry the annotations below
kubeStateMetrics.enabled true Run kube-state-metrics. Managed vCPU is measured from it. With it off, the cluster shows as not measured, and most dashboard templates are empty

What is always scraped: each node’s kubelet, its cAdvisor endpoint, kube-state-metrics, and the collector’s own health. Duplicate pod-level series from cAdvisor are dropped, so that a pod’s CPU and memory are not counted twice. Pod network traffic and each node’s own CPU and memory are kept from chart 0.2.5.

The kubelet and cAdvisor expose hundreds of metric families, and most of them describe the kubelet itself. From chart 0.2.8 only the families Seylo, its dashboard templates and its query catalogue read are stored, plus a few that people commonly chart:

  • cAdvisor: CPU usage and throttling; memory working set, RSS, cache and usage; out-of-memory events; pod network; filesystem reads, writes, usage and limit.
  • kubelet: volume usage, running pod and container counts, and probe results.

Not stored: block-I/O, task-state, thread, socket, file-descriptor, process and spec series, and the kubelet’s own request and storage histograms.

Key Default What it does
metrics.keepAllMetrics false true stores every family, as before 0.2.8
metrics.extraKeepMetrics [] More families to keep, as names or regular expressions, for example ["container_threads", "kubelet_pleg_.+"]
storeFlushInterval 30s How often each store saves what it holds in memory to disk. The stores’ own default is 5s; fewer saves mean fewer disk writes

Put these on the pod template: spec.template.metadata.annotations of a Deployment, StatefulSet or DaemonSet. On the workload’s own metadata they do nothing, because the scrape job discovers pods, not workloads.

Annotation Required Meaning
prometheus.io/scrape: "true" yes Scrape this pod. It must be the string "true"
prometheus.io/port in practice, yes The port serving metrics
prometheus.io/path no The metrics path. Without it, /metrics

The scrape is plain HTTP. The job does not read prometheus.io/scheme.

Every series gets namespace and pod from Kubernetes, not from what the pod reports. It also gets app from the pod’s app.kubernetes.io/name label. The namespace label decides who may see the series, which is why it is never taken from the target.

If your application namespace has a NetworkPolicy that limits ingress, it must admit VictoriaMetrics from the chart’s namespace on the metrics port. Otherwise the target is never scraped.

A coding agent with the scrape_target_propose capability can propose these annotations for a Deployment. Like any change, the proposal waits for approval.

The OpenTelemetry collector accepts OTLP on two ports:

Protocol Address
gRPC seylo-otel.seylo-observability.svc:4317
HTTP http://seylo-otel.seylo-observability.svc:4318

It takes traces only. Application logs come from stdout and stderr through vlagent. Metrics come from scraping.

Send traces and APM data shows four ways in, with copyable steps:

  • auto-instrumentation with no code;
  • an SDK;
  • forwarding from a collector you already run;
  • eBPF.

It also covers what the collector does not receive (Datadog, Jaeger Thrift, Zipkin) and how to check that spans arrive.

networkPolicy.otelIngressFrom limits which namespaces may send to the collector. The default is [], which admits every namespace. It takes standard NetworkPolicy from entries.

opencost.enabled: true runs OpenCost next to the stores. It works out what each namespace and workload costs from the data this chart already collects, and writes the result back to VictoriaMetrics as metrics. It is off by default.

What it runs, and with what permissions:

  • One pod. No DaemonSet and no node-exporter: nothing privileged, and nothing reads the host.
  • Read-only access to the workload, node, namespace, volume and job objects it prices. No Secrets. ConfigMaps are readable only in its own namespace. No exec, attach or proxy.
  • Its MCP server stays off, and its port admits two callers: VictoriaMetrics, which scrapes it, and — from chart 0.2.11 — the Seylo agent, which asks OpenCost’s API a fixed list of read questions for chargeback. OpenCost’s API has no access control of its own, so the agent narrows every answer to the asking person’s namespaces before it leaves your cluster. Seylo reads the stored metrics through the agent too, scoped like every other metric.
  • It needs outbound HTTPS to fetch public list prices: Azure’s Retail Prices API, or AWS’s public price files. No cloud credential is needed for that, and Seylo never holds one.
  • kube-state-metrics also exports PersistentVolumes from 0.2.8, because OpenCost prices storage by capacity.
opencost:
enabled: true
Key Default Notes
opencost.enabled false Needs metrics.enabled and kubeStateMetrics.enabled; the chart refuses to render otherwise and says why
opencost.clusterId default-cluster Written on every series OpenCost exports, as cluster_id
opencost.image see Images Pinned by digest
opencost.networkCosts.enabled removed Charts 0.2.10 to 0.2.19 only: a network-costs collector whose image is Kubecost’s, not open source. Leave it off; later charts refuse true. See network traffic
opencost.azureBilling.priceSheet.*, opencost.azureBilling.cloudCosts.*, opencost.plugins.datadog.*, opencost.plugins.openai.* off Billing integrations, from chart 0.2.11. Each names a Secret you create; no key is ever a chart value. Seylo now offers only the Datadog and OpenAI plugins for new setups; Azure prices come from your Azure bill through a cloud account. See make the figures match your invoice
Key Default Notes
namePrefix seylo Names every Service (seylo-vm, seylo-vl, seylo-vt, seylo-otel, seylo-ksm). The agent looks for these names. If you change the prefix, also set the agent’s observability.metricsUrl, logsUrl and tracesUrl
agent.namespace, agent.podLabels seylo-agent, app.kubernetes.io/name: seylo-agent Who the NetworkPolicy admits to the stores
networkPolicy.enabled true Turn it off only for a cluster Seylo reaches directly, or use extraIngressFrom
networkPolicy.extraIngressFrom [] More from entries admitted to the stores
logs.collector.excludeOwnNamespace true Do not collect the chart’s own logs. A collector reading its own output can feed itself on a busy node
podSecurity.runAsUser, podSecurity.fsGroup 65534, 65534 The non-root user for everything except vlagent
metrics.enabled, logs.enabled, traces.enabled true Turn a whole signal off. kube-state-metrics runs only when metrics is on
You change What happens
metrics.scrapeInterval, metrics.scrapeAnnotatedPods, metrics.keepAllMetrics, metrics.extraKeepMetrics VictoriaMetrics re-reads its scrape file within a minute. No restart
storeFlushInterval All three stores restart. Data on the volumes is kept
opencost.enabled OpenCost starts (or is removed). VictoriaMetrics starts scraping it within a minute
Annotations on your own pods Picked up by discovery. Nothing to do on the chart
retention.period, a disk cap, a store’s resources or image That store’s pod restarts. Data on the volume is kept
traces.collector.* The collector restarts
logs.collector.* vlagent restarts node by node. Its position files survive the restart, so lines are neither lost nor sent twice
networkPolicy.*, agent.* The NetworkPolicies are updated. No restart
*.persistence.size, storageClassName Refused by Kubernetes. See growing a volume
metrics.enabled, logs.enabled or traces.enabled set to false That signal’s components are removed. Its claim is kept until you delete it