Configure the observability chart
Keep your changes in one values file, for example observability-values.yaml,
and apply it with:
helm upgrade seylo-observability oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22 --namespace seylo-observability -f observability-values.yamlTo see every value with its default:
helm show values oci://ghcr.io/seylo-inc/charts/seylo-observability --version 0.2.22Every default quoted on this page comes from the chart’s values.yaml at
version 0.2.22.
Retention: capped by time and by disk
Section titled “Retention: capped by time and by disk”Every store keeps data for retention.period and has a disk cap. With only
a time cap, one noisy workload could fill the volume. With only a disk cap, you
could not tell how far back you can look.
| Key | Default | What it does |
|---|---|---|
retention.period |
30d |
How long all three stores keep data |
logs.retentionPeriod, traces.retentionPeriod, metrics.retentionPeriod |
empty | From chart 0.2.16, one store’s own period, overriding retention.period. See keeping data longer |
metrics.minFreeDiskSpace |
2GiB |
VictoriaMetrics has no setting that drops old data when the disk fills. Instead, it stops accepting writes when free space on its volume falls below this |
logs.maxDiskSpaceUsage |
16GiB |
When VictoriaLogs’ data passes this, it drops whole days, oldest first. Keep it below logs.persistence.size |
traces.maxDiskSpaceUsage |
8GiB |
The same for VictoriaTraces. Keep it below traces.persistence.size |
Whichever limit is reached first applies. Admin → Integrations → Observability stores reads both back from each running store, so it shows what is running, not what you intended.
Example: keep 14 days, and give logs a bigger volume and cap:
retention: period: 14dlogs: persistence: size: 40Gi maxDiskSpaceUsage: 32GiBThe numbers in this example are illustrations, not recommendations. Seylo has
no sizing guidance for the stores yet (see below). If you change
logs.persistence.size on an existing install, read
growing a volume first.
Storage
Section titled “Storage”| Key | Default |
|---|---|
storageClassName |
"", your cluster’s default StorageClass |
metrics.persistence.size |
20Gi |
logs.persistence.size |
20Gi |
traces.persistence.size |
10Gi |
Each store is a StatefulSet with one replica and one ReadWriteOnce claim:
data-seylo-vm-0, data-seylo-vl-0 and data-seylo-vt-0. A store has no
replica, so while its pod restarts, that signal is neither collected nor
queryable.
Growing a volume
Section titled “Growing a volume”Kubernetes does not let Helm change a StatefulSet’s volume size or
StorageClass after install. Changing *.persistence.size or
storageClassName in your values file makes helm upgrade fail with
updates to statefulset spec for fields other than ... are forbidden.
You can grow a volume in place if its StorageClass allows expansion. Check the
ALLOWVOLUMEEXPANSION column of kubectl get storageclass. The example below
grows the logs volume:
-
Edit the claim and raise
spec.resources.requests.storage:Terminal window kubectl -n seylo-observability edit pvc data-seylo-vl-0 -
Remove the StatefulSet object but leave its pod and volume running:
Terminal window kubectl -n seylo-observability delete statefulset seylo-vl --cascade=orphan -
Put the new size, and a matching
logs.maxDiskSpaceUsage, in your values file and run thehelm upgradeabove. Helm recreates the StatefulSet, which adopts the existing pod and claim.
The StatefulSets are seylo-vm (metrics), seylo-vl (logs) and seylo-vt
(traces). Changing the StorageClass of existing data is a migration, and the
chart does not do it. The simplest route is to uninstall, delete the claims and
install again, which loses the stored data.
Resources
Section titled “Resources”| Component | Key | Requests | Limits |
|---|---|---|---|
| VictoriaMetrics | metrics.resources |
cpu: 100m, memory: 256Mi |
cpu: "1", memory: 1Gi |
| kube-state-metrics | kubeStateMetrics.resources |
cpu: 20m, memory: 64Mi |
cpu: 200m, memory: 256Mi |
| VictoriaLogs | logs.resources |
cpu: 100m, memory: 512Mi |
cpu: "1", memory: 2Gi |
| vlagent (per node) | logs.collector.resources |
cpu: 50m, memory: 96Mi |
cpu: 300m, memory: 512Mi |
| VictoriaTraces | traces.resources |
cpu: 50m, memory: 128Mi |
cpu: 500m, memory: 512Mi |
| OpenTelemetry collector | traces.collector.resources |
cpu: 50m, memory: 128Mi |
cpu: 500m, memory: 512Mi |
| OpenCost (when enabled) | opencost.resources |
cpu: 10m, memory: 55Mi |
cpu: 500m, memory: 1Gi |
Scheduling
Section titled “Scheduling”nodeSelector, tolerations and affinity (all empty by default) apply to the
stores, kube-state-metrics and the collector. vlagent ignores them. It has
its own logs.collector.tolerations, which defaults to operator: Exists so
that it runs on every node, tainted ones included. A node without vlagent has
no logs in Seylo.
Images
Section titled “Images”| Key | Default |
|---|---|
metrics.image |
victoriametrics/victoria-metrics:v1.153.0 |
kubeStateMetrics.image |
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 |
logs.image |
victoriametrics/victoria-logs:v1.52.0 |
logs.collector.image |
victoriametrics/vlagent:v1.52.0 |
traces.image |
victoriametrics/victoria-traces:v0.12.0 |
traces.collector.image |
otel/opentelemetry-collector-contrib:0.161.0 |
opencost.image |
ghcr.io/opencost/opencost:1.121.3, pinned by digest |
Each has repository and tag. To pull from a mirror, set repository. Keep
the tags the chart release names. Seylo tests its queries against those
versions.
Scraping
Section titled “Scraping”VictoriaMetrics does its own scraping. There is no separate scraping agent.
| Key | Default | What it does |
|---|---|---|
metrics.scrapeInterval |
30s |
How often every target is scraped |
metrics.scrapeAnnotatedPods |
true |
Also scrape pods that carry the annotations below |
kubeStateMetrics.enabled |
true |
Run kube-state-metrics. Managed vCPU is measured from it. With it off, the cluster shows as not measured, and most dashboard templates are empty |
What is always scraped: each node’s kubelet, its cAdvisor endpoint, kube-state-metrics, and the collector’s own health. Duplicate pod-level series from cAdvisor are dropped, so that a pod’s CPU and memory are not counted twice. Pod network traffic and each node’s own CPU and memory are kept from chart 0.2.5.
What is stored
Section titled “What is stored”The kubelet and cAdvisor expose hundreds of metric families, and most of them describe the kubelet itself. From chart 0.2.8 only the families Seylo, its dashboard templates and its query catalogue read are stored, plus a few that people commonly chart:
- cAdvisor: CPU usage and throttling; memory working set, RSS, cache and usage; out-of-memory events; pod network; filesystem reads, writes, usage and limit.
- kubelet: volume usage, running pod and container counts, and probe results.
Not stored: block-I/O, task-state, thread, socket, file-descriptor, process and spec series, and the kubelet’s own request and storage histograms.
| Key | Default | What it does |
|---|---|---|
metrics.keepAllMetrics |
false |
true stores every family, as before 0.2.8 |
metrics.extraKeepMetrics |
[] |
More families to keep, as names or regular expressions, for example ["container_threads", "kubelet_pleg_.+"] |
storeFlushInterval |
30s |
How often each store saves what it holds in memory to disk. The stores’ own default is 5s; fewer saves mean fewer disk writes |
The scrape annotations Seylo reads
Section titled “The scrape annotations Seylo reads”Put these on the pod template: spec.template.metadata.annotations of a
Deployment, StatefulSet or DaemonSet. On the workload’s own metadata they do
nothing, because the scrape job discovers pods, not workloads.
| Annotation | Required | Meaning |
|---|---|---|
prometheus.io/scrape: "true" |
yes | Scrape this pod. It must be the string "true" |
prometheus.io/port |
in practice, yes | The port serving metrics |
prometheus.io/path |
no | The metrics path. Without it, /metrics |
The scrape is plain HTTP. The job does not read prometheus.io/scheme.
Every series gets namespace and pod from Kubernetes, not from what the pod
reports. It also gets app from the pod’s app.kubernetes.io/name label. The
namespace label decides who may see the series, which is why it is never
taken from the target.
If your application namespace has a NetworkPolicy that limits ingress, it must admit VictoriaMetrics from the chart’s namespace on the metrics port. Otherwise the target is never scraped.
A coding agent with the scrape_target_propose capability can propose these
annotations for a Deployment. Like any change, the proposal waits for approval.
Send traces from your services
Section titled “Send traces from your services”The OpenTelemetry collector accepts OTLP on two ports:
| Protocol | Address |
|---|---|
| gRPC | seylo-otel.seylo-observability.svc:4317 |
| HTTP | http://seylo-otel.seylo-observability.svc:4318 |
It takes traces only. Application logs come from stdout and stderr through vlagent. Metrics come from scraping.
Send traces and APM data shows four ways in, with copyable steps:
- auto-instrumentation with no code;
- an SDK;
- forwarding from a collector you already run;
- eBPF.
It also covers what the collector does not receive (Datadog, Jaeger Thrift, Zipkin) and how to check that spans arrive.
networkPolicy.otelIngressFrom limits which namespaces may send to the
collector. The default is [], which admits every namespace. It takes standard
NetworkPolicy from entries.
Cost allocation with OpenCost
Section titled “Cost allocation with OpenCost”opencost.enabled: true runs OpenCost next to the
stores. It works out what each namespace and workload costs from the data this
chart already collects, and writes the result back to VictoriaMetrics as
metrics. It is off by default.
What it runs, and with what permissions:
- One pod. No DaemonSet and no node-exporter: nothing privileged, and nothing reads the host.
- Read-only access to the workload, node, namespace, volume and job objects it prices. No Secrets. ConfigMaps are readable only in its own namespace. No exec, attach or proxy.
- Its MCP server stays off, and its port admits two callers: VictoriaMetrics, which scrapes it, and — from chart 0.2.11 — the Seylo agent, which asks OpenCost’s API a fixed list of read questions for chargeback. OpenCost’s API has no access control of its own, so the agent narrows every answer to the asking person’s namespaces before it leaves your cluster. Seylo reads the stored metrics through the agent too, scoped like every other metric.
- It needs outbound HTTPS to fetch public list prices: Azure’s Retail Prices API, or AWS’s public price files. No cloud credential is needed for that, and Seylo never holds one.
- kube-state-metrics also exports PersistentVolumes from 0.2.8, because OpenCost prices storage by capacity.
opencost: enabled: true| Key | Default | Notes |
|---|---|---|
opencost.enabled |
false |
Needs metrics.enabled and kubeStateMetrics.enabled; the chart refuses to render otherwise and says why |
opencost.clusterId |
default-cluster |
Written on every series OpenCost exports, as cluster_id |
opencost.image |
see Images | Pinned by digest |
opencost.networkCosts.enabled |
removed | Charts 0.2.10 to 0.2.19 only: a network-costs collector whose image is Kubecost’s, not open source. Leave it off; later charts refuse true. See network traffic |
opencost.azureBilling.priceSheet.*, opencost.azureBilling.cloudCosts.*, opencost.plugins.datadog.*, opencost.plugins.openai.* |
off | Billing integrations, from chart 0.2.11. Each names a Secret you create; no key is ever a chart value. Seylo now offers only the Datadog and OpenAI plugins for new setups; Azure prices come from your Azure bill through a cloud account. See make the figures match your invoice |
Other values
Section titled “Other values”| Key | Default | Notes |
|---|---|---|
namePrefix |
seylo |
Names every Service (seylo-vm, seylo-vl, seylo-vt, seylo-otel, seylo-ksm). The agent looks for these names. If you change the prefix, also set the agent’s observability.metricsUrl, logsUrl and tracesUrl |
agent.namespace, agent.podLabels |
seylo-agent, app.kubernetes.io/name: seylo-agent |
Who the NetworkPolicy admits to the stores |
networkPolicy.enabled |
true |
Turn it off only for a cluster Seylo reaches directly, or use extraIngressFrom |
networkPolicy.extraIngressFrom |
[] |
More from entries admitted to the stores |
logs.collector.excludeOwnNamespace |
true |
Do not collect the chart’s own logs. A collector reading its own output can feed itself on a busy node |
podSecurity.runAsUser, podSecurity.fsGroup |
65534, 65534 |
The non-root user for everything except vlagent |
metrics.enabled, logs.enabled, traces.enabled |
true |
Turn a whole signal off. kube-state-metrics runs only when metrics is on |
What each change sets off
Section titled “What each change sets off”| You change | What happens |
|---|---|
metrics.scrapeInterval, metrics.scrapeAnnotatedPods, metrics.keepAllMetrics, metrics.extraKeepMetrics |
VictoriaMetrics re-reads its scrape file within a minute. No restart |
storeFlushInterval |
All three stores restart. Data on the volumes is kept |
opencost.enabled |
OpenCost starts (or is removed). VictoriaMetrics starts scraping it within a minute |
| Annotations on your own pods | Picked up by discovery. Nothing to do on the chart |
retention.period, a disk cap, a store’s resources or image |
That store’s pod restarts. Data on the volume is kept |
traces.collector.* |
The collector restarts |
logs.collector.* |
vlagent restarts node by node. Its position files survive the restart, so lines are neither lost nor sent twice |
networkPolicy.*, agent.* |
The NetworkPolicies are updated. No restart |
*.persistence.size, storageClassName |
Refused by Kubernetes. See growing a volume |
metrics.enabled, logs.enabled or traces.enabled set to false |
That signal’s components are removed. Its claim is kept until you delete it |