Skip to content

Observability overview

Seylo’s metrics, logs, traces, APM, dashboards and coding-agent queries all read from stores that run inside your cluster. You install them with one Helm chart, seylo-observability. Seylo reads them through the agent when a page or a tool asks, and keeps none of what it reads.

The chart is separate from the agent on purpose. The agent holds nothing and opens no port. A set of stateful stores with their own volumes is a different decision for a platform team, so it is a different helm install. You can turn it down and keep the agent.

Chart seylo-observability version 0.2.22, from oci://ghcr.io/seylo-inc/charts/seylo-observability. Each chart release ships with the agent release of the same number.

Component Runs as What it does Image (default tag)
VictoriaMetrics (seylo-vm) StatefulSet, 1 replica, own volume Stores metrics. It does its own scraping of each node’s kubelet and cAdvisor, kube-state-metrics, and any pods you annotate. It also receives span metrics from the collector victoriametrics/victoria-metrics:v1.153.0
kube-state-metrics (seylo-ksm) Deployment, 1 replica Turns object state into metrics: restarts, replica counts, pod phase, requests and limits. It is limited to the kinds a dashboard needs and does not read Secrets or ConfigMaps registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0
VictoriaLogs (seylo-vl) StatefulSet, 1 replica, own volume Stores container logs victoriametrics/victoria-logs:v1.52.0
vlagent (seylo-vlagent) DaemonSet, one pod per node Reads each container’s log files on the node and adds the pod’s namespace, name, container and labels victoriametrics/vlagent:v1.52.0
VictoriaTraces (seylo-vt) StatefulSet, 1 replica, own volume Stores traces victoriametrics/victoria-traces:v0.12.0
OpenTelemetry collector (seylo-otel) Deployment, 1 replica Receives OTLP traces from your services, adds the pod and namespace, and sends spans to VictoriaTraces. It also counts request rate, errors and latency from every span into VictoriaMetrics for the APM view. From chart 0.2.19 a span’s namespace is the pod it came from; see which namespace a span belongs to otel/opentelemetry-collector-contrib:0.161.0
node-exporter (from chart 0.2.19) DaemonSet, one pod per Linux node, on the host network Each node’s CPU, memory, filesystems, load and network interfaces, for the Hosts dashboard. Not privileged, non-root, host mounts read-only; on by default. See host and ingress metrics quay.io/prometheus/node-exporter:v1.12.1

The chart also creates a NetworkPolicy for each store. Only the Seylo agent and the chart’s own collectors may reach the stores. Your services may reach the OpenTelemetry collector on its OTLP ports.

What the chart does not install: scraping of the Kubernetes API server or etcd, and exporters for databases or runtimes. Up to chart 0.2.18 it also has no node-exporter and does not find ingress-nginx controllers by itself; from 0.2.19 it does both, see host and ingress metrics. The dashboard templates that need something more say what to install. See observability dashboards.

  • Nothing is copied out. Each store keeps its data on its own PersistentVolumeClaim in your cluster. Retention is capped by time and by disk. See configure.
  • Seylo reads it on demand, through the agent. The agent dials out to Seylo as it always does, so there is still no inbound connection to your cluster. When a page needs data, Seylo sends the agent a query. The agent runs it against the store, sends the answer back over the same connection, and drops it.
  • The agent decides where to send a query. The store addresses come from the agent’s own install, never from the Seylo server. Seylo can ask for “metrics” or “logs”, but it cannot point the agent at any other Service in your cluster.
  • Every query is checked twice. Seylo and the agent each check the query against the same fixed list of read-only store endpoints and parameters. A request that tries to set the namespace filter itself is refused, because Seylo adds that filter.
  • Logs are redacted on the way out. Seylo runs its redactor over every message and field, and blanks any field whose name looks like a credential, before a line reaches your browser. The Logs page shows how many redactions it made.

A query read through the agent has these limits:

Limit Value
Response size 4 MiB. A larger answer is cut and marked as cut
Query time at the agent 30 seconds
Log lines per query 1,000
Traces per search 100
Spans read for one trace 5,000
Points per series 11,000

Logs is a log explorer. It opens on the stream: the newest lines across every namespace you may see, from the last hour, newest first. Nothing has to be typed first.

  • Search for a value. Type a word or a value, such as 503 or timeout, and press Enter. It matches the message and every field of a structured line, in any case. You can also type:

    • key:value for an exact field, or key:>=500 to compare;
    • x* for a prefix, *x* for part of a word, -x to leave a word out;
    • namespace:, service:, pod:, container:, node: or level:.

    Switch the box to LogsQL to write a query in the store’s own language.

  • Columns. The list has Time, Namespace, Service, Level and Message columns. Add a column from any field a line carries, or remove one; your choice is kept in this browser. Click Time to change the order.

  • Facets. The panel on the left counts lines by Namespace, Service, Level, Container, Pod and Node for the current search and range. Include or exclude any value; + Add makes a facet from any field. Service is the pod’s app.kubernetes.io/name label, else app, else k8s-app, else the container name, so it stays the same when pods restart. With Source: Rehydrated archive, the counts come from the archive, like the results.

  • Pods that no longer exist. Facets and results come from the stored logs, so lines from deleted or restarted pods are found too, for as long as the chart keeps them.

  • Time range. Pick a preset or a custom From and To. Drag across the volume chart to zoom into that stretch.

  • All the lines, not just the first page. Seylo reads up to 1,000 lines at a time. Load older, or scrolling to the end of the list, reads the next page back in time. The total for the range is shown. One tab holds up to 20,000 lines; past that it asks you to narrow the range.

  • Live: on by default, every 5 seconds, with new lines at the top. It pauses while the browser tab is in the background.

  • One line in detail. Click a line to see its message, its pod, container, node and namespace, and every field. From there, filter by or exclude a value, add it as a column, see the lines around it from the same container (View in context), copy the line, or Open trace when the line carries a trace id.

  • Group into and Visualize as. Group lines by Fields (count by any field), Patterns (lines that differ only in numbers counted together) or Transactions (lines sharing a trace or request id). Show the result as a list, timeseries, top list, bar chart, table, tree map or pie chart.

  • Which namespaces Logs shows by default. An admin opens Logs → Set a default and chooses Hide system namespaces (kube-* and the platform’s own, such as gatekeeper-system, calico-system, tigera-operator, app-routing-system), Only these namespaces, and/or Never these namespaces. Everyone opens Logs with this as a chip. Select × on the chip to see everything for the rest of your session, or Use the organisation default to bring it back. It never shows a namespace you are not allowed to see.

  • Service is the pod’s app.kubernetes.io/name label, else app, else k8s-app, else app.kubernetes.io/instance, else the container name - in the column, the facet and service: search alike.

  • Multi-line JSON. A JSON document your application prints across several lines appears as one row marked JSON · N lines. Open it to see the document as a tree. Untick Join multi-line JSON to see the raw lines.

  • A line with no message. When an application logs JSON with no message or msg key, the list shows the line’s own fields as key=value.

  • Ask AI and Query catalogue are at the right end of the filter bar. See the query catalogue.

The lines you are shown are still limited to your namespaces and redacted, as described above.

Metrics opens on CPU used by namespace for the chosen cluster, so there is a chart before you write a query. Change the query, or start from Ask AI or the Query catalogue at the right of the filter bar.

Seylo stores aggregates and configuration. It does not store raw samples, log lines or spans.

Seylo keeps Where it comes from
One usage summary per container per reporting window: CPU and memory (p50, p95, max, average), network, CPU throttling, restarts, OOM kills, and the container’s requests and limits The agent, which aggregates inside your cluster before it sends anything. See how usage is collected
One managed-vCPU reading per cluster per day kube-state-metrics in the chart. See plans and billing
Dashboard layouts, alert records and uptime results You, and Seylo’s own checks
An audit row for each coding-agent query: the query text (redacted) and the size of the answer Seylo’s MCP server. The answer itself is never stored
Feature Reads Where in Seylo
Metrics, Logs The store for that signal Observability → Metrics, Logs
Traces and APM Traces from the trace store; APM from span metrics in the metrics store, which the collector counts from your traces Observability → Traces & APM: APM lists your services, and each one opens its traces
APM coverage Traces, counted per workload Observability → Traces & APM → Coverage. See APM coverage
Service map Traces (a sample) and mesh metrics, beside what the cluster’s objects say Observability → Service map. See service map
Health signals Your exporters’ metrics, and logs as a fallback A node’s panel on the service map, incidents and Nightshift. See health signals
Follow a request One trace’s spans Follow this request on a log line or a trace. See follow a request
Dashboards Metrics and logs Observability → Dashboards. See observability dashboards
Query catalogue and Ask Metrics and logs Metrics and Logs pages. See the query catalogue
Alert rules over metrics and logs A query, run through the agent on the rule’s interval Alerts → Rules
Usage, right-sizing and managed vCPU The metrics store, including kube-state-metrics FinOps → Cost & savings. See how usage is collected
Coding agents Every signal, through Seylo’s MCP server A deploy token with the observe capability

Pod-failure alerts (CrashLoopBackOff, OOMKilled and the rest) do not use the stores. They come from pod status and Kubernetes events, so they work without the chart. See alerts. Uptime checks run from the Seylo server against public URLs and do not use the chart either.

A coding agent connected with a deploy token can read the stores through Seylo’s MCP server. The read tools are metrics_query, logs_query, traces_search and similar, and they need the observe capability. Two write tools only make proposals. dashboard_propose saves a draft dashboard that someone must publish. scrape_target_propose files an ordinary change request that adds scrape annotations to a Deployment, and it waits for approval.

A deploy token is always limited to one namespace. The store applies that limit, just as it does for a person. Seylo shows nothing through a token until it has proved that the store enforces the filter. See namespace-scoped installs.

Already running Prometheus, Grafana or Datadog?

Section titled “Already running Prometheus, Grafana or Datadog?”

You do not have to start from an empty page, or switch anything off first.

  • Import Grafana dashboards, Prometheus rules, Alertmanager routes, Datadog monitors, dashboards and HTTP tests, and blackbox probes, with a preview and every refusal listed with its reason. Import has no menu entry of its own: the Import button is on Dashboards, Alerts → Rules and Uptime, and Admin → Integrations links to it.
  • Compare with Datadog: read your Datadog in place and prove, with numbers, whether Seylo records the same.