Observability overview
Seylo’s metrics, logs, traces, APM, dashboards and coding-agent queries all read
from stores that run inside your cluster. You install them with one Helm
chart, seylo-observability. Seylo reads them through the agent when a page or
a tool asks, and keeps none of what it reads.
The chart is separate from the agent on purpose. The agent holds nothing and
opens no port. A set of stateful stores with their own volumes is a different
decision for a platform team, so it is a different helm install. You can turn
it down and keep the agent.
What the chart installs
Section titled “What the chart installs”Chart seylo-observability version 0.2.22, from
oci://ghcr.io/seylo-inc/charts/seylo-observability. Each chart release ships
with the agent release of the same number.
| Component | Runs as | What it does | Image (default tag) |
|---|---|---|---|
VictoriaMetrics (seylo-vm) |
StatefulSet, 1 replica, own volume | Stores metrics. It does its own scraping of each node’s kubelet and cAdvisor, kube-state-metrics, and any pods you annotate. It also receives span metrics from the collector | victoriametrics/victoria-metrics:v1.153.0 |
kube-state-metrics (seylo-ksm) |
Deployment, 1 replica | Turns object state into metrics: restarts, replica counts, pod phase, requests and limits. It is limited to the kinds a dashboard needs and does not read Secrets or ConfigMaps | registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 |
VictoriaLogs (seylo-vl) |
StatefulSet, 1 replica, own volume | Stores container logs | victoriametrics/victoria-logs:v1.52.0 |
vlagent (seylo-vlagent) |
DaemonSet, one pod per node | Reads each container’s log files on the node and adds the pod’s namespace, name, container and labels | victoriametrics/vlagent:v1.52.0 |
VictoriaTraces (seylo-vt) |
StatefulSet, 1 replica, own volume | Stores traces | victoriametrics/victoria-traces:v0.12.0 |
OpenTelemetry collector (seylo-otel) |
Deployment, 1 replica | Receives OTLP traces from your services, adds the pod and namespace, and sends spans to VictoriaTraces. It also counts request rate, errors and latency from every span into VictoriaMetrics for the APM view. From chart 0.2.19 a span’s namespace is the pod it came from; see which namespace a span belongs to | otel/opentelemetry-collector-contrib:0.161.0 |
| node-exporter (from chart 0.2.19) | DaemonSet, one pod per Linux node, on the host network | Each node’s CPU, memory, filesystems, load and network interfaces, for the Hosts dashboard. Not privileged, non-root, host mounts read-only; on by default. See host and ingress metrics | quay.io/prometheus/node-exporter:v1.12.1 |
The chart also creates a NetworkPolicy for each store. Only the Seylo agent and the chart’s own collectors may reach the stores. Your services may reach the OpenTelemetry collector on its OTLP ports.
What the chart does not install: scraping of the Kubernetes API server or etcd, and exporters for databases or runtimes. Up to chart 0.2.18 it also has no node-exporter and does not find ingress-nginx controllers by itself; from 0.2.19 it does both, see host and ingress metrics. The dashboard templates that need something more say what to install. See observability dashboards.
Raw telemetry stays in your cluster
Section titled “Raw telemetry stays in your cluster”- Nothing is copied out. Each store keeps its data on its own PersistentVolumeClaim in your cluster. Retention is capped by time and by disk. See configure.
- Seylo reads it on demand, through the agent. The agent dials out to Seylo as it always does, so there is still no inbound connection to your cluster. When a page needs data, Seylo sends the agent a query. The agent runs it against the store, sends the answer back over the same connection, and drops it.
- The agent decides where to send a query. The store addresses come from the agent’s own install, never from the Seylo server. Seylo can ask for “metrics” or “logs”, but it cannot point the agent at any other Service in your cluster.
- Every query is checked twice. Seylo and the agent each check the query against the same fixed list of read-only store endpoints and parameters. A request that tries to set the namespace filter itself is refused, because Seylo adds that filter.
- Logs are redacted on the way out. Seylo runs its redactor over every message and field, and blanks any field whose name looks like a credential, before a line reaches your browser. The Logs page shows how many redactions it made.
A query read through the agent has these limits:
| Limit | Value |
|---|---|
| Response size | 4 MiB. A larger answer is cut and marked as cut |
| Query time at the agent | 30 seconds |
| Log lines per query | 1,000 |
| Traces per search | 100 |
| Spans read for one trace | 5,000 |
| Points per series | 11,000 |
The Logs and Metrics pages
Section titled “The Logs and Metrics pages”Logs is a log explorer. It opens on the stream: the newest lines across every namespace you may see, from the last hour, newest first. Nothing has to be typed first.
-
Search for a value. Type a word or a value, such as
503ortimeout, and press Enter. It matches the message and every field of a structured line, in any case. You can also type:key:valuefor an exact field, orkey:>=500to compare;x*for a prefix,*x*for part of a word,-xto leave a word out;namespace:,service:,pod:,container:,node:orlevel:.
Switch the box to LogsQL to write a query in the store’s own language.
-
Columns. The list has Time, Namespace, Service, Level and Message columns. Add a column from any field a line carries, or remove one; your choice is kept in this browser. Click Time to change the order.
-
Facets. The panel on the left counts lines by Namespace, Service, Level, Container, Pod and Node for the current search and range. Include or exclude any value; + Add makes a facet from any field. Service is the pod’s
app.kubernetes.io/namelabel, elseapp, elsek8s-app, else the container name, so it stays the same when pods restart. With Source: Rehydrated archive, the counts come from the archive, like the results. -
Pods that no longer exist. Facets and results come from the stored logs, so lines from deleted or restarted pods are found too, for as long as the chart keeps them.
-
Time range. Pick a preset or a custom From and To. Drag across the volume chart to zoom into that stretch.
-
All the lines, not just the first page. Seylo reads up to 1,000 lines at a time. Load older, or scrolling to the end of the list, reads the next page back in time. The total for the range is shown. One tab holds up to 20,000 lines; past that it asks you to narrow the range.
-
Live: on by default, every 5 seconds, with new lines at the top. It pauses while the browser tab is in the background.
-
One line in detail. Click a line to see its message, its pod, container, node and namespace, and every field. From there, filter by or exclude a value, add it as a column, see the lines around it from the same container (View in context), copy the line, or Open trace when the line carries a trace id.
-
Group into and Visualize as. Group lines by Fields (count by any field), Patterns (lines that differ only in numbers counted together) or Transactions (lines sharing a trace or request id). Show the result as a list, timeseries, top list, bar chart, table, tree map or pie chart.
-
Which namespaces Logs shows by default. An
adminopens Logs → Set a default and chooses Hide system namespaces (kube-*and the platform’s own, such asgatekeeper-system,calico-system,tigera-operator,app-routing-system), Only these namespaces, and/or Never these namespaces. Everyone opens Logs with this as a chip. Select × on the chip to see everything for the rest of your session, or Use the organisation default to bring it back. It never shows a namespace you are not allowed to see. -
Service is the pod’s
app.kubernetes.io/namelabel, elseapp, elsek8s-app, elseapp.kubernetes.io/instance, else the container name - in the column, the facet andservice:search alike. -
Multi-line JSON. A JSON document your application prints across several lines appears as one row marked JSON · N lines. Open it to see the document as a tree. Untick Join multi-line JSON to see the raw lines.
-
A line with no message. When an application logs JSON with no
messageormsgkey, the list shows the line’s own fields askey=value. -
Ask AI and Query catalogue are at the right end of the filter bar. See the query catalogue.
The lines you are shown are still limited to your namespaces and redacted, as described above.
Metrics opens on CPU used by namespace for the chosen cluster, so there is a chart before you write a query. Change the query, or start from Ask AI or the Query catalogue at the right of the filter bar.
What Seylo stores
Section titled “What Seylo stores”Seylo stores aggregates and configuration. It does not store raw samples, log lines or spans.
| Seylo keeps | Where it comes from |
|---|---|
| One usage summary per container per reporting window: CPU and memory (p50, p95, max, average), network, CPU throttling, restarts, OOM kills, and the container’s requests and limits | The agent, which aggregates inside your cluster before it sends anything. See how usage is collected |
| One managed-vCPU reading per cluster per day | kube-state-metrics in the chart. See plans and billing |
| Dashboard layouts, alert records and uptime results | You, and Seylo’s own checks |
| An audit row for each coding-agent query: the query text (redacted) and the size of the answer | Seylo’s MCP server. The answer itself is never stored |
What uses the stores
Section titled “What uses the stores”| Feature | Reads | Where in Seylo |
|---|---|---|
| Metrics, Logs | The store for that signal | Observability → Metrics, Logs |
| Traces and APM | Traces from the trace store; APM from span metrics in the metrics store, which the collector counts from your traces | Observability → Traces & APM: APM lists your services, and each one opens its traces |
| APM coverage | Traces, counted per workload | Observability → Traces & APM → Coverage. See APM coverage |
| Service map | Traces (a sample) and mesh metrics, beside what the cluster’s objects say | Observability → Service map. See service map |
| Health signals | Your exporters’ metrics, and logs as a fallback | A node’s panel on the service map, incidents and Nightshift. See health signals |
| Follow a request | One trace’s spans | Follow this request on a log line or a trace. See follow a request |
| Dashboards | Metrics and logs | Observability → Dashboards. See observability dashboards |
| Query catalogue and Ask | Metrics and logs | Metrics and Logs pages. See the query catalogue |
| Alert rules over metrics and logs | A query, run through the agent on the rule’s interval | Alerts → Rules |
| Usage, right-sizing and managed vCPU | The metrics store, including kube-state-metrics | FinOps → Cost & savings. See how usage is collected |
| Coding agents | Every signal, through Seylo’s MCP server | A deploy token with the observe capability |
Pod-failure alerts (CrashLoopBackOff, OOMKilled and the rest) do not use
the stores. They come from pod status and Kubernetes events, so they work
without the chart. See alerts. Uptime checks run from
the Seylo server against public URLs and do not use the chart either.
Coding agents
Section titled “Coding agents”A coding agent connected with a deploy token can read the stores through
Seylo’s MCP server. The read tools are metrics_query, logs_query,
traces_search and similar, and they need the observe capability. Two write
tools only make proposals. dashboard_propose saves a draft dashboard that
someone must publish. scrape_target_propose files an ordinary change request
that adds scrape annotations to a Deployment, and it waits for approval.
A deploy token is always limited to one namespace. The store applies that limit, just as it does for a person. Seylo shows nothing through a token until it has proved that the store enforces the filter. See namespace-scoped installs.
Already running Prometheus, Grafana or Datadog?
Section titled “Already running Prometheus, Grafana or Datadog?”You do not have to start from an empty page, or switch anything off first.
- Import Grafana dashboards, Prometheus rules, Alertmanager routes, Datadog monitors, dashboards and HTTP tests, and blackbox probes, with a preview and every refusal listed with its reason. Import has no menu entry of its own: the Import button is on Dashboards, Alerts → Rules and Uptime, and Admin → Integrations links to it.
- Compare with Datadog: read your Datadog in place and prove, with numbers, whether Seylo records the same.
Related
Section titled “Related”- What a coding agent can query
- Right-sizing and usage evidence — built on the usage this chart collects
- Investigate and fix with Nightshift — Seylo’s AI SRE reads these stores as numbered evidence
- Observability dashboards