Skip to content

Query catalogue

Every query here is also on the Metrics and Logs pages: Query catalogue is a button at the right end of the filter bar, and opens a panel over the results. Press Use and the query runs, then change it in the editor. Each one was run against VictoriaMetrics and VictoriaLogs with data shaped like what the seylo-observability chart collects, so it parses and it answers.

Prefer plain words? Ask AI, the button beside it, turns a question into one of these queries using the model your organisation chose under Nightshift → Settings: Seylo’s model or your own key (see Nightshift’s model). The model is told metric and field names, never your data, and the query it writes is shown before anything else happens.

  • $namespace is a regular expression. The portal fills it from the page’s namespace field: ^shop$ for one namespace, .* for every namespace you may see. In a dashboard it is the namespace variable. Whatever a query says, you only ever see the namespaces you have access to.
  • by (pod) decides what each line on the chart is. Use by (pod, container) for one line per container, by (namespace) for one per namespace.
  • [5m] is how far back rate() and increase() look. Longer is smoother, shorter reacts faster.
  • topk(10, ...) keeps the ten largest; bottomk the ten smallest.
  • A comparison at the end (> 0.9) keeps only the rows that cross it: the quickest way to turn a chart into a list of problems.
  • Logs: a bare word matches the message. _msg:~"(?i)(a|b)" matches either word anywhere, including inside longer words (OutOfMemoryError); i(word) matches whole words only. Combine filters with and, or, NOT and parentheses. The Logs page adds the namespace and pod you pick.

Cores each pod is using, averaged over 5 minutes.

sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=~"$namespace", container!=""}[5m]))

Unit: cores.

Change it:

  • Per container: change by (pod) to by (pod, container).
  • Smoother or more jumpy: change [5m] to [15m] or [1m].
  • One workload only: add pod=~"api-.*" inside the braces.

1 means a container is using all the CPU its limit allows; it is throttled from there.

sum by (pod, container) (rate(container_cpu_usage_seconds_total{namespace=~"$namespace", container!=""}[5m])) / sum by (pod, container) (kube_pod_container_resource_limits{namespace=~"$namespace", resource="cpu"})

Unit: percentunit.

Change it:

  • Against the request instead: change kube_pod_container_resource_limits to kube_pod_container_resource_requests.
  • Only the busy ones: append > 0.8.
  • A container with no limit has no row: there is nothing to divide by.

Share of scheduling periods in which a container was held back by its CPU limit.

sum by (pod, container) (rate(container_cpu_cfs_throttled_periods_total{namespace=~"$namespace", container!=""}[5m])) / sum by (pod, container) (rate(container_cpu_cfs_periods_total{namespace=~"$namespace", container!=""}[5m]))

Unit: percentunit.

Change it:

  • Throttled most of the time: append > 0.25.
  • Latency rising with this means the CPU limit is too low, not that the code is slow.

The ten containers using the most CPU right now.

topk(10, sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{namespace=~"$namespace", container!=""}[5m])))

Unit: cores.

Change it:

  • Top 5 or 20: change 10.
  • Least used instead: change topk to bottomk.

Working set: the memory the kernel counts against the limit when it decides to OOM-kill.

sum by (pod) (container_memory_working_set_bytes{namespace=~"$namespace", container!=""})

Unit: bytes.

Change it:

  • Per container: by (pod, container).
  • Peak over the range instead of now: wrap in max_over_time(...[$__range]) on a dashboard, or max_over_time(...[1h]).

Near 1 means the container is about to be OOM-killed.

sum by (pod, container) (container_memory_working_set_bytes{namespace=~"$namespace", container!=""}) / sum by (pod, container) (kube_pod_container_resource_limits{namespace=~"$namespace", resource="memory"})

Unit: percentunit.

Change it:

  • Only the risky ones: append > 0.9.
  • Against the request: use kube_pod_container_resource_requests.

Bytes each pod reserves beyond what it uses: what right-sizing could hand back.

sum by (pod) (kube_pod_container_resource_requests{namespace=~"$namespace", resource="memory"}) - sum by (pod) (container_memory_working_set_bytes{namespace=~"$namespace", container!=""})

Unit: bytes.

Change it:

  • Negative means using MORE than requested: the scheduler did not plan for it.
  • Biggest first: wrap in topk(10, ...).

A pod’s containers share one network, so traffic is per pod, never per container. These return nothing on seylo-observability chart 0.2.4 and earlier, which drop the series pod traffic is reported in.

Bytes per second each pod received, averaged over 5 minutes.

sum by (pod) (rate(container_network_receive_bytes_total{namespace=~"$namespace", pod!=""}[5m]))

Unit: bytes per second.

Change it:

  • Sent instead: change receive to transmit.
  • Per namespace: change by (pod) to by (namespace).
  • Dropped packets: container_network_receive_packets_dropped_total.
  • A pod on the host network reports the node’s interfaces, not its own.

Bytes per second each pod sent, averaged over 5 minutes.

sum by (pod) (rate(container_network_transmit_bytes_total{namespace=~"$namespace", pod!=""}[5m]))

Unit: bytes per second.

Change it:

  • The busiest ten: wrap in topk(10, ...).
  • Errors: container_network_transmit_errors_total.

How many times each container restarted.

sum by (pod, container) (increase(kube_pod_container_status_restarts_total{namespace=~"$namespace"}[1h])) > 0

Change it:

  • Over a day: change [1h] to [24h].
  • Every container, including those that did not restart: remove > 0.

Containers last killed for running out of memory

Section titled “Containers last killed for running out of memory”

1 for each container whose last termination was OOMKilled.

sum by (pod, container) (kube_pod_container_status_last_terminated_reason{namespace=~"$namespace", reason="OOMKilled"}) > 0

Change it:

  • Any other reason: change OOMKilled to Error or Completed.
  • Pair it with Memory used as a share of the limit to see how close the others are.

CrashLoopBackOff, ImagePullBackOff, CreateContainerConfigError and the like.

sum by (pod, container, reason) (kube_pod_container_status_waiting_reason{namespace=~"$namespace"}) > 0

Change it:

  • One reason: add reason="CrashLoopBackOff" inside the braces.

Pods failing their readiness checks, so they receive no traffic.

sum by (pod) (kube_pod_status_ready{namespace=~"$namespace", condition="false"}) > 0

Change it:

  • Count per namespace: change by (pod) to by (namespace).

How many pods are Running, Pending, Failed, Succeeded or Unknown.

sum by (phase) (kube_pod_status_phase{namespace=~"$namespace"})

Change it:

  • Only stuck ones: add phase=~"Pending|Failed|Unknown" inside the braces.

Replicas wanted minus replicas available, for each Deployment that is short.

sum by (deployment) (kube_deployment_spec_replicas{namespace=~"$namespace"}) - sum by (deployment) (kube_deployment_status_replicas_available{namespace=~"$namespace"}) > 0

Change it:

  • StatefulSets: kube_statefulset_replicas minus kube_statefulset_status_replicas_ready, by (statefulset).

Failed pod count for each Job that has any.

sum by (job_name) (kube_job_status_failed{namespace=~"$namespace"}) > 0

Change it:

  • CronJobs not run recently: time() - kube_cronjob_status_last_schedule_time{namespace=~"$namespace"} gives seconds since each last ran.

HorizontalPodAutoscalers that cannot add more pods: load beyond this is not absorbed.

sum by (horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas{namespace=~"$namespace"}) >= sum by (horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"$namespace"})

Change it:

  • Close to the maximum: multiply the right-hand side by 0.8.

Share of each PersistentVolumeClaim that is full.

sum by (persistentvolumeclaim) (kubelet_volume_stats_used_bytes{namespace=~"$namespace"}) / sum by (persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes{namespace=~"$namespace"})

Unit: percentunit.

Change it:

  • Nearly full: append > 0.85.
  • Free bytes: kubelet_volume_stats_available_bytes.
  • Some storage drivers do not report volume stats; those claims have no row.

Server and consumer spans per second, from the traces your services send.

sum by (service_name) (rate(traces_span_metrics_calls{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

Unit: reqps.

Change it:

  • By endpoint: change by (service_name) to by (service_name, span_name).
  • Needs services instrumented with OpenTelemetry; nothing traced, nothing here.

Share of requests that ended in an error status.

sum by (service_name) (rate(traces_span_metrics_calls{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER", status_code="STATUS_CODE_ERROR"}[5m])) / sum by (service_name) (rate(traces_span_metrics_calls{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

Unit: percentunit.

Change it:

  • Over 1%: append > 0.01.

95% of requests finished faster than this.

histogram_quantile(0.95, sum by (le, service_name) (rate(traces_span_metrics_duration_bucket{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m])))

Unit: ms.

Change it:

  • Median or p99: change 0.95 to 0.5 or 0.99.
  • By endpoint: add span_name to by (le, service_name).

How full each node is from the scheduler’s point of view; near 1, new pods stay Pending. Answers only for people who can see the whole cluster: these series carry no namespace.

sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / sum by (node) (kube_node_status_allocatable{resource="cpu"})

Unit: percentunit.

Change it:

  • Memory: change cpu to memory in both places.

Nodes whose Ready condition is not true. Answers only for people who can see the whole cluster: these series carry no namespace.

sum by (node) (kube_node_status_condition{condition="Ready", status="true"}) == 0

Change it:

  • Memory or disk pressure: condition="MemoryPressure" or "DiskPressure" with status="true" and == 1.

Lines mentioning an error, exception, panic or fatal, including inside words like OutOfMemoryError.

_msg:~"(?i)(error|exception|panic|fatal|traceback)"

Change it:

  • Add a word to look for: put it inside the brackets, separated by |.
  • Exclude noise: append NOT _msg:~"(?i)healthz".

Lines about time-outs and exceeded deadlines.

_msg:~"(?i)(timed? ?out|deadline exceeded)"

Change it:

  • Only one dependency: append a word, such as postgres.

Refused connections, unreachable hosts and failed DNS lookups.

_msg:~"(?i)(connection refused|econnrefused|no route to host|no such host|connection reset)"

Change it:

  • Pair with Timeouts to see whether a dependency is down or just slow.

Lines from runtimes that report running out of memory (Java, Node, Python, Go).

_msg:~"(?i)(out ?of ?memory|heap out of|memoryerror|cannot allocate memory)"

Change it:

  • The kill itself is not a log line: use Containers last killed for running out of memory on Metrics.

Access-log lines with a 5xx status, in the common “status 503” / “ 503 “ shapes.

_msg:~"(?i)(status[=: ]+5[0-9][0-9]|\" 5[0-9][0-9] | 5[0-9][0-9] [0-9]+ )"

Change it:

  • Your services log JSON? Filter the field instead, e.g. status:>=500.
  • 4xx: change 5[0-9][0-9] to 4[0-9][0-9].

All lines, without the probe and health-check noise.

NOT _msg:~"(?i)(healthz|readyz|livez|/health|kube-probe)"

Change it:

  • Combine with any filter above by putting both on the line.