Query catalogue
Every query here is also on the Metrics and Logs pages: Query catalogue is a button at the right end of
the filter bar, and opens a panel over the results. Press Use and the query runs, then change it in the editor. Each one was run against VictoriaMetrics and VictoriaLogs with data shaped like what the
seylo-observability chart collects, so it parses and it answers.
Prefer plain words? Ask AI, the button beside it, turns a question into one of these queries using the model your organisation chose under Nightshift → Settings: Seylo’s model or your own key (see Nightshift’s model). The model is told metric and field names, never your data, and the query it writes is shown before anything else happens.
How to read and change a query
Section titled “How to read and change a query”$namespaceis a regular expression. The portal fills it from the page’s namespace field:^shop$for one namespace,.*for every namespace you may see. In a dashboard it is the namespace variable. Whatever a query says, you only ever see the namespaces you have access to.by (pod)decides what each line on the chart is. Useby (pod, container)for one line per container,by (namespace)for one per namespace.[5m]is how far backrate()andincrease()look. Longer is smoother, shorter reacts faster.topk(10, ...)keeps the ten largest;bottomkthe ten smallest.- A comparison at the end (
> 0.9) keeps only the rows that cross it: the quickest way to turn a chart into a list of problems. - Logs: a bare word matches the message.
_msg:~"(?i)(a|b)"matches either word anywhere, including inside longer words (OutOfMemoryError);i(word)matches whole words only. Combine filters withand,or,NOTand parentheses. The Logs page adds the namespace and pod you pick.
CPU used, by pod
Section titled “CPU used, by pod”Cores each pod is using, averaged over 5 minutes.
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=~"$namespace", container!=""}[5m]))Unit: cores.
Change it:
- Per container: change
by (pod)toby (pod, container). - Smoother or more jumpy: change
[5m]to[15m]or[1m]. - One workload only: add
pod=~"api-.*"inside the braces.
CPU used as a share of the limit
Section titled “CPU used as a share of the limit”1 means a container is using all the CPU its limit allows; it is throttled from there.
sum by (pod, container) (rate(container_cpu_usage_seconds_total{namespace=~"$namespace", container!=""}[5m])) / sum by (pod, container) (kube_pod_container_resource_limits{namespace=~"$namespace", resource="cpu"})Unit: percentunit.
Change it:
- Against the request instead: change
kube_pod_container_resource_limitstokube_pod_container_resource_requests. - Only the busy ones: append
> 0.8. - A container with no limit has no row: there is nothing to divide by.
CPU throttling
Section titled “CPU throttling”Share of scheduling periods in which a container was held back by its CPU limit.
sum by (pod, container) (rate(container_cpu_cfs_throttled_periods_total{namespace=~"$namespace", container!=""}[5m])) / sum by (pod, container) (rate(container_cpu_cfs_periods_total{namespace=~"$namespace", container!=""}[5m]))Unit: percentunit.
Change it:
- Throttled most of the time: append
> 0.25. - Latency rising with this means the CPU limit is too low, not that the code is slow.
Top 10 containers by CPU
Section titled “Top 10 containers by CPU”The ten containers using the most CPU right now.
topk(10, sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{namespace=~"$namespace", container!=""}[5m])))Unit: cores.
Change it:
- Top 5 or 20: change
10. - Least used instead: change
topktobottomk.
Memory
Section titled “Memory”Memory used, by pod
Section titled “Memory used, by pod”Working set: the memory the kernel counts against the limit when it decides to OOM-kill.
sum by (pod) (container_memory_working_set_bytes{namespace=~"$namespace", container!=""})Unit: bytes.
Change it:
- Per container:
by (pod, container). - Peak over the range instead of now: wrap in
max_over_time(...[$__range])on a dashboard, ormax_over_time(...[1h]).
Memory used as a share of the limit
Section titled “Memory used as a share of the limit”Near 1 means the container is about to be OOM-killed.
sum by (pod, container) (container_memory_working_set_bytes{namespace=~"$namespace", container!=""}) / sum by (pod, container) (kube_pod_container_resource_limits{namespace=~"$namespace", resource="memory"})Unit: percentunit.
Change it:
- Only the risky ones: append
> 0.9. - Against the request: use
kube_pod_container_resource_requests.
Memory requested but not used
Section titled “Memory requested but not used”Bytes each pod reserves beyond what it uses: what right-sizing could hand back.
sum by (pod) (kube_pod_container_resource_requests{namespace=~"$namespace", resource="memory"}) - sum by (pod) (container_memory_working_set_bytes{namespace=~"$namespace", container!=""})Unit: bytes.
Change it:
- Negative means using MORE than requested: the scheduler did not plan for it.
- Biggest first: wrap in
topk(10, ...).
Network
Section titled “Network”A pod’s containers share one network, so traffic is per pod, never per container. These return nothing on
seylo-observability chart 0.2.4 and earlier, which drop the series pod traffic is reported in.
Network received, by pod
Section titled “Network received, by pod”Bytes per second each pod received, averaged over 5 minutes.
sum by (pod) (rate(container_network_receive_bytes_total{namespace=~"$namespace", pod!=""}[5m]))Unit: bytes per second.
Change it:
- Sent instead: change
receivetotransmit. - Per namespace: change
by (pod)toby (namespace). - Dropped packets:
container_network_receive_packets_dropped_total. - A pod on the host network reports the node’s interfaces, not its own.
Network sent, by pod
Section titled “Network sent, by pod”Bytes per second each pod sent, averaged over 5 minutes.
sum by (pod) (rate(container_network_transmit_bytes_total{namespace=~"$namespace", pod!=""}[5m]))Unit: bytes per second.
Change it:
- The busiest ten: wrap in
topk(10, ...). - Errors:
container_network_transmit_errors_total.
Reliability
Section titled “Reliability”Container restarts in the last hour
Section titled “Container restarts in the last hour”How many times each container restarted.
sum by (pod, container) (increase(kube_pod_container_status_restarts_total{namespace=~"$namespace"}[1h])) > 0Change it:
- Over a day: change
[1h]to[24h]. - Every container, including those that did not restart: remove
> 0.
Containers last killed for running out of memory
Section titled “Containers last killed for running out of memory”1 for each container whose last termination was OOMKilled.
sum by (pod, container) (kube_pod_container_status_last_terminated_reason{namespace=~"$namespace", reason="OOMKilled"}) > 0Change it:
- Any other reason: change
OOMKilledtoErrororCompleted. - Pair it with Memory used as a share of the limit to see how close the others are.
Containers stuck waiting, with the reason
Section titled “Containers stuck waiting, with the reason”CrashLoopBackOff, ImagePullBackOff, CreateContainerConfigError and the like.
sum by (pod, container, reason) (kube_pod_container_status_waiting_reason{namespace=~"$namespace"}) > 0Change it:
- One reason: add
reason="CrashLoopBackOff"inside the braces.
Pods that are not ready
Section titled “Pods that are not ready”Pods failing their readiness checks, so they receive no traffic.
sum by (pod) (kube_pod_status_ready{namespace=~"$namespace", condition="false"}) > 0Change it:
- Count per namespace: change
by (pod)toby (namespace).
Pods by phase
Section titled “Pods by phase”How many pods are Running, Pending, Failed, Succeeded or Unknown.
sum by (phase) (kube_pod_status_phase{namespace=~"$namespace"})Change it:
- Only stuck ones: add
phase=~"Pending|Failed|Unknown"inside the braces.
Deployments missing replicas
Section titled “Deployments missing replicas”Replicas wanted minus replicas available, for each Deployment that is short.
sum by (deployment) (kube_deployment_spec_replicas{namespace=~"$namespace"}) - sum by (deployment) (kube_deployment_status_replicas_available{namespace=~"$namespace"}) > 0Change it:
- StatefulSets:
kube_statefulset_replicasminuskube_statefulset_status_replicas_ready,by (statefulset).
Jobs with failed pods
Section titled “Jobs with failed pods”Failed pod count for each Job that has any.
sum by (job_name) (kube_job_status_failed{namespace=~"$namespace"}) > 0Change it:
- CronJobs not run recently:
time() - kube_cronjob_status_last_schedule_time{namespace=~"$namespace"}gives seconds since each last ran.
Autoscalers at their maximum
Section titled “Autoscalers at their maximum”HorizontalPodAutoscalers that cannot add more pods: load beyond this is not absorbed.
sum by (horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas{namespace=~"$namespace"}) >= sum by (horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"$namespace"})Change it:
- Close to the maximum: multiply the right-hand side by 0.8.
Storage
Section titled “Storage”Volume space used
Section titled “Volume space used”Share of each PersistentVolumeClaim that is full.
sum by (persistentvolumeclaim) (kubelet_volume_stats_used_bytes{namespace=~"$namespace"}) / sum by (persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes{namespace=~"$namespace"})Unit: percentunit.
Change it:
- Nearly full: append
> 0.85. - Free bytes:
kubelet_volume_stats_available_bytes. - Some storage drivers do not report volume stats; those claims have no row.
Requests (traces)
Section titled “Requests (traces)”Requests per second, by service
Section titled “Requests per second, by service”Server and consumer spans per second, from the traces your services send.
sum by (service_name) (rate(traces_span_metrics_calls{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))Unit: reqps.
Change it:
- By endpoint: change
by (service_name)toby (service_name, span_name). - Needs services instrumented with OpenTelemetry; nothing traced, nothing here.
Error ratio, by service
Section titled “Error ratio, by service”Share of requests that ended in an error status.
sum by (service_name) (rate(traces_span_metrics_calls{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER", status_code="STATUS_CODE_ERROR"}[5m])) / sum by (service_name) (rate(traces_span_metrics_calls{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))Unit: percentunit.
Change it:
- Over 1%: append
> 0.01.
95th percentile latency, by service
Section titled “95th percentile latency, by service”95% of requests finished faster than this.
histogram_quantile(0.95, sum by (le, service_name) (rate(traces_span_metrics_duration_bucket{namespace=~"$namespace", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m])))Unit: ms.
Change it:
- Median or p99: change
0.95to0.5or0.99. - By endpoint: add
span_nametoby (le, service_name).
Cluster
Section titled “Cluster”CPU requested, as a share of each node
Section titled “CPU requested, as a share of each node”How full each node is from the scheduler’s point of view; near 1, new pods stay Pending. Answers only for people who can see the whole cluster: these series carry no namespace.
sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / sum by (node) (kube_node_status_allocatable{resource="cpu"})Unit: percentunit.
Change it:
- Memory: change
cputomemoryin both places.
Nodes not ready
Section titled “Nodes not ready”Nodes whose Ready condition is not true. Answers only for people who can see the whole cluster: these series carry no namespace.
sum by (node) (kube_node_status_condition{condition="Ready", status="true"}) == 0Change it:
- Memory or disk pressure:
condition="MemoryPressure"or"DiskPressure"withstatus="true"and== 1.
Error lines
Section titled “Error lines”Lines mentioning an error, exception, panic or fatal, including inside words like OutOfMemoryError.
_msg:~"(?i)(error|exception|panic|fatal|traceback)"Change it:
- Add a word to look for: put it inside the brackets, separated by
|. - Exclude noise: append
NOT _msg:~"(?i)healthz".
Timeouts
Section titled “Timeouts”Lines about time-outs and exceeded deadlines.
_msg:~"(?i)(timed? ?out|deadline exceeded)"Change it:
- Only one dependency: append a word, such as
postgres.
Connection failures
Section titled “Connection failures”Refused connections, unreachable hosts and failed DNS lookups.
_msg:~"(?i)(connection refused|econnrefused|no route to host|no such host|connection reset)"Change it:
- Pair with Timeouts to see whether a dependency is down or just slow.
Out of memory
Section titled “Out of memory”Lines from runtimes that report running out of memory (Java, Node, Python, Go).
_msg:~"(?i)(out ?of ?memory|heap out of|memoryerror|cannot allocate memory)"Change it:
- The kill itself is not a log line: use Containers last killed for running out of memory on Metrics.
HTTP 5xx responses
Section titled “HTTP 5xx responses”Access-log lines with a 5xx status, in the common “status 503” / “ 503 “ shapes.
_msg:~"(?i)(status[=: ]+5[0-9][0-9]|\" 5[0-9][0-9] | 5[0-9][0-9] [0-9]+ )"Change it:
- Your services log JSON? Filter the field instead, e.g.
status:>=500. - 4xx: change
5[0-9][0-9]to4[0-9][0-9].
Everything except health checks
Section titled “Everything except health checks”All lines, without the probe and health-check noise.
NOT _msg:~"(?i)(healthz|readyz|livez|/health|kube-probe)"Change it:
- Combine with any filter above by putting both on the line.