Skip to content

Teach Nightshift your runbooks

A skill is one of your organisation’s runbooks, written so that Nightshift can use it while it investigates an alert. Nightshift reads your cluster the same way with or without skills. What a skill adds is what your team knows: what a log line means here, which setting is usually at fault, and what not to do.

Skills live on Nightshift → Skills. You can write them there, or keep them in a Git repository and let Seylo read them. Below your skills, the same page lists Seylo’s core runbooks: the runbooks Nightshift ships with, the same for every organisation.

Action Role
Read your skills and Seylo’s core runbooks; see which ones an investigation used viewer
Write, edit and delete skills; connect, sync and disconnect the repository; disable or enable a core runbook admin

Skills use the open Agent Skills format, the same one HolmesGPT’s skill repositories use, so a skill you already have for another tool usually works here unchanged. A skill is a directory:

checkout-timeouts/
SKILL.md required: frontmatter, then the runbook in Markdown
references/ optional: more Markdown, loaded only when asked for
pool-sizing.md
scripts/ listed by name only; never stored, never run
---
name: checkout-timeouts
description: Explains checkout-api requests that time out while its pods look healthy. Use when checkout-api returns 504s or its p95 latency jumps after a deploy.
metadata:
seylo.clusters: "prod-*"
seylo.namespaces: "shop"
seylo.alerts: "ProbeFailure, HighLatency*"
owner: payments-team
allowed-tools: pods describe logs events metrics_query logs_query
---
## Goal
Explain why checkout-api times out while its pods are Running and Ready.
## Workflow
1. Query checkout-api's p95 latency per route over the alert window.
2. Look in its logs for "connection pool exhausted".
3. Read ConfigMap checkout-settings (its keys) and the Deployment's envFrom.
## Synthesize Findings
"connection pool exhausted" with healthy pods means DB_POOL_SIZE in ConfigMap
checkout-settings is lower than the replica count needs.
## Recommended Remediation Steps
- A person raises DB_POOL_SIZE in ConfigMap checkout-settings, then restarts checkout-api.
- Do NOT scale checkout-api up first: each new replica opens more connections.
- Do NOT roll back: the image is not the cause.

Any Markdown works in the body. The four sections above - Goal, Workflow, Synthesize Findings, Recommended Remediation Steps - are what the editor’s template starts from.

Field Required Rule
name Yes in the editor. In Git, the directory’s name when it is left out 1 to 64 characters: a-z, 0-9 and single hyphens, not starting or ending with one. It must equal the directory’s name
description Yes 1 to 1,024 characters: what the skill does and when to use it. This is what Nightshift chooses by, so say when it applies
license, compatibility No Up to 500 characters each
metadata No Up to 32 keys, each a string. Keys starting seylo. set where the skill applies (below)
allowed-tools No Space-separated names of Nightshift’s reads. It can only narrow what Nightshift reads (below)

Every frontmatter value is read as text: version: 1.0 stays "1.0". Any other top-level field is ignored with a warning. An unknown seylo. key is refused, because a mistyped scope key would otherwise make the skill apply everywhere.

Limit
SKILL.md, frontmatter included 64 KiB
Frontmatter 8 KiB
Body 20,000 characters. A body over 8,000 characters is accepted with a warning: one load gives Nightshift the first 8,000
Reference files: references/*.md, .markdown or .txt, one level deep 8 files, 16 KiB each
Skills per organisation, written and synced together 200

Five metadata keys decide which alerts a skill is offered for. A key you leave out matches everything. Where a skill sets several keys, all of them must match; within one key, any listed value may match.

Key Value Example
seylo.clusters Cluster IDs, comma-separated; * and ? are wildcards "prod-*, staging-eu"
seylo.namespaces Namespaces, comma-separated; wildcards "shop, shop-*"
seylo.labels The workload’s labels: k=v, k!=v, k (the key exists), !k (the key is absent), comma-separated; all must hold "app=checkout-api, tier!=batch"
seylo.alerts The alert’s reason or rule name; wildcards; case does not matter "CrashLoopBackOff, HighLatency*"
seylo.services Services on the service map: a name, namespace/name, a database’s or cache’s host name, or the name an architecture file gives the workload; wildcards; case does not matter "payment, ledger-db, shop/checkout-api"

Labels are taken from the Deployment and its pod template, as Nightshift read them in its opening steps. If the labels could not be read, a skill scoped by labels does not match: an unknown is never treated as a match.

A skill scoped to a service matches the alert’s own workload on the map, by any of its names - so seylo.services: payment matches the workload payment-api that your Backstage catalog calls payment. While the cluster’s map is not built, or cannot be read, a service-scoped skill does not match. In an incident, Nightshift lists the skills that match any service in the incident, the root’s first, and says which service each matched.

  1. Open Nightshift → Skills and choose New skill.
  2. Write or paste the whole SKILL.md, frontmatter included. The editor starts from a template.
  3. Seylo checks it as you type, with the same rules Save applies, so the errors you see are the ones saving would return. Save stays disabled until the skill is valid.

Deleting a skill takes effect from the next investigation. Investigations that already cited it keep their record.

A draft from memory. When a fix keeps working, Suggest a skill on Nightshift → Memory opens a draft here, unsaved, for you to review and save.

Each organisation can connect one GitHub repository. It is read through Seylo’s GitHub App, so the repository must be in one of your organisation’s installations of the App (Admin → Integrations → GitHub); a repository outside them is refused before any file is read.

  1. Under Skills, choose Connect a repository.
  2. Fill in Repository (owner/name, for example acme/runbooks), Directory (skills if you leave it empty) and, optionally, Branch (the repository’s default branch if you leave it empty).
  3. Put each skill at <directory>/<skill>/SKILL.md, or one level deeper at <directory>/<group>/<skill>/SKILL.md. A SKILL.md inside another skill’s directory belongs to that skill.

Read-only, one repository at a time. Each sync uses a token for that one repository that can only read its contents. It cannot write, cannot see any other repository, and is never stored.

When it syncs: every 5 minutes, and when an admin presses Sync now (at most once every 30 seconds). If the commit has not changed, nothing is read again.

What happened Result
The read failed: a GitHub error, no installation, a missing path Nothing changes. The skills from the last good commit stay in use, and the status line says why the sync failed
A skill that was valid is now invalid Its last good version stays in use. The status line names the skill, the reason, and the commit still in use
A skill’s directory is gone The skill is deleted
The same text appears under a new name Reported as a rename
Two directories have the same skill name The first by path is used; the other is refused
The name belongs to a skill written in Seylo The Git skill is refused. Git and the editor never overwrite each other
A file is over its size limit The skill is refused without downloading the file

Skills from Git are read-only in Seylo: change them in the repository, and the next sync brings the change in. Change points Skills at another repository, directory or branch; Disconnect removes the synced skills from Nightshift at once. Either way, skills written in Seylo stay, and nothing in the repository changes.

  1. After its opening reads, Nightshift matches your skills against the alert and sees a short index: each matching skill’s name, its description (up to 300 characters), its scope and its reference files. The most specific scopes come first, then up to 6 matching core runbooks. At most 12 entries and 4,800 characters are listed, and Nightshift is told how many more matched.
  2. It loads the ones that fit, by name, or one of a skill’s reference files. It can load only skills from the index, at most 3 loads per run, and each load counts toward the run’s 18 reads.
  3. The text becomes one numbered piece of evidence, labelled as guidance your organisation wrote, not a fact read from the cluster. It is redacted like all evidence. Findings cite it by number, the same way they cite a log line.
  4. The investigation records which skills it used, and from where. The panel under the alert lists them under What Nightshift read: Organisation skills used: checkout-timeouts ... A skill steers what Nightshift reads; it never adds an action. When a core runbook was used too, the line starts Skills and Seylo core runbooks used:. The answer itself ends with Guided by: and the names of the skills and runbooks that shaped it.

A run uses your skills as they were when it started, so its record can be read back later exactly as it happened.

Nightshift ships with 45 core runbooks, one per kind of failure, the same for every organisation, cluster and region. Each is written in the order an SRE works an outage: Mitigate now, Diagnose, Fix, Prevent. They use the same SKILL.md format as your skills, and Nightshift reads them the same way: as guidance it loads and cites, never as an action.

Area Runbooks
Crashes and start-up crashloop-backoff, crashloop-after-deploy, config-missing-key, liveness-kills-slow-start, liveness-probe-heavy, readiness-probe-failing, oom-killed, memory-leak, cpu-throttling, noisy-neighbour
Images image-pull-tag-missing, image-pull-auth, image-name-invalid, image-never-pull
Rollouts and configuration deployment-rollout-stuck, statefulset-rollout-stuck, old-replicasets-pile-up, configmap-not-picked-up, secret-not-picked-up, limitrange-rejects-pods, resource-quota-exceeded, job-failing
Scheduling, nodes and storage pending-insufficient-resources, pending-capacity-buffer, pending-untolerated-taint, pending-unbound-pvc, failed-mount, pvc-nearly-full, pods-evicted, pods-stuck-terminating, node-not-ready
Scaling and traffic hpa-at-max, hpa-missing-requests, hot-path-overload, service-no-endpoints, network-policy-blocks, dns-resolution-failing, tls-certificate-expiry
Dependencies connection-pool-exhausted, db-connections-saturated, replication-lag, queue-backlog-stuck-consumer, retry-storm-circuit-open, upstream-failure-cascade
Seylo’s own stores seylo-store-unhealthy

Every number a core runbook states comes from a Kubernetes or component default, or from a setting in Seylo, never an estimate. Read on Nightshift → Skills → Seylo core runbooks shows any of them, with its version.

Your skills come first. An investigation lists your matching skills, then at most 6 matching core runbooks, ranked by the failure Seylo’s own rules recognised. A core runbook for the same alert as one of your listed skills is marked superseded by your skill; Nightshift can still load it for what your skill does not cover. In an incident, the incident runbook (upstream-failure-cascade) is listed first.

Replace one for your scope by naming it in your skill:

metadata:
seylo.overrides: "oom-killed, memory-leak"

Turn one off. An admin presses Disable for scope beside a runbook and chooses the organisation, a cluster, or one namespace of a cluster. A disabled runbook is neither listed nor loadable there; the Offered column then reads Disabled: and the scope, with Enable beside it. Your own skills are unaffected, and investigations that already cited it keep their record. Both are recorded in the audit log.

Some seylo. keys, such as seylo.core, seylo.version and seylo.signals, belong to Seylo’s runbooks only: a skill of yours that carries one is refused.

These limits are enforced by Seylo, not left to the model’s good behaviour.

  • Run anything. Files in scripts/, and any file other than SKILL.md and references/, are ignored. Their names are listed for the author; their contents are never stored, never run and never offered to Nightshift.
  • Add a fix, a target, a namespace or autonomy - and neither can a core runbook. Nightshift’s list of proposed actions is the same with or without skills, and every proposal is checked against what the run actually read. Standing grants, the per-class fix settings and automatic investigation do not read skills. A skill that says “restart it” can lead Nightshift to propose a restart; whether that restart waits for a person is decided exactly as it would be without the skill.
  • Widen what Nightshift reads. allowed-tools is matched against Nightshift’s reads and can only remove some of them. A second skill narrows further. A name that is not one of Nightshift’s reads, such as Bash(kubectl:*) or an action name, grants nothing. Once a narrowing skill is loaded, a read outside its list is refused: Skill checkout-timeouts limits this run's reads to ... (its allowed-tools). Use one of those, or answer from the evidence.
  • Reach outside the alert’s namespace. Skills change none of the checks on Nightshift’s reads and queries.
  • Carry a credential. Every line of the frontmatter, the body and each reference file is checked for credentials. A skill with one is refused, naming the line and the kind of value, for example DB_PASSWORD (12 characters) - never the value. Placeholders are fine: <db password>, ${DB_PASSWORD}, {{ .Values.password }}, ***, REDACTED, xxxx.

Every change to a skill or to the repository setting is recorded in the audit log, by the skill’s name, not its text.