Teach Nightshift your runbooks
A skill is one of your organisation’s runbooks, written so that Nightshift can use it while it investigates an alert. Nightshift reads your cluster the same way with or without skills. What a skill adds is what your team knows: what a log line means here, which setting is usually at fault, and what not to do.
Skills live on Nightshift → Skills. You can write them there, or keep them in a Git repository and let Seylo read them. Below your skills, the same page lists Seylo’s core runbooks: the runbooks Nightshift ships with, the same for every organisation.
| Action | Role |
|---|---|
| Read your skills and Seylo’s core runbooks; see which ones an investigation used | viewer |
| Write, edit and delete skills; connect, sync and disconnect the repository; disable or enable a core runbook | admin |
The format
Section titled “The format”Skills use the open Agent Skills format, the same one HolmesGPT’s skill repositories use, so a skill you already have for another tool usually works here unchanged. A skill is a directory:
checkout-timeouts/ SKILL.md required: frontmatter, then the runbook in Markdown references/ optional: more Markdown, loaded only when asked for pool-sizing.md scripts/ listed by name only; never stored, never runAn example
Section titled “An example”---name: checkout-timeoutsdescription: Explains checkout-api requests that time out while its pods look healthy. Use when checkout-api returns 504s or its p95 latency jumps after a deploy.metadata: seylo.clusters: "prod-*" seylo.namespaces: "shop" seylo.alerts: "ProbeFailure, HighLatency*" owner: payments-teamallowed-tools: pods describe logs events metrics_query logs_query---## GoalExplain why checkout-api times out while its pods are Running and Ready.
## Workflow1. Query checkout-api's p95 latency per route over the alert window.2. Look in its logs for "connection pool exhausted".3. Read ConfigMap checkout-settings (its keys) and the Deployment's envFrom.
## Synthesize Findings"connection pool exhausted" with healthy pods means DB_POOL_SIZE in ConfigMapcheckout-settings is lower than the replica count needs.
## Recommended Remediation Steps- A person raises DB_POOL_SIZE in ConfigMap checkout-settings, then restarts checkout-api.- Do NOT scale checkout-api up first: each new replica opens more connections.- Do NOT roll back: the image is not the cause.Any Markdown works in the body. The four sections above - Goal, Workflow, Synthesize Findings, Recommended Remediation Steps - are what the editor’s template starts from.
Frontmatter
Section titled “Frontmatter”| Field | Required | Rule |
|---|---|---|
name |
Yes in the editor. In Git, the directory’s name when it is left out | 1 to 64 characters: a-z, 0-9 and single hyphens, not starting or ending with one. It must equal the directory’s name |
description |
Yes | 1 to 1,024 characters: what the skill does and when to use it. This is what Nightshift chooses by, so say when it applies |
license, compatibility |
No | Up to 500 characters each |
metadata |
No | Up to 32 keys, each a string. Keys starting seylo. set where the skill applies (below) |
allowed-tools |
No | Space-separated names of Nightshift’s reads. It can only narrow what Nightshift reads (below) |
Every frontmatter value is read as text: version: 1.0 stays "1.0". Any
other top-level field is ignored with a warning. An unknown seylo. key is
refused, because a mistyped scope key would otherwise make the skill apply
everywhere.
Limits
Section titled “Limits”| Limit | |
|---|---|
SKILL.md, frontmatter included |
64 KiB |
| Frontmatter | 8 KiB |
| Body | 20,000 characters. A body over 8,000 characters is accepted with a warning: one load gives Nightshift the first 8,000 |
Reference files: references/*.md, .markdown or .txt, one level deep |
8 files, 16 KiB each |
| Skills per organisation, written and synced together | 200 |
Where a skill applies
Section titled “Where a skill applies”Five metadata keys decide which alerts a skill is offered for. A key you
leave out matches everything. Where a skill sets several keys, all of them
must match; within one key, any listed value may match.
| Key | Value | Example |
|---|---|---|
seylo.clusters |
Cluster IDs, comma-separated; * and ? are wildcards |
"prod-*, staging-eu" |
seylo.namespaces |
Namespaces, comma-separated; wildcards | "shop, shop-*" |
seylo.labels |
The workload’s labels: k=v, k!=v, k (the key exists), !k (the key is absent), comma-separated; all must hold |
"app=checkout-api, tier!=batch" |
seylo.alerts |
The alert’s reason or rule name; wildcards; case does not matter | "CrashLoopBackOff, HighLatency*" |
seylo.services |
Services on the service map: a name, namespace/name, a database’s or cache’s host name, or the name an architecture file gives the workload; wildcards; case does not matter |
"payment, ledger-db, shop/checkout-api" |
Labels are taken from the Deployment and its pod template, as Nightshift read them in its opening steps. If the labels could not be read, a skill scoped by labels does not match: an unknown is never treated as a match.
A skill scoped to a service matches the alert’s own workload on the map,
by any of its names - so seylo.services: payment matches the workload
payment-api that your Backstage catalog calls payment. While the cluster’s
map is not built, or cannot be read, a service-scoped skill does not match. In
an incident, Nightshift lists the skills that match
any service in the incident, the root’s first, and says which service each
matched.
Write a skill in Seylo
Section titled “Write a skill in Seylo”- Open Nightshift → Skills and choose New skill.
- Write or paste the whole
SKILL.md, frontmatter included. The editor starts from a template. - Seylo checks it as you type, with the same rules Save applies, so the errors you see are the ones saving would return. Save stays disabled until the skill is valid.
Deleting a skill takes effect from the next investigation. Investigations that already cited it keep their record.
A draft from memory. When a fix keeps working, Suggest a skill on Nightshift → Memory opens a draft here, unsaved, for you to review and save.
Sync skills from your Git repository
Section titled “Sync skills from your Git repository”Each organisation can connect one GitHub repository. It is read through Seylo’s GitHub App, so the repository must be in one of your organisation’s installations of the App (Admin → Integrations → GitHub); a repository outside them is refused before any file is read.
- Under Skills, choose Connect a repository.
- Fill in Repository (
owner/name, for exampleacme/runbooks), Directory (skillsif you leave it empty) and, optionally, Branch (the repository’s default branch if you leave it empty). - Put each skill at
<directory>/<skill>/SKILL.md, or one level deeper at<directory>/<group>/<skill>/SKILL.md. ASKILL.mdinside another skill’s directory belongs to that skill.
Read-only, one repository at a time. Each sync uses a token for that one repository that can only read its contents. It cannot write, cannot see any other repository, and is never stored.
When it syncs: every 5 minutes, and when an admin presses Sync now
(at most once every 30 seconds). If the commit has not changed, nothing is read
again.
| What happened | Result |
|---|---|
| The read failed: a GitHub error, no installation, a missing path | Nothing changes. The skills from the last good commit stay in use, and the status line says why the sync failed |
| A skill that was valid is now invalid | Its last good version stays in use. The status line names the skill, the reason, and the commit still in use |
| A skill’s directory is gone | The skill is deleted |
| The same text appears under a new name | Reported as a rename |
| Two directories have the same skill name | The first by path is used; the other is refused |
| The name belongs to a skill written in Seylo | The Git skill is refused. Git and the editor never overwrite each other |
| A file is over its size limit | The skill is refused without downloading the file |
Skills from Git are read-only in Seylo: change them in the repository, and the next sync brings the change in. Change points Skills at another repository, directory or branch; Disconnect removes the synced skills from Nightshift at once. Either way, skills written in Seylo stay, and nothing in the repository changes.
How Nightshift uses a skill
Section titled “How Nightshift uses a skill”- After its opening reads, Nightshift matches your skills against the alert and sees a short index: each matching skill’s name, its description (up to 300 characters), its scope and its reference files. The most specific scopes come first, then up to 6 matching core runbooks. At most 12 entries and 4,800 characters are listed, and Nightshift is told how many more matched.
- It loads the ones that fit, by name, or one of a skill’s reference files. It can load only skills from the index, at most 3 loads per run, and each load counts toward the run’s 18 reads.
- The text becomes one numbered piece of evidence, labelled as guidance your organisation wrote, not a fact read from the cluster. It is redacted like all evidence. Findings cite it by number, the same way they cite a log line.
- The investigation records which skills it used, and from where. The
panel under the alert lists them under What Nightshift read:
Organisation skills used: checkout-timeouts ... A skill steers what Nightshift reads; it never adds an action.When a core runbook was used too, the line startsSkills and Seylo core runbooks used:. The answer itself ends withGuided by:and the names of the skills and runbooks that shaped it.
A run uses your skills as they were when it started, so its record can be read back later exactly as it happened.
Seylo’s core runbooks
Section titled “Seylo’s core runbooks”Nightshift ships with 45 core runbooks, one per kind of failure, the same
for every organisation, cluster and region. Each is written in the order an
SRE works an outage: Mitigate now, Diagnose, Fix, Prevent. They
use the same SKILL.md format as your skills, and Nightshift reads them the
same way: as guidance it loads and cites, never as an action.
| Area | Runbooks |
|---|---|
| Crashes and start-up | crashloop-backoff, crashloop-after-deploy, config-missing-key, liveness-kills-slow-start, liveness-probe-heavy, readiness-probe-failing, oom-killed, memory-leak, cpu-throttling, noisy-neighbour |
| Images | image-pull-tag-missing, image-pull-auth, image-name-invalid, image-never-pull |
| Rollouts and configuration | deployment-rollout-stuck, statefulset-rollout-stuck, old-replicasets-pile-up, configmap-not-picked-up, secret-not-picked-up, limitrange-rejects-pods, resource-quota-exceeded, job-failing |
| Scheduling, nodes and storage | pending-insufficient-resources, pending-capacity-buffer, pending-untolerated-taint, pending-unbound-pvc, failed-mount, pvc-nearly-full, pods-evicted, pods-stuck-terminating, node-not-ready |
| Scaling and traffic | hpa-at-max, hpa-missing-requests, hot-path-overload, service-no-endpoints, network-policy-blocks, dns-resolution-failing, tls-certificate-expiry |
| Dependencies | connection-pool-exhausted, db-connections-saturated, replication-lag, queue-backlog-stuck-consumer, retry-storm-circuit-open, upstream-failure-cascade |
| Seylo’s own stores | seylo-store-unhealthy |
Every number a core runbook states comes from a Kubernetes or component default, or from a setting in Seylo, never an estimate. Read on Nightshift → Skills → Seylo core runbooks shows any of them, with its version.
Your skills come first. An investigation lists your matching skills, then
at most 6 matching core runbooks, ranked by the failure Seylo’s own rules
recognised. A core runbook for the same alert as one of your listed skills is
marked superseded by your skill; Nightshift can still load it for what
your skill does not cover. In an incident, the incident
runbook (upstream-failure-cascade) is listed first.
Replace one for your scope by naming it in your skill:
metadata: seylo.overrides: "oom-killed, memory-leak"Turn one off. An admin presses Disable for scope beside a runbook
and chooses the organisation, a cluster, or one namespace of a cluster. A
disabled runbook is neither listed nor loadable there; the Offered column
then reads Disabled: and the scope, with Enable beside it. Your own
skills are unaffected, and investigations that already cited it keep their
record. Both are recorded in the audit log.
Some seylo. keys, such as seylo.core, seylo.version and
seylo.signals, belong to Seylo’s runbooks only: a skill of yours that carries
one is refused.
What a skill can never do
Section titled “What a skill can never do”These limits are enforced by Seylo, not left to the model’s good behaviour.
- Run anything. Files in
scripts/, and any file other thanSKILL.mdandreferences/, are ignored. Their names are listed for the author; their contents are never stored, never run and never offered to Nightshift. - Add a fix, a target, a namespace or autonomy - and neither can a core runbook. Nightshift’s list of proposed actions is the same with or without skills, and every proposal is checked against what the run actually read. Standing grants, the per-class fix settings and automatic investigation do not read skills. A skill that says “restart it” can lead Nightshift to propose a restart; whether that restart waits for a person is decided exactly as it would be without the skill.
- Widen what Nightshift reads.
allowed-toolsis matched against Nightshift’s reads and can only remove some of them. A second skill narrows further. A name that is not one of Nightshift’s reads, such asBash(kubectl:*)or an action name, grants nothing. Once a narrowing skill is loaded, a read outside its list is refused:Skill checkout-timeouts limits this run's reads to ... (its allowed-tools). Use one of those, or answer from the evidence. - Reach outside the alert’s namespace. Skills change none of the checks on Nightshift’s reads and queries.
- Carry a credential. Every line of the frontmatter, the body and each
reference file is checked for credentials. A skill with one is refused,
naming the line and the kind of value, for example
DB_PASSWORD (12 characters)- never the value. Placeholders are fine:<db password>,${DB_PASSWORD},{{ .Values.password }},***,REDACTED,xxxx.
Every change to a skill or to the repository setting is recorded in the audit log, by the skill’s name, not its text.