Notification channels and routes
A channel is somewhere an alert can be sent. A route decides which alerts go to which channels. A channel does nothing until a route sends alerts to it. Both are under Observability → Alerts, on the Channels and Routes tabs.
| Action | Role |
|---|---|
| See channels, routes and delivery logs | viewer |
| Create, edit or delete a channel; Send test | admin |
| Create, edit, reorder or delete a route | admin |
For what an alert is and how to acknowledge, snooze or silence one, see Alerts.
Before your first route
Section titled “Before your first route”With no enabled route, nothing changes: your organisation keeps the role-based alert email it already had (a daily digest, by default).
The first enabled route switches your organisation to routing. From then on routes are the whole answer to “who is told”, and the role-based alert email is no longer sent. The Routes tab says so before you save. To keep emailing people by role, create an Email channel and tick the roles.
Set up a channel
Section titled “Set up a channel”- Channels → New channel. Choose the kind and give it a name of 1 to 80
characters, such as
#ops-alertsorOn-call (PagerDuty). - Paste what the kind needs (below).
- Create channel, then Send test. The test sends a fictional alert through the same code a real alert uses and shows the receiver’s answer.
- On the Routes tab, send alerts to it.
A channel’s kind cannot change after it is created. Create a new channel instead. An organisation can have at most 50 channels.
| Kind | You paste | Acknowledge and resolve arrive as |
|---|---|---|
| Up to 20 addresses, and/or roles (everyone holding the role is emailed) | Follow-up messages | |
| Slack | An incoming-webhook URL, https://hooks.slack.com/services/... |
Follow-up messages (a webhook cannot thread or edit) |
| Microsoft Teams | The URL of a Workflows flow | Follow-up messages |
| Google Chat | A space webhook URL, https://chat.googleapis.com/v1/spaces/SPACE/messages?key=...&token=... |
Replies in the alert’s own thread |
| Webhook | An https:// receiver URL and a signing secret you generate |
alert.acknowledged and alert.resolved requests |
| PagerDuty | The 32-character Integration Key of an Events API v2 integration, and the US or EU service region | Acknowledge and resolve on the same incident |
| Opsgenie / Jira Service Management | An API integration key, and where: Opsgenie US, Opsgenie EU, or Jira Service Management operations | Acknowledge and close on the same alert |
Emails go through Seylo’s mailer. Your organisation’s daily cap on notification email (200 a day) still applies. They are always sent straight away, never held for a person’s daily digest, because an administrator routed them on purpose.
Each recipient can still turn routed alerts off for themselves, under Your notification email. Nobody is muted silently: the channel’s delivery log names the recipients who turned them off, and if every recipient did, the delivery is recorded as not delivered, with the reason. For alerts someone must act on at any hour, or for volume, route to a chat, paging or webhook channel.
In your Slack app, open Incoming Webhooks, add one for the channel, and
paste its URL. A URL that is not on hooks.slack.com/services/ is refused — the
common mistake is pasting the app’s settings page.
Microsoft Teams
Section titled “Microsoft Teams”In the Teams channel, add the Workflows template “Post to a channel when a
webhook request is received” and paste its URL. Workflows URLs are on
logic.azure.com or powerplatform.com.
Google Chat
Section titled “Google Chat”In the space: Apps & integrations → Webhooks, then copy the URL. Each alert gets its own thread, so acknowledge and resolve appear under the firing message.
Webhook
Section titled “Webhook”Seylo sends a signed JSON POST to your https:// receiver. You supply the
signing secret: 32 to 256 characters with no spaces, for example from
openssl rand -hex 32. Keep a copy for your receiver — Seylo never shows it
again, and does not generate one for you, because showing a generated secret
even once would be a response containing a secret.
The receiver URL is treated as a secret too, because people put tokens in URLs. Verifying the signature is below.
PagerDuty
Section titled “PagerDuty”On the PagerDuty service: Integrations → Events API v2, then paste its Integration Key (32 characters). A REST API key does not work here. Choose the EU region if your account is in PagerDuty’s EU service region.
The Seylo alert ID is the incident’s dedup key, so acknowledge and resolve reach the same incident. Send test opens a real test incident at the lowest severity and closes it again at once; depending on the service’s settings, someone may be notified. The page says so before you press it.
Opsgenie and Jira Service Management
Section titled “Opsgenie and Jira Service Management”Paste the key of an API integration on its own — without GenieKey.
Atlassian shuts Opsgenie down on 5 April 2027 and moves it into Jira Service
Management operations, which accepts the same key: choose Jira Service
Management operations once you have moved. The Seylo alert ID is the alert’s
alias, so acknowledge and close reach the same alert. Send test opens a test
alert and closes it again at once.
How secrets are handled
Section titled “How secrets are handled”The URL, key or signing secret you paste is what lets a stranger post into your Slack or page your on-call, so it is write-only:
- It is encrypted when you save it and never returned — not by the page, the API, an export, an error message or the audit log.
- In its place the page shows that it is set, with a hint you can recognise it by: the host a URL points at, or “routing key set” / “API key set”. Not even the last four characters.
- When you edit a channel, leave the secret blank to keep the stored one.
- If a send fails, the error names the host, and every part of the secret — the whole URL, its path, each query value — is removed from it before it is stored.
- Alert messages are redacted before they leave for a third party, because a Kubernetes event message can quote a command line.
To change a secret, paste a new one. To see the old one, you cannot.
Deleting a channel is real deletion: its stored secret and delivery log go with it, and the audit log records that it existed and who removed it. A channel still named by a route cannot be deleted until the route stops naming it.
Routes
Section titled “Routes”A route says: alerts that match these matchers go to these channels.
| Setting | Meaning | Default and limits |
|---|---|---|
| Matchers | cluster, namespace, workload, reason, severity — the same grammar as silences | None at all = matches every alert |
| Send to | One or more channels | At least one |
| Send when the alert | fired (always), still firing, acknowledged, resolved | All four |
| Continue | Keep evaluating routes below this one after it matches | Off |
| Group wait | How long a new alert waits before it is sent | 30 seconds; 0 to 600 |
| Repeat interval | At most one “still firing” reminder per channel this often | 4 hours; 5 minutes to 7 days |
An organisation can have at most 50 routes.
Order: the first match wins
Section titled “Order: the first match wins”Routes are evaluated top to bottom, and the first route that matches decides, unless it has Continue set. Reorder them with the arrows on the Routes tab.
The usual shape is two routes:
severity = critical,cluster = prod-*→ PagerDuty- no matchers (the catch-all, last) →
#ops-alertsin Slack
With first-match, a critical alert in production pages on-call and is not
also posted to #ops-alerts. Set Continue on route 1 if you want both.
If two matching routes name the same channel, the channel is sent to once.
What is sent, and when
Section titled “What is sent, and when”| Change | Who is told |
|---|---|
| Fired (new, or recurred after resolving) | Every matching route’s channels, after the group wait |
| Still firing | Matching routes again, at most once per repeat interval per channel. Never to PagerDuty or Opsgenie, which re-notify by their own escalation policy |
| Acknowledged, resolved | Only channels that were told it fired, whatever the routes say now — unless the route that sent it has since turned that update off |
| Snooze, un-acknowledge, assign, comment | Nothing is sent |
Group wait does two jobs. An alert that resolves within the wait pages nobody: the waiting notification is cancelled instead. And on Email, Slack, Teams and Google Chat, three or more alerts due at the same moment arrive as one message, so a storm reads as one notification, not forty.
Snooze means “remind me”. When a snooze ends, the next “still firing” is sent even inside the repeat interval: snoozing for an hour means being reminded in an hour, not waiting out the rest of a four-hour interval.
Deleting a route is real deletion, recorded in the audit log. Alerts it matched fall through to the routes below it — or, if no enabled route is left, back to the role-based alert email.
Delivery and retries
Section titled “Delivery and retries”Every notification is written down first and then sent, so a restart or a vendor’s bad minute does not lose it.
- Just before sending, Seylo checks again whether the alert was acknowledged, snoozed, silenced or resolved in the meantime. If so, the notification is cancelled, with that reason.
- Requests go to public addresses only, never follow a redirect, and time out after 10 seconds.
- Retried, because time can fix them: timeouts, network and DNS errors, and
HTTP 408, 425, 429 and 5xx. Up to 8 attempts, waiting 30 seconds, then 1,
2, 4, 8, 16 and 32 minutes between them. If the receiver sends
Retry-Afterasking for longer, that wait is used, up to 1 hour. - Dead at once, because retrying cannot fix them — each with a sentence saying what to change:
| Answer | What it means | What to do |
|---|---|---|
| 3xx | A redirect; Seylo does not follow them | Paste the final URL |
| 400 and other 4xx not listed here | The receiver refused the message | Read the receiver’s quoted answer in the delivery log |
| 401, 403 | The credential was rejected: a revoked key, a disabled integration, or a workflow whose owner left | Re-enter the secret on the channel |
| 404, 410 | The webhook or integration no longer exists | Create a new one and paste it |
| An address that is not public | Channels reach public addresses only | Change the channel’s URL |
| Older than 24 hours | Could not be delivered within a day; a day-old alert is no longer news | Fix the channel; the next alert is sent normally |
Every outcome is on the channel’s Delivery log (status, event, attempts, the receiver’s answer) and on the alert’s timeline.
At-least-once. If Seylo stops after the receiver accepted a message but before recording it as sent, it sends again. PagerDuty and Opsgenie deduplicate on the alert ID; a chat channel may rarely show a message twice. The opposite choice would occasionally lose a page, which is worse.
Verify a webhook signature
Section titled “Verify a webhook signature”Every webhook request carries:
| Header | Value |
|---|---|
Seylo-Signature |
t=<unix seconds>,v1=<hex HMAC-SHA256> |
Seylo-Event |
alert.firing, alert.repeat, alert.acknowledged, alert.resolved, or alert.test |
Seylo-Delivery |
One ID per delivery, the same across its retries — use it to deduplicate |
User-Agent |
Seylo-Notify/1 (+https://seylo.in) |
To verify:
expected = hex( HMAC-SHA256( signing_secret, t + "." + raw_request_body ) )accept if constant_time_equal(expected, v1) and |now - t| <= 300 seconds- Compute over the raw body bytes, before any JSON parsing. Re-serialised JSON will not match.
- The timestamp is inside the signed string, so a captured request cannot be replayed later with a new timestamp. Reject anything older than five minutes.
v1=names the scheme, so a second scheme can be added during a rotation without breaking receivers that know only the first.
A minimal receiver check in Node.js:
import crypto from "node:crypto";
function verify(secret, header, rawBody, nowSec = Math.floor(Date.now() / 1000)) { const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2).map((s) => s.trim()))); const t = Number(parts.t); if (!Number.isInteger(t) || !parts.v1 || Math.abs(nowSec - t) > 300) return false; const expected = crypto.createHmac("sha256", secret).update(`${t}.${rawBody}`, "utf8").digest(); const given = Buffer.from(parts.v1, "hex"); return given.length === expected.length && crypto.timingSafeEqual(given, expected);}The body is JSON:
{ "version": "1", "event": "alert.firing", "deliveryId": "...", "sentAt": "2026-10-04T09:30:00.000Z", "alert": { "id": "...", "clusterId": "prod-eu", "namespace": "shop", "workload": "checkout", "container": "app", "pod": "checkout-7c9f8b6d5-x2k4q", "reason": "CrashLoopBackOff", "message": "...", "severity": "critical", "status": "firing", "firstSeen": "2026-10-04T09:29:12.000Z", "lastSeen": "2026-10-04T09:29:42.000Z", "count": 3 }, "actor": null, "note": null, "link": "https://portal.seylo.in/..."}actor is who acknowledged or resolved it (null when the cluster resolved
it). note is the resolve reason. message is redacted before it is sent. link
points into the portal, and is null when there is no link to give.