Connect Grafana, Prometheus and Alertmanager
A connection is the address of one of your tools and a key to read it with. Once Grafana, Prometheus or Alertmanager is connected, Import reads dashboards, alert rules, blackbox probes and routes straight from it - no exporting files.
Connections live on Admin → Integrations → Grafana, Prometheus,
Alertmanager, one card per tool. An admin can also connect a tool from the
Import page itself, at the moment it is needed.
What a connection can do is fixed: it only reads, and only these things.
| Tool | What Seylo reads | Nothing else |
|---|---|---|
| Grafana | Its health, the list of dashboards, one dashboard at a time, the list of Grafana-managed alert rules | No writes, no users, no data sources, no queries |
| Prometheus | Its version, its loaded rules, its active scrape targets | No queries, no writes, no admin APIs |
| Alertmanager | Its status, which includes the running configuration - reduced to routes and receiver names before it goes anywhere | No silences, no alerts, no writes |
Two ways Seylo reaches a tool
Section titled “Two ways Seylo reaches a tool”Choose one per connection, under How Seylo reaches it.
| From Seylo | Through the agent | |
|---|---|---|
| For | An address on the internet: Grafana Cloud, a public Grafana or Prometheus | A Service inside one of your clusters |
| URL looks like | https://acme.grafana.net, https://prometheus-prod-01-eu-west-0.grafana.net/api/prom |
http://grafana.monitoring.svc, http://prometheus-operated.monitoring.svc:9090 |
| Who dials it | The Seylo server, over https only | The Seylo agent in that cluster |
| Where the key is kept | In Seylo, encrypted, never shown again | In your cluster, in a Secret you create. Seylo never receives it |
| Needs | Nothing in your cluster | The Seylo agent 0.2.15 or later on that cluster |
From Seylo refuses private addresses (10.x, 192.168.x, …), names that
only resolve inside a network (*.svc, *.internal, a name with no dot) and
cloud metadata addresses, and says why. The check is made again on every
request against the addresses the name actually resolves to, and Seylo never
follows a redirect, so a public name cannot be pointed at something private
later.
Through the agent accepts only a Kubernetes Service name,
name.namespace.svc (or ...svc.cluster.local), with any port. The agent will
not dial any other host, so a connection can never turn it into a way into the
rest of your network. To reach a tool outside the cluster that only the cluster
can reach, create an ExternalName Service for it and use that name. The
Service’s namespace must be one the agent is allowed to work in.
Connect a tool
Section titled “Connect a tool”- Make a read-only key in the tool (below, per tool).
- Open Admin → Integrations → Grafana, Prometheus, Alertmanager and choose Connect Grafana (or Prometheus, or Alertmanager) on its card.
- Give it a Name, choose How Seylo reaches it, and enter the URL:
the address you open the tool at, with its sub-path if it has one (for
example
https://ops.acme.example/grafana), and nothing after it. - Choose the Authentication:
- From Seylo: paste the token (or the user name and password).
- Through the agent: leave the key name as it is (for example
grafana-token) and run the command the form shows, in your own terminal, to put the key in the agent’s Secret.
- Choose Test connection. It shows each check and, if one fails, what went wrong and what to do.
- Choose Save.
The key field is empty after Save and stays empty. The card shows Key set, when and by whom. To change a key, choose Replace key; to stop using a connection, Remove (the stored key is deleted with it). There is no way to show a key again.
Through the agent: the key in your cluster
Section titled “Through the agent: the key in your cluster”The agent reads keys from one Secret, seylo-connections, in its own
namespace, by key name, for each request. The chart lets it read that one
Secret and no other. Create it once, then add a key per connection:
# The first connection's keykubectl -n seylo-agent create secret generic seylo-connections \ --from-literal=grafana-token='<paste the key here>'
# Each later key, added to the same Secretkubectl -n seylo-agent patch secret seylo-connections --type merge \ -p '{"stringData":{"prometheus-password":"<paste the key here>"}}'Use your agent’s namespace if you installed it elsewhere. Rotating a key is
the same patch; the agent reads it again on the next request, with no
restart.
A tool with no authentication inside the cluster needs no Secret: choose None.
Grafana and Grafana Cloud
Section titled “Grafana and Grafana Cloud”Key: a service account token with the Viewer role.
- In Grafana, open Administration → Users and access → Service accounts
and choose Add service account. Name it (for example
seylo) and give it the Viewer role. - On the new account, choose Add service account token, then copy the token. Grafana shows it once.
- Connect Grafana with that token as Service account token.
Viewer is enough. Seylo only lists and reads dashboards and alert rules. A token with Editor or Admin works too, but holds more than Seylo needs. To import only some folders, give the service account Viewer on those folders instead of the Viewer role.
URL:
- Grafana Cloud: your stack’s address,
https://<stack>.grafana.net. From Seylo. - Your own Grafana on the internet: the address you open it at, with its sub-path if any. From Seylo.
- Grafana in your cluster: its Service, for example
http://grafana.monitoring.svc(kube-prometheus-stack names it<release>-grafana). Through the agent.
Test checks that Grafana answers (and its version) and that the token can list dashboards, and says how many it sees.
Prometheus
Section titled “Prometheus”Key: Prometheus has no keys of its own. What to use depends on what is in front of it:
- Grafana Cloud (hosted Prometheus): User name and password (Basic).
The user name is the instance ID (your stack → Prometheus →
Details); the password is an access policy token with the
metrics:read and rules:read scopes. The URL is the query endpoint
from the same page, ending in
/api/prom. From Seylo. - Behind a proxy (oauth2-proxy, an ingress with authentication): the credential that proxy expects - a bearer token or a user name and password.
- In your cluster with no authentication (the usual
kube-prometheus-stack install): choose None, and use the Service, for
example
http://prometheus-operated.monitoring.svc:9090. Through the agent.
For Mimir, the URL ends in /prometheus. For Thanos, use the Query address;
rules are listed only if Thanos Query is configured with rule endpoints.
Test checks Prometheus’ version (Thanos, Mimir and some VictoriaMetrics builds do not report one; that is a note, not a failure), that the rules can be read and how many alerting and recording rules there are, and whether scrape targets can be listed for blackbox probes. Grafana Cloud and Mimir do not list scrape targets; import blackbox probes from a file there.
Blackbox probes come from Prometheus. Connect the Prometheus that scrapes
your blackbox exporter; Seylo finds every target whose scrape calls the
exporter’s /probe with a module and a target. There is no separate blackbox
connection.
Alertmanager
Section titled “Alertmanager”Key: Alertmanager has no API keys. If its web configuration
(--web.config.file) sets basic authentication, use that user name and
password (Basic); behind a proxy, the credential the proxy expects;
otherwise None.
URL: in a cluster, its Service, for example
http://alertmanager-operated.monitoring.svc:9093, through the agent. If
Alertmanager runs with a route prefix (--web.route-prefix, or routePrefix
in kube-prometheus-stack), include it in the URL.
What Seylo keeps: Alertmanager’s status includes its whole running configuration, receivers’ webhook URLs and keys included. That configuration is reduced to routes, receiver names and kinds and email addresses where it is read - in the agent, inside your cluster, for a connection through the agent - before it goes anywhere. No receiver URL or key reaches Seylo. You type each channel’s secret once, on the import preview.
Test checks that Alertmanager answers and reports its version, and how many receivers and routes its configuration has.
Who can do what
Section titled “Who can do what”| Action | Role |
|---|---|
| See connections (names, URLs, whether a key is set) | viewer |
| Connect, edit, test, replace a key, remove | admin |
| Import dashboards, alert rules and uptime checks from a connection | releaser |
| Import Alertmanager routes and channels from a connection | admin |
Grafana dashboards, Prometheus rules and Alertmanager routes can be about any namespace, so importing from a connection is for people whose access covers every cluster and namespace. Someone limited to some namespaces sees why and who can do it instead.
When Test fails
Section titled “When Test fails”The test shows what went wrong in the tool’s own terms. The common ones:
| You see | What to do |
|---|---|
Grafana answered 401: the token is invalid or expired. |
Make a new service account token with the Viewer role and choose Replace key |
Grafana answered 403: the token is valid but may not list dashboards. |
Give the service account the Viewer role, or Viewer on the folders to import |
Nothing at ... answered as Grafana (404). |
Use the address you open Grafana at, with its sub-path, without /d/... |
Prometheus answered 401: the credentials were not accepted. |
For Grafana Cloud: Basic, the instance ID as user, an access policy token with metrics:read and rules:read |
... /api/v1/rules was not found (404). |
Use Prometheus’ own address: ending in /api/prom for Grafana Cloud, /prometheus for Mimir |
... /api/v2/status was not found (404). |
Alertmanager 0.16 or newer is needed; include any route prefix in the URL |
... answered 302, a redirect |
Seylo does not follow redirects. Use the final address, often https:// instead of http:// |
... is a private, loopback or reserved address |
The tool is not on the internet: connect it through the agent |
... is a Kubernetes Service name, which only resolves inside its cluster |
Choose Through the agent and the cluster it runs in |
The agent found no Secret "seylo-connections" |
Run the create secret command above |
The Secret "seylo-connections" ... has no key |
Run the patch secret command above with that key name |
... needs agent 0.2.15 or later |
Upgrade the agent on that cluster with the command shown, or connect From Seylo if the tool is on the internet |
...'s certificate was not accepted |
From Seylo needs a certificate a public CA signed. Connect through the agent instead |