Skip to content

Connect Grafana, Prometheus and Alertmanager

A connection is the address of one of your tools and a key to read it with. Once Grafana, Prometheus or Alertmanager is connected, Import reads dashboards, alert rules, blackbox probes and routes straight from it - no exporting files.

Connections live on Admin → Integrations → Grafana, Prometheus, Alertmanager, one card per tool. An admin can also connect a tool from the Import page itself, at the moment it is needed.

What a connection can do is fixed: it only reads, and only these things.

Tool What Seylo reads Nothing else
Grafana Its health, the list of dashboards, one dashboard at a time, the list of Grafana-managed alert rules No writes, no users, no data sources, no queries
Prometheus Its version, its loaded rules, its active scrape targets No queries, no writes, no admin APIs
Alertmanager Its status, which includes the running configuration - reduced to routes and receiver names before it goes anywhere No silences, no alerts, no writes

Choose one per connection, under How Seylo reaches it.

From Seylo Through the agent
For An address on the internet: Grafana Cloud, a public Grafana or Prometheus A Service inside one of your clusters
URL looks like https://acme.grafana.net, https://prometheus-prod-01-eu-west-0.grafana.net/api/prom http://grafana.monitoring.svc, http://prometheus-operated.monitoring.svc:9090
Who dials it The Seylo server, over https only The Seylo agent in that cluster
Where the key is kept In Seylo, encrypted, never shown again In your cluster, in a Secret you create. Seylo never receives it
Needs Nothing in your cluster The Seylo agent 0.2.15 or later on that cluster

From Seylo refuses private addresses (10.x, 192.168.x, …), names that only resolve inside a network (*.svc, *.internal, a name with no dot) and cloud metadata addresses, and says why. The check is made again on every request against the addresses the name actually resolves to, and Seylo never follows a redirect, so a public name cannot be pointed at something private later.

Through the agent accepts only a Kubernetes Service name, name.namespace.svc (or ...svc.cluster.local), with any port. The agent will not dial any other host, so a connection can never turn it into a way into the rest of your network. To reach a tool outside the cluster that only the cluster can reach, create an ExternalName Service for it and use that name. The Service’s namespace must be one the agent is allowed to work in.

  1. Make a read-only key in the tool (below, per tool).
  2. Open Admin → Integrations → Grafana, Prometheus, Alertmanager and choose Connect Grafana (or Prometheus, or Alertmanager) on its card.
  3. Give it a Name, choose How Seylo reaches it, and enter the URL: the address you open the tool at, with its sub-path if it has one (for example https://ops.acme.example/grafana), and nothing after it.
  4. Choose the Authentication:
    • From Seylo: paste the token (or the user name and password).
    • Through the agent: leave the key name as it is (for example grafana-token) and run the command the form shows, in your own terminal, to put the key in the agent’s Secret.
  5. Choose Test connection. It shows each check and, if one fails, what went wrong and what to do.
  6. Choose Save.

The key field is empty after Save and stays empty. The card shows Key set, when and by whom. To change a key, choose Replace key; to stop using a connection, Remove (the stored key is deleted with it). There is no way to show a key again.

Through the agent: the key in your cluster

Section titled “Through the agent: the key in your cluster”

The agent reads keys from one Secret, seylo-connections, in its own namespace, by key name, for each request. The chart lets it read that one Secret and no other. Create it once, then add a key per connection:

Terminal window
# The first connection's key
kubectl -n seylo-agent create secret generic seylo-connections \
--from-literal=grafana-token='<paste the key here>'
# Each later key, added to the same Secret
kubectl -n seylo-agent patch secret seylo-connections --type merge \
-p '{"stringData":{"prometheus-password":"<paste the key here>"}}'

Use your agent’s namespace if you installed it elsewhere. Rotating a key is the same patch; the agent reads it again on the next request, with no restart.

A tool with no authentication inside the cluster needs no Secret: choose None.

Key: a service account token with the Viewer role.

  1. In Grafana, open Administration → Users and access → Service accounts and choose Add service account. Name it (for example seylo) and give it the Viewer role.
  2. On the new account, choose Add service account token, then copy the token. Grafana shows it once.
  3. Connect Grafana with that token as Service account token.

Viewer is enough. Seylo only lists and reads dashboards and alert rules. A token with Editor or Admin works too, but holds more than Seylo needs. To import only some folders, give the service account Viewer on those folders instead of the Viewer role.

URL:

  • Grafana Cloud: your stack’s address, https://<stack>.grafana.net. From Seylo.
  • Your own Grafana on the internet: the address you open it at, with its sub-path if any. From Seylo.
  • Grafana in your cluster: its Service, for example http://grafana.monitoring.svc (kube-prometheus-stack names it <release>-grafana). Through the agent.

Test checks that Grafana answers (and its version) and that the token can list dashboards, and says how many it sees.

Key: Prometheus has no keys of its own. What to use depends on what is in front of it:

  • Grafana Cloud (hosted Prometheus): User name and password (Basic). The user name is the instance ID (your stack → Prometheus → Details); the password is an access policy token with the metrics:read and rules:read scopes. The URL is the query endpoint from the same page, ending in /api/prom. From Seylo.
  • Behind a proxy (oauth2-proxy, an ingress with authentication): the credential that proxy expects - a bearer token or a user name and password.
  • In your cluster with no authentication (the usual kube-prometheus-stack install): choose None, and use the Service, for example http://prometheus-operated.monitoring.svc:9090. Through the agent.

For Mimir, the URL ends in /prometheus. For Thanos, use the Query address; rules are listed only if Thanos Query is configured with rule endpoints.

Test checks Prometheus’ version (Thanos, Mimir and some VictoriaMetrics builds do not report one; that is a note, not a failure), that the rules can be read and how many alerting and recording rules there are, and whether scrape targets can be listed for blackbox probes. Grafana Cloud and Mimir do not list scrape targets; import blackbox probes from a file there.

Blackbox probes come from Prometheus. Connect the Prometheus that scrapes your blackbox exporter; Seylo finds every target whose scrape calls the exporter’s /probe with a module and a target. There is no separate blackbox connection.

Key: Alertmanager has no API keys. If its web configuration (--web.config.file) sets basic authentication, use that user name and password (Basic); behind a proxy, the credential the proxy expects; otherwise None.

URL: in a cluster, its Service, for example http://alertmanager-operated.monitoring.svc:9093, through the agent. If Alertmanager runs with a route prefix (--web.route-prefix, or routePrefix in kube-prometheus-stack), include it in the URL.

What Seylo keeps: Alertmanager’s status includes its whole running configuration, receivers’ webhook URLs and keys included. That configuration is reduced to routes, receiver names and kinds and email addresses where it is read - in the agent, inside your cluster, for a connection through the agent - before it goes anywhere. No receiver URL or key reaches Seylo. You type each channel’s secret once, on the import preview.

Test checks that Alertmanager answers and reports its version, and how many receivers and routes its configuration has.

Action Role
See connections (names, URLs, whether a key is set) viewer
Connect, edit, test, replace a key, remove admin
Import dashboards, alert rules and uptime checks from a connection releaser
Import Alertmanager routes and channels from a connection admin

Grafana dashboards, Prometheus rules and Alertmanager routes can be about any namespace, so importing from a connection is for people whose access covers every cluster and namespace. Someone limited to some namespaces sees why and who can do it instead.

The test shows what went wrong in the tool’s own terms. The common ones:

You see What to do
Grafana answered 401: the token is invalid or expired. Make a new service account token with the Viewer role and choose Replace key
Grafana answered 403: the token is valid but may not list dashboards. Give the service account the Viewer role, or Viewer on the folders to import
Nothing at ... answered as Grafana (404). Use the address you open Grafana at, with its sub-path, without /d/...
Prometheus answered 401: the credentials were not accepted. For Grafana Cloud: Basic, the instance ID as user, an access policy token with metrics:read and rules:read
... /api/v1/rules was not found (404). Use Prometheus’ own address: ending in /api/prom for Grafana Cloud, /prometheus for Mimir
... /api/v2/status was not found (404). Alertmanager 0.16 or newer is needed; include any route prefix in the URL
... answered 302, a redirect Seylo does not follow redirects. Use the final address, often https:// instead of http://
... is a private, loopback or reserved address The tool is not on the internet: connect it through the agent
... is a Kubernetes Service name, which only resolves inside its cluster Choose Through the agent and the cluster it runs in
The agent found no Secret "seylo-connections" Run the create secret command above
The Secret "seylo-connections" ... has no key Run the patch secret command above with that key name
... needs agent 0.2.15 or later Upgrade the agent on that cluster with the command shown, or connect From Seylo if the tool is on the internet
...'s certificate was not accepted From Seylo needs a certificate a public CA signed. Connect through the agent instead