Compare with Datadog
If you already run Datadog, you do not have to choose on day one. Seylo can read your Datadog where it is, and compare it with what Seylo’s own collection records for the same cluster. When the numbers agree, you can switch Datadog off knowing what you keep. If they do not, the report shows where and by how much. Switching Datadog off is always your decision and your action; Seylo never touches your Datadog setup.
Nothing about your Datadog setup changes. Seylo does not install anything next to the Datadog Agent, does not copy your data out of Datadog, and does not ask you to send the same data to two places.
How it works
Section titled “How it works”- The Seylo agent in your cluster reads Datadog through Datadog’s API, with an API key and an application key that you store in a Kubernetes Secret in your cluster.
- The keys never leave your cluster. They are not sent to Seylo, not logged and not shown anywhere. The agent sends them only to the API of the Datadog site you name.
- Every query is limited to this cluster — its
kube_cluster_nametag in Datadog — and to the namespaces the person asking can see. Seylo writes the Datadog query itself from a fixed set of metrics; nobody can send it a query of their own.
The same connector lets you import Datadog monitors, dashboards and HTTP tests.
Before you start
Section titled “Before you start”- Seylo agent 0.2.10 or later on the cluster. With an older agent, the page says Needs agent 0.2.10 for Datadog, with a link to upgrade it; see upgrade the agent.
- The seylo-observability chart installed, so Seylo has its own numbers to compare. See install observability.
- A Datadog API key and an application key from the same organisation.
If the application key is scoped, it needs only the
timeseries_querypermission. - The cluster’s name in Datadog: the value of the
kube_cluster_nametag, which is what your Datadog Agent reports asDD_CLUSTER_NAME. - Outbound HTTPS from the agent’s namespace to your Datadog site’s API host
(for example
api.datadoghq.com), directly or through the proxy the agent already uses (proxy.url).
Connect Datadog
Section titled “Connect Datadog”The same two commands are shown on Admin → Integrations → Datadog, and on the Import page’s Datadog tab, whenever the connector is off.
-
Create the Secret in the agent’s namespace. Type the keys into your own terminal — never into Seylo:
Terminal window kubectl -n seylo-agent create secret generic seylo-datadog \--from-literal=DD_API_KEY=<api key> \--from-literal=DD_APP_KEY=<application key> -
Turn the connector on (Helm 3.14 or later):
Terminal window helm upgrade seylo-agent oci://ghcr.io/seylo-inc/charts/seylo-agent \--namespace seylo-agent --reset-then-reuse-values \--set datadog.enabled=true \--set datadog.site=datadoghq.com \--set datadog.clusterName=<kube_cluster_name> \--set datadog.existingSecret=seylo-datadogUse your own site and the cluster name your Datadog Agent reports. On Helm older than 3.14, run
helm get values seylo-agent -n seylo-agent -o yaml > mine.yamland pass-f mine.yamlinstead of the reuse flag.datadog.siteis the site you sign in to. Seylo accepts these and nothing else:datadoghq.com,us3.datadoghq.com,us5.datadoghq.com,datadoghq.eu,ap1.datadoghq.com,ap2.datadoghq.com,uk1.datadoghq.com,ddog-gov.com,us2.ddog-gov.com. -
In Seylo, open Admin → Integrations → Datadog, pick the cluster and choose Check connection. The check tests the API key, then runs one small query that needs the application key, and reports how many namespaces of this cluster Datadog has data for in the last 30 minutes.
The Connection card then shows the site, the cluster tag, whether each key is present in the Secret (never the key), the last check and the agent version.
Run a parity report
Section titled “Run a parity report”Under Admin → Integrations → Datadog, choose a window — 1, 3, 7, 14 or 30 days, 7 by default — and choose Run parity report. The page shows its progress as it runs.
For each metric below, Seylo reads its own store and Datadog over the same windows — 5-minute windows per namespace, hourly windows per pod and node — and compares them window by window.
| Metric | Compared per | Seylo reads | Datadog reads |
|---|---|---|---|
| Container CPU used | namespace, pod | container_cpu_usage_seconds_total (rate) |
kubernetes.cpu.usage.total (nanocores, converted to cores) |
| Memory working set | namespace, pod | container_memory_working_set_bytes |
kubernetes.memory.working_set |
| Container restarts | namespace, pod | kube_pod_container_status_restarts_total |
kubernetes_state.container.restarts |
| Network received / sent | namespace, pod | container_network_receive_bytes_total / container_network_transmit_bytes_total (rate) |
kubernetes.network.rx_bytes / kubernetes.network.tx_bytes |
| Running pods | namespace | kube_pod_status_phase{phase="Running"} |
kubernetes_state.pod.status_phase with pod_phase:running |
| Node CPU and memory capacity | node | kube_node_status_capacity |
kubernetes_state.node.cpu_capacity / kubernetes_state.node.memory_capacity |
The (i) beside each row shows the exact query each side ran, so you can paste the Datadog one into Datadog’s Metrics Explorer and check it yourself.
Reading the result
Section titled “Reading the result”The summary says how many rows match, for example 7 of 9 match. Each row,
and each namespace, pod or node inside it, gets one of these:
- match — both sides have a value in every window, and every window is within 1%.
- differs — both sides have it, but some windows are missing on one side, or some values are more than 1% apart. Seylo shows both averages and how far apart they are.
- missing in Seylo — Datadog has it and Seylo does not. This is the one to look at before switching Datadog off.
- missing in Datadog — Seylo has it and Datadog does not.
- not comparable — one side could not answer (the reason is shown), or neither side has any data. Missing data is never shown as zero.
Entities seen by one side only are listed under Seen by one side only.
Why two sides can differ
Section titled “Why two sides can differ”- Pods on the host network report the node’s interfaces, which can make network figures differ.
- Windows at the edges. A pod that starts or stops in the middle of a window can be counted by one side and not the other in that window.
- Collection gaps. If either collector was down for a while, its windows are missing, and the row shows which side lacks them.
- Retention. Seylo keeps up to 30 days by default, and logs stop at 16 GiB first on a busy cluster (see retention); a window longer than what Seylo has kept shows the early windows as missing in Seylo.
What is not compared
Section titled “What is not compared”Listed on the report under Not compared:
- Node CPU and memory used. Datadog does not document a node-level series for these that Seylo can match exactly, so node capacity is compared instead rather than guessed.
- Per-Deployment figures. The two products work out a pod’s Deployment differently, so a difference would not mean either side missed data. Namespaces and pods are compared instead.
- Memory usage including cache. The working set is the figure Seylo’s pages and alerts use.
- Logs, traces and APM. The report compares metrics only; the connector does not read Datadog’s logs or traces.
Who can run it
Section titled “Who can run it”Anyone who can view observability for the cluster. A person who can see only some namespaces gets a report limited to those namespaces, and no node rows. Before such a report runs, Seylo checks that Datadog really returns nothing for a namespace that does not exist. If that check cannot pass — for example, Datadog has no data for the cluster yet — the report is refused with the reason, rather than run unfiltered.
A report is shown only to the person who ran it, and kept for an hour.
Datadog’s limits
Section titled “Datadog’s limits”A 7-day report makes about 20 Datadog API requests, one at a time; a 30-day report about 45. Only one report runs per cluster at a time, because it spends your Datadog API quota. If Datadog rate-limits your organisation, the report stops and says when to retry, using the time Datadog gives. Seylo never retries on your quota by itself.
Troubleshooting
Section titled “Troubleshooting”| What you see | What to do |
|---|---|
The Datadog connector is off on this agent (datadog.enabled=false). |
Run the two commands above. See messages you may see |
datadog.site is not set. |
Set datadog.site to the site you sign in to |
"..." is not a Datadog site Seylo knows |
Use one of the sites listed above |
The Datadog Secret mounted at ... has no DD_API_KEY (or DD_APP_KEY) |
The Secret is missing a key, or datadog.existingSecret names the wrong Secret. The keys must be named exactly DD_API_KEY and DD_APP_KEY |
unauthorized: Datadog did not accept the API key |
The API key is wrong, is an application key, or belongs to another site. Check datadog.site |
forbidden: Datadog refused the metrics query |
The application key is from another organisation, or a scoped key lacks timeseries_query |
no-data |
datadog.clusterName does not match the name your Datadog Agent reports. Copy it from Datadog |
rate-limited |
Wait for the time shown and run again |
unreachable or timeout |
Allow outbound HTTPS from the agent’s namespace to your site’s API host, or set the agent’s proxy.url |
A parity report is already running for this cluster |
Wait for it to finish |
which has no Datadog connector |
Upgrade the agent to 0.2.10 or later |