Autoscaler or a bigger request
When a workload uses more CPU than it requests, there are two fixes, and picking the wrong one costs money or does nothing:
- An autoscaler (HPA) adds pods when load rises. It helps only when the load splits across pods.
- A bigger request gives each pod what it uses. It is right when the load is flat, or when one pod does all the work.
Nightshift does not guess which. It asks the question an SRE asks first - if I add a pod, does each pod’s CPU go down? - and answers it from measurements.
What it measures
Section titled “What it measures”| Measurement | Read from |
|---|---|
| Each pod’s CPU over the last 24 hours, or since the workload was created if that is more recent | Your metrics store, at a 2-minute step: its 95th percentile, its quietest level, and how much the total moves. A workload deleted and created again under the same name does not inherit the old one’s load |
| The workload’s traffic | The first source that answers: span metrics from traces, http_requests_total (and its Spring and OpenTelemetry names), or the ingress-nginx controller in front of its Services |
| Whether CPU follows traffic | How closely the two move together over the same timestamps |
| How the load spreads | With two or more pods, whether one of them does the work |
| What already scales it | An existing HPA, or a VerticalPodAutoscaler that sets its CPU |
| Room and limits | Room on the nodes for another pod, and its PodDisruptionBudget. Said beside the answer, never deciding it |
These come from the observability chart’s stores, so the chart needs to be installed. Without a traffic metric, Nightshift cannot tell whether CPU follows traffic, and says so.
The four answers
Section titled “The four answers”| Answer | When | What Nightshift proposes |
|---|---|---|
| Autoscaler | CPU rises and falls with traffic, and the load splits across pods | A HorizontalPodAutoscaler on CPU utilisation |
| Request | CPU is flat, or does not follow traffic, or one pod does the work, or it is a DaemonSet | A CPU request sized from the measured 95th percentile |
| Both | An autoscaler is right, but today’s request is wrong for one | The request first; the HPA once the request holds. An HPA reads utilisation against the request, so a wrong request makes the HPA wrong too |
| Unknown | Something needed was not read, or the measurements sit between the rules | Nothing on CPU. The answer names the read that would decide it |
The rules
Section titled “The rules”Each rule is a stated number over measured values:
| Rule | Value |
|---|---|
| Enough history to judge a shape | At least 30 samples, an hour at the 2-minute step |
| CPU is flat | Its variation is below 0.15 |
| CPU follows traffic | Correlation of 0.7 or more |
| CPU does not follow traffic | Correlation of 0.3 or less. Between 0.3 and 0.7 is unknown |
| One pod does the work | The busiest pod averages 3 times the others or more |
| The request is wrong for an HPA | Even its quietest pods use more than the request (an HPA would never scale in), or its busiest 5% use under a quarter of it (an HPA would never scale out) |
| A VerticalPodAutoscaler sets CPU | Its own bounds are the request to change. Never an HPA on CPU beside it |
| A one-replica StatefulSet, an uneven split, a loose correlation | Unknown: a second pod sharing the load is measured, not assumed |
What it does with the answer
Section titled “What it does with the answer”The answer is a numbered piece of evidence in the investigation, with its measurements, and Nightshift’s proposal must cite it:
- An HPA is created only when the answer is “autoscaler” and the proposal cites it. Otherwise the proposal is refused, with the reason and what to send instead.
- A larger CPU request is refused when the answer is “autoscaler”: a bigger request would pay for the peak on every pod all day.
- A new HPA waits for a person’s approval, unless a grant that ticks
Add an HPA covers the workload.
Then it may be created on its own, only when the answer is “autoscaler”,
with
minReplicasset to the pods running now andmaxReplicasnever above the grant’s Most replicas, which the grant must set. - A larger CPU request waits for a person’s approval, unless a grant that ticks Raise a CPU request covers the workload. Then it may be raised on its own, only when the answer is “request” or “both”, to at most the measured 95th percentile plus 20%, at most 4 times today’s request, and never above the grant’s CPU ceiling.
- A new HPA is verified by the HPA reporting it can scale and holding utilisation at or under its target (within 10%) for 5 minutes, within 30 minutes. A daily traffic curve may not reach a scale-out inside that time, so a scale-out event is not what is checked.
Raising an existing HPA’s maxReplicas by one step can run on its own under a
one-fix grant.
Not the same as Savings’ raise
Section titled “Not the same as Savings’ raise”Savings also raises sizes, but a different field: when a container presses on its CPU or memory limit, Rightsizing raises the limit. This page is about the CPU request and the replica count, decided from how load behaves. When managed savings and Nightshift’s CPU throttling fix would both raise the same limit on their own, the grants’ rate limit on that workload keeps them from both acting in the same window: whichever comes first acts.