The deployment is green. Latency is normal. No pods are crash-looping, and the on-call channel has been quiet for days.
Then finance forwards the cloud bill.
That disconnect catches teams because Kubernetes health checks answer an operational question: can the workload run?
They don’t answer the financial one: how much capacity did the cluster buy, and how much did the application actually need?
A cluster can be stable, responsive, and wasteful at the same time. Finding the waste starts with separating what the nodes provide, what workloads request, and what they use.
A healthy dashboard can hide expensive decisions
Most cluster dashboards are designed to surface failure. They show unavailable replicas, restarts, CPU pressure, memory pressure, pending pods, and unhealthy nodes. If those indicators are calm, the cluster looks well managed.
Cost often lives in the gaps those dashboards don’t emphasize. A cluster may provide 80 virtual CPUs, while its workloads request 48 and consume 14 during a normal week.
A Kubernetes cloud cost monitoring view that separates provisioned, requested, and used resources by namespace and workload exposes the distance between those figures. The bill pays for the infrastructure at the top of that chain, not just the CPU cycles at the bottom.
That doesn’t mean 14 used CPUs should become 14 provisioned CPUs. Production systems need room for bursts, rolling deployments, node failures, and uneven traffic. The question is whether that headroom reflects a deliberate reliability decision or an accumulation of old guesses.
A quick kubectl top pods -A check won’t settle it. The official kubectl top documentation describes a recent CPU and memory view built primarily for autoscaling signals, not historical cost reporting.
It can identify a busy pod now, but not a staging namespace that sat nearly idle on large nodes for the previous month.
Compare at least a week of usage against requests, node capacity, replica counts, and spend. A snapshot during lunch traffic can make oversized requests look sensible; one at 3 a.m. can make necessary capacity look ridiculous.
Start with requests, not the largest CPU bar
When a bill jumps, engineers often sort pods by CPU usage and investigate the busiest workload. That helps with performance troubleshooting, but the most expensive workload may be the one asking for far more than it uses.
Kubernetes schedules pods according to resource requests. As the Kubernetes resource-management documentation explains, the scheduler checks requested CPU and memory against node capacity even when actual usage is low. Requests are capacity claims, not harmless notes in a YAML file.
Consider an API deployment with 12 replicas. Each pod requests one CPU but averages 120 millicores outside brief peaks.
In scheduler math, it claims 12 CPUs while normally using about 1.44. Those requests may retain extra nodes, and every additional node can carry DaemonSets for logging, security, networking, and monitoring.
Three commands give a useful starting picture:
kubectl top pods -A --sort-by=cpukubectl get pods -A \ -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory'kubectl describe node <node-name>Read them together. The first shows recent consumption, the second exposes declared requests, and the third shows how much allocatable node capacity has been claimed. The mismatch tells you where deeper investigation is worth the time.
Node-level inspection matters when Kubernetes data looks strange. A system daemon, container runtime issue, or logging process can consume resources outside the application view, so checking running Linux processes with ps, top, or htop can explain why a node appears busier than its pods suggest. Not every unexplained spike belongs to a deployment.
Don’t cut every request to the seven-day average. A workload averaging 120 millicores may hit 900 during an import job.
Memory is less forgiving because a container can run comfortably for hours, cross its limit during one large operation, and be killed. Check peaks, restart history, and application behavior before changing values.
Check what changed before you resize anything
A cost increase is a change-management problem before it’s an optimization problem. Something became larger, ran longer, spread further, or moved onto a more expensive resource.
Find the first day spend moved outside its normal range, then compare deployment activity around it. Replica changes, new node groups, instance upgrades, GPU requests, persistent volumes, retention settings, and cross-zone traffic can all increase the bill without producing a health alert.
A familiar example is a temporary scaling change for a product launch. The deployment moves from six replicas to eighteen, the event ends, and the minimum never returns to six.
The cluster remains healthy because the extra pods make it comfortable. The mistake survives because nothing breaks.
Autoscaling can create a less obvious version. The Horizontal Pod Autoscaler calculates CPU utilization relative to resource requests.
If a pod uses 350 millicores against a 500-millicore request, that appears as 70%; lower the request to 250 without changing the target, and the same load appears as 140%. A rushed rightsizing change can trigger more replicas and replace one kind of waste with another.
Roll out memory changes slowly. Lower requests and limits on one low-risk workload, watch several traffic cycles, and check OOM events, throttling, pending pods, latency, and restarts.
When failures appear, the log-first workflow used for debugging Kubernetes container exit errors is better than immediately restoring every old value. Determine whether the new boundary exposed a peak, a leak, or a bad assumption.
Also check whether the increase belongs to compute at all. Unattached disks, oversized persistent volumes, old snapshots, load balancers, public IPs, and data transfer can grow while CPU charts look ordinary.
Make cost visible before it becomes a finance ticket
The best cost reviews don’t begin after a surprising bill. They give engineers enough context to notice a bad change while it’s still small.
That requires ownership. “Platform” is not a useful owner for every shared expense, and a namespace such as production tells finance little.
Labels and allocation rules should connect workloads to a team, service, and environment. The goal is being able to ask a specific person why one workload’s daily cost doubled.
A practical weekly review can stay brief:
- Rank namespaces and workloads by cost, then compare them with the previous week.
- Flag large gaps between requested and used CPU or memory.
- Check changes in replicas, nodes, instance types, storage, and network charges.
- Review idle non-production resources separately from production headroom.
- Assign an owner and review date before changing requests or limits.
Trends matter more than isolated red numbers. A batch worker that rests for 22 hours and consumes several CPUs for two may be correctly sized. A service using 8% of requested CPU every hour for a month deserves attention. Context separates intentional spare capacity from forgotten capacity.
Linux host metrics provide another reality check. If Kubernetes reports memory pressure but workload usage doesn’t explain it, compare the node view with free, /proc/meminfo, and top; CommandLinux’s guide to checking RAM in Linux explains why available memory is more useful than the completely unused figure. Cache and system services can otherwise lead to a bad decision.
Cost alerts should point to technical evidence. “Spend is up 20%” creates a meeting. “The payments namespace added nine replicas, requested CPU rose 11 cores, and cross-zone traffic doubled after Tuesday’s release” creates an investigation. That level of detail turns cost into an operational signal.
Wrap-up takeaway
A green cluster only proves that the current configuration is keeping workloads alive. It doesn’t prove that the capacity underneath them is being used well or that anyone remembers why it was provisioned.
Compare provisioned, requested, and actual usage over a meaningful window, then trace the largest gaps to deployments, autoscaling, nodes, storage, and network activity. Treat headroom as a reliability choice with an owner and a reason, not as waste by default.
Make changes gradually, as careless rightsizing can lead to throttling, OOM kills, or unnecessary scale-outs.
Today, export the ten most expensive workloads, place their CPU and memory requests beside seven-day usage, and assign an owner to review the three largest gaps.
