AWS FOR THE REAL WORLD
⏱️
Reading time: 11 minutes
🎯
Main Learning: A Karpenter NodePool is a placement policy, not an instance preference. Pin one instance
family, and a Spot shortage moves your fleet across availability zones, where every internal call starts costing $0.01 per GB.
Hey 👋🏽
Important off topic things first: Sandro got married! 🎉
We were in Munich for it and it was a fantastic day! ☀️
Highly recommend a wedding over sprint planning or fighting with AWS shenanigans!
Now to something less fun: A story I keep hearing, and it always starts the same way.
A cluster gets noticeably more expensive over two days.
1. Nothing was deployed and traffic is flat.
2. All dashboards stay green: pods Running, nodes Ready, no alerts and no errors.
3. But the bill still goes up... significantly 😅
The only thing that changed is where the nodes are running. That's enough.
In this issue: how a Spot shortage quietly rearranges a cluster across three availability zones, and why that makes internal traffic a costly liability!
Sponsored
|
Search without the cluster
Typesense — the open source alternative to Algolia
|
Typo-tolerant search from a single binary. No shard count to pick, no JVM heap to tune, no cluster to keep alive — and natural language
search is built in.
Smaller learning curve than Elasticsearch • Django & Laravel Scout integrations
Check it out on GitHub →
Sponsored by Typesense, a tool we run ourselves.
📚 This Week's Deep Dive
Three things are true about a lot of EKS clusters using Karpenter (great choice!) as their scheduler:
1. One instance family, benchmarked and standardized across teams
2. Spot preferred, with On-Demand as the fallback
3. No zone-aware routing, because while all the capacity happens to live in one zone, "route anywhere" and "route locally" are the same instruction.
Then Spot capacity for that family runs out in the region. Karpenter does exactly what it was told: it falls back to On-Demand, and it
places new nodes wherever it can still get instances. Within a day, a fleet that had effectively lived in one availability zone is
spread across three.
Nothing about that is wrong. Spreading across zones is what you are supposed to want. Except the services still talk to each other
constantly, and now roughly two thirds of every internal call crosses a zone boundary at $0.01 per GB in each direction.
The traffic pattern never changed. Only the placement did. And there is no Kubernetes signal for placement: no restarts, no failing
probes, zero errors. In Cost Explorer it hides inside EC2-Other.
The full story is on the blog: how a NodePool quietly became a placement policy, why one instance family is the thing that set it up,
and the four changes that stop it from happening again.
|
👋 Recommended reading — not a sponsor
10 Most Common AWS Savings Opportunities
Not a sponsor and not paid, we just know the team and rate the work. No affiliate link either. They went through more than 100 AWS accounts and wrote
up what they actually found, with how often each thing shows up and the median yearly waste attached to it.
Oversized EC2 instances in 46% of accounts. Redundant NAT gateways in another 46%. Unused EKS clusters at a median $876 a year. Every item comes with
the CLI commands to check your own account, which is what makes it worth the 18 minutes.
Read the breakdown →
📰 This Week in AWS
🩺EC2 application status checks
|
EC2 now calls your app over HTTP or HTTPS and fails the check when it does not answer. You pick protocol, port, path, and
the healthy response codes, and Auto Scaling can replace the instance on that signal. One less custom health Lambda.
|
Read More →
|
🔐The AWS sign-in page is changing
|
Unified email entry for root users, sign-in with Google, GitHub, Apple or Amazon, and a session picker when you have several
accounts open. Rolling out gradually. If you script or automate console login anywhere, test it now.
|
Read More →
|
💰Managed dashboards in Billing and Cost Management
|
Five preconfigured read-only dashboards for compute, databases, reservations, and Savings Plans, with no setup at all.
Fitting timing for this issue: go look at yours right after you finish reading.
|
Read More →
|
|
Go read your own NodePool with this in mind!
If it names exactly one instance family, you already know what to do.
And if you have had a cost surprise that no dashboard warned you about, hit reply and tell me.
I collect these!
See you next week! 🙏
Tobi & Sandro
|