[{"data":1,"prerenderedAt":497},["ShallowReactive",2],{"blog-list-en":3,"blog-article-en-kubernetes-production-lessons":37},[4,16,27],{"path":5,"title":6,"description":7,"date":8,"updated":9,"tags":10,"cover":9,"coverAlt":9,"icon":9},"\u002Fnews\u002Fblog\u002Fhuman-plus-llm-why-mastering-your-software-is-still-the-best-guarantee","Human + LLM > human alone > LLM alone: why mastering your software is still the best guarantee","The studies agree. A developer working with an LLM produces more than a developer alone, who produces better than an LLM alone. But the ordering holds on one condition, and it is the one that matters to a client.","2026-09-29",null,[11,12,13,14,15],"ai","llm","engineering","quality","security",{"path":17,"title":18,"description":19,"date":20,"updated":21,"tags":22,"cover":9,"coverAlt":9,"icon":9},"\u002Fnews\u002Fblog\u002Fkubernetes-production-lessons","What a Kubernetes cluster asks of you the week after go-live","On 11 December 2024, a telemetry service saturated every one of OpenAI's Kubernetes control planes for 4 hours 22 minutes, after a staging test that passed. Here is what a cluster asks for every week, and what going live never tests.","2026-02-03","2026-09-28",[23,24,25,26],"kubernetes","devops","infrastructure","operations",{"path":28,"title":29,"description":30,"date":31,"updated":21,"tags":32,"cover":9,"coverAlt":9,"icon":9},"\u002Fnews\u002Fblog\u002Ftechnical-leadership-as-a-service","External technical leadership: the decisions you can delegate, and the ones you cannot","Past roughly two and a half days a week, an external technical director costs more than an employee. Here is the arithmetic with its assumptions, the decisions an outsider can genuinely settle, and the ones whose accountability never leaves the executive.","2025-12-10",[33,34,35,36],"strategy","technical-leadership","governance","compliance",{"id":38,"title":18,"body":39,"cover":9,"coverAlt":9,"date":20,"description":19,"extension":487,"icon":9,"meta":488,"navigation":489,"path":17,"seo":490,"stem":491,"tags":492,"translationKey":23,"updated":21,"__hash__":493,"alternates":494},"blog_en\u002Fnews\u002Fblog\u002Fkubernetes-production-lessons.md",{"type":40,"value":41,"toc":476},"minimark",[42,46,55,71,74,86,89,125,143,153,171,188,197,209,216,220,231,235,243,316,319,328,332],[43,44,18],"h1",{"id":45},"what-a-kubernetes-cluster-asks-of-you-the-week-after-go-live",[47,48,49,50,54],"p",{},"On 11 December 2024, every OpenAI service was degraded or unavailable for ",[51,52,53],"strong",{},"4 hours and 22 minutes",". The cause, in the words of their own post-mortem: \"a new telemetry service configuration that unexpectedly generated massive Kubernetes API load across large clusters, overwhelming the control plane and breaking DNS-based service discovery\". The change had been deployed to a staging cluster the day before and tested without incident; fifty-three minutes separate its merge to the main branch from the start of customer impact. Recovery was slow for a reason worth reading twice: the engineers had identified the offending service but could no longer reach the control plane to remove it, because that control plane was the thing the service was saturating.",[47,56,57,58,62,63,66,67,70],{},"One post-mortem proves nothing about your cluster, and this one describes clusters of several thousand nodes that almost nobody runs. What it does establish is a mechanism: the cost of the API call grew with the size of the cluster, a property no staging environment could reveal. Render published the general analysis six days later — the API server is not required to ",[59,60,61],"em",{},"resolve"," DNS, but it is required to ",[59,64,65],{},"initialise"," it, so a saturated control plane stops CoreDNS from restarting; Render infers that mechanism at OpenAI without asserting it, and we cite it with the same caution. The CNCF annual survey published on 20 January 2026 separately measures that ",[51,68,69],{},"82%"," of container users run Kubernetes in production, against 66% in 2023 — a figure we cite with a caveat, as neither the methodology nor the sample size appears on the report's own pages.",[47,72,73],{},"What a cluster costs once nobody is looking is a separate subject, and we will come back to it. This article looks at what it takes to run one: the failures that exist only in production, and the maintenance nobody put on their roadmap.",[75,76,78,79,85],"h2",{"id":77},"five-failures-a-tutorial-never-shows-you","Five ",[80,81,84],"span",{"className":82},[83],"mark","failures"," a tutorial never shows you",[47,87,88],{},"A staging environment validates behaviour, not scale, duration or concurrency. Half of a cluster's properties only exist at its real size, and Kubernetes skill does not change that. Here are the five failures we meet most often, in that order and not by severity.",[47,90,91,94,95,98,99,103,104,107,108,111,112,116,117,120,121,124],{},[51,92,93],{},"1. The 502s on every deployment."," The documentation describes the race plainly: ",[59,96,97],{},"at the same time"," as the kubelet begins shutting the pod down, the control plane evaluates whether to remove it from ",[100,101,102],"code",{},"EndpointSlice"," objects. The process receives its ",[100,105,106],{},"SIGTERM"," while that removal propagates to kube-proxy, to the ingress controller and to client caches; any request arriving in that window hits a closing listener. The fix is a ",[100,109,110],{},"preStop"," that waits out the ",[80,113,115],{"className":114},[83],"propagation delay",", with the subtlety most recipes miss: it is counted ",[59,118,119],{},"inside"," the 30 seconds of ",[100,122,123],{},"terminationGracePeriodSeconds",", not added to them.",[47,126,127,130,131,134,135,138,139,142],{},[51,128,129],{},"2. The CPU limit that degrades instead of protecting."," The kernel enforces ",[100,132,133],{},"limits.cpu"," through the CFS quota, over a ",[51,136,137],{},"100 ms"," window by default. A one-core limit is 100 ms of CPU time per 100 ms window, across all threads: an application with eight threads exhausts its quota in 12.5 ms of wall-clock time, then sits throttled. What you see is staircase latency, average CPU at 25% and no alert, because the average is what dashboards display. There is an important difference: the memory limit ",[59,140,141],{},"is"," necessary, since it is enforced by OOM kill. Our rule: requests everywhere, a memory limit everywhere, a CPU limit only where you bill or arbitrate a noisy neighbour.",[47,144,145,148,149,152],{},[51,146,147],{},"3. etcd, if you run your own control plane."," The default backend quota is ",[51,150,151],{},"2 GiB",". Past it, etcd puts the cluster into a maintenance mode that accepts only reads and deletes: no pod creation, no status update, no lease renewal. The cluster is still running, but paralysed. The lesser-known second step: compaction frees space for etcd, not for the filesystem — you have to defragment, member by member, each unavailable while it runs. The real subject is not the backup, which everyone has; it is the restore into a fresh cluster, which almost nobody has performed.",[47,154,155,158,159,162,163,166,167,170],{},[51,156,157],{},"4. The missing PodDisruptionBudget."," ",[100,160,161],{},"kubectl drain"," respects the PodDisruptionBudgets you have declared, and only those: without one, a drain evicts every replica co-located on a node at once, which default spreading permits: it does exist, at ",[100,164,165],{},"maxSkew: 3"," across hosts, but in ",[100,168,169],{},"whenUnsatisfiable: ScheduleAnyway"," — a wish, not a constraint. The more serious failure is a PDB that allows no voluntary eviction: the drain never finishes, turning an hour of maintenance into a night-time incident.",[47,172,173,176,177,180,181,180,184,187],{},[51,174,175],{},"5. The liveness probe that manufactures the failure it claims to cure."," Under load, the HTTP probe queues behind the same traffic as the application. The defaults are ",[100,178,179],{},"timeoutSeconds: 1",", ",[100,182,183],{},"failureThreshold: 3",[100,185,186],{},"periodSeconds: 10",": thirty seconds of degraded latency is enough for the kubelet to kill a healthy pod, whose load moves to the survivors, which then time out in turn. The documentation carries the warning itself — an incorrect implementation \"can lead to cascading failures\". Corollary: a probe that checks the database turns a database outage into a simultaneous restart of every pod.",[75,189,191,192,196],{"id":190},"the-platform-keeps-changing-and-not-on-your-schedule","The ",[80,193,195],{"className":194},[83],"platform keeps changing",", and not on your schedule",[47,198,199,200,204,205,208],{},"Kubernetes maintains the three most recent minor releases, roughly fourteen months each. As of 28 September 2026: 1.37 released on 26 August 2026, 1.36 on 22 April 2026, 1.35 on 17 December 2025, and everything before that out of upstream support. This is not industry news, it is a subscription — two to three control-plane upgrades a year, each followed by a node roll, each preceded by a check of ",[80,201,203],{"className":202},[83],"deprecated APIs"," in your own manifests ",[59,206,207],{},"and"," in those of your operators.",[47,210,211,212,215],{},"2026 proved the point. On 11 November 2025 the project announced the retirement of ",[100,213,214],{},"ingress-nginx"," in March 2026: no further releases, no bug fixes, no security fixes. The statement from the Steering Committee and the Security Response Committee, on 29 January 2026, leaves no room for doubt: \"choosing to remain with Ingress NGINX after its retirement leaves you and your users vulnerable to attack. None of the available alternatives are direct drop-in replacements.\" The component ran in roughly half of all cloud native environments — Datadog telemetry, so Datadog's installed base — and one or two people maintained it in their spare time. Nobody had \"replace the ingress controller\" on their 2026 roadmap, and half the market got four months.",[75,217,219],{"id":218},"what-does-not-work-and-what-does","What does not work, and what does",[47,221,222,223,230],{},"What does not work: the single architecture review before go-live. It sorts the decisions at the moment you know least, and never comes back. What works is boring, and it is the same triptych as for cost: ",[51,224,225,229],{},[80,226,228],{"className":227},[83],"an owner, a meeting, a measurement","."," A named person, time set aside for every version upgrade, and a written list of the decisions that can no longer be undone. The etcd restore drill belongs in that slot, not in the documentation.",[75,232,234],{"id":233},"do-you-need-a-cluster","Do you need a cluster?",[47,236,237,238,242],{},"We run Kubernetes clusters and it is an expertise we sell. We do not recommend Kubernetes by default. A managed cluster can still suit a small team if its developers already know how to operate it. Some managed offerings go further, with a serverless-like way to run pods: the provider provisions and scales compute based on their declared needs. This reduces node administration; it does not remove the need to understand Kubernetes or design workloads appropriately. No neutral source puts a person-day figure on running a cluster; what follows is our own estimate, derived from the upstream cadence above and from our engagements. Managed, stateless, single-tenant: 2 to 4 person-days a month in steady state, incidents excluded. With stateful workloads, several teams and per-branch environments: 4 to 8. Self-managed, where you own etcd, the certificates and the node operating system: 8 to 15, and that floor is not negotiable because upstream cadence sets it, not your traffic. This ",[80,239,241],{"className":240},[83],"operational cost"," is measured in one person's time, not invoice euros, which is exactly why it stays invisible until the incident.",[244,245,246,259],"table",{},[247,248,249],"thead",{},[250,251,252,256],"tr",{},[253,254,255],"th",{},"Your situation",[253,257,258],{},"Verdict",[260,261,262,274,285,296,306],"tbody",{},[250,263,264,268],{},[265,266,267],"td",{},"Fewer than ten long-running services, one team deploying, the whole thing fits on two to four machines",[265,269,270,273],{},[51,271,272],{},"Not necessarily."," Two VMs, systemd and a reverse proxy are often enough; a managed cluster can still fit if the team already has Kubernetes expertise and a real need for it.",[250,275,276,279],{},[265,277,278],{},"Bursty load, no state to keep, nobody on call",[265,280,281,284],{},[51,282,283],{},"No."," Serverless containers.",[250,286,287,290],{},[265,288,289],{},"Three or more teams deploying independently, twenty to thirty deployable units",[265,291,292,295],{},[51,293,294],{},"Yes."," Orchestration now costs less than the human coordination it replaces.",[250,297,298,301],{},[265,299,300],{},"Unevenly sized workloads to pack, or batch and GPU work to schedule",[265,302,303,305],{},[51,304,294],{}," The only case where bin-packing alone repays the operational cost.",[250,307,308,311],{},[265,309,310],{},"The same platform has to run on public cloud, on premises and on a sovereign cloud",[265,312,313],{},[51,314,315],{},"Yes, and it is the best reason of all.",[47,317,318],{},"Kubernetes is portable as an API; a Kubernetes installation is not. Provider-specific components create lock-in: the ingress controller, CSI driver, identity model and node autoscaler. They belong in any exit plan, and should be among the first things you test.",[47,320,321,322,327],{},"Deciding whether you need the cluster at all, and who carries its operation, is part of ",[323,324,326],"a",{"href":325},"\u002Fen\u002Fnews\u002Fblog\u002Ftechnical-leadership-as-a-service","what you can delegate to technical leadership, and what you cannot",". What to ask of whoever runs your cluster, on the other hand, does not come down to a certification. Rather: \"show me the written list of this cluster's decisions that can no longer be undone, and the date it was last reviewed.\" A team that produces the list understands what it operates. A team that answers that everything is reversible has not had its upgrade yet.",[75,329,331],{"id":330},"sources","Sources",[333,334,335,345,356,367,390,408,446,460],"ul",{},[336,337,338,339],"li",{},"OpenAI, ",[323,340,344],{"href":341,"rel":342},"https:\u002F\u002Fstatus.openai.com\u002Fincidents\u002Fctrsv3lwd797",[343],"nofollow","post-mortem of the 11 December 2024 incident",[336,346,347,348,355],{},"Render, ",[323,349,352],{"href":350,"rel":351},"https:\u002F\u002Frender.com\u002Fblog\u002Fa-hidden-dns-dependency-in-kubernetes",[343],[59,353,354],{},"Breaking down OpenAI's outage: How to avoid a hidden DNS dependency in Kubernetes",", 17 December 2024",[336,357,358,359,366],{},"CNCF, ",[323,360,363],{"href":361,"rel":362},"https:\u002F\u002Fwww.cncf.io\u002Freports\u002Fthe-cncf-annual-cloud-native-survey\u002F",[343],[59,364,365],{},"Annual Cloud Native Survey",", 2025 edition, published 20 January 2026",[336,368,369,370,377,378,383,384,389],{},"Kubernetes, ",[323,371,374],{"href":372,"rel":373},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2025\u002F11\u002F11\u002Fingress-nginx-retirement\u002F",[343],[59,375,376],{},"Ingress NGINX Retirement",", 11 November 2025, and ",[323,379,382],{"href":380,"rel":381},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2026\u002F01\u002F29\u002Fingress-nginx-statement\u002F",[343],"the Steering Committee and Security Response Committee statement",", 29 January 2026, with ",[323,385,388],{"href":386,"rel":387},"https:\u002F\u002Fsecuritylabs.datadoghq.com\u002Farticles\u002Fkubernetes-ingress-nginx-retirement-warning\u002F",[343],"Datadog Security Labs' commentary"," of 19 February 2026",[336,391,369,392,180,397,402,403],{},[323,393,396],{"href":394,"rel":395},"https:\u002F\u002Fkubernetes.io\u002Freleases\u002Fversion-skew-policy\u002F",[343],"version skew policy",[323,398,401],{"href":399,"rel":400},"https:\u002F\u002Fkubernetes.io\u002Freleases\u002Fpatch-releases\u002F",[343],"patch releases"," and ",[323,404,407],{"href":405,"rel":406},"https:\u002F\u002Fkubernetes.io\u002Freleases\u002F",[343],"the release list",[336,409,410,411,402,416,180,421,180,426,402,431,180,436,402,441],{},"Kubernetes documentation: ",[323,412,415],{"href":413,"rel":414},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fconfiguration\u002Fmanage-resources-containers\u002F",[343],"managing container resources",[323,417,420],{"href":418,"rel":419},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Freference\u002Fconfig-api\u002Fkubelet-config.v1beta1\u002F",[343],"the kubelet's CFS quota period",[323,422,425],{"href":423,"rel":424},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F",[343],"pod lifecycle",[323,427,430],{"href":428,"rel":429},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fconfiguration\u002Fliveness-readiness-startup-probes\u002F",[343],"liveness probes",[323,432,435],{"href":433,"rel":434},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Freference\u002Fkubernetes-api\u002Fworkload-resources\u002Fpod-v1\u002F",[343],"their default values",[323,437,440],{"href":438,"rel":439},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fscheduling-eviction\u002Ftopology-spread-constraints\u002F",[343],"topology spread constraints",[323,442,445],{"href":443,"rel":444},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Frun-application\u002Fconfigure-pdb\u002F",[343],"PodDisruptionBudget",[336,447,448,449,454,455],{},"AWS, ",[323,450,453],{"href":451,"rel":452},"https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Ffargate.html",[343],"running EKS pods on Fargate","; Google Cloud, ",[323,456,459],{"href":457,"rel":458},"https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fautopilot-overview",[343],"GKE Autopilot overview",[336,461,462,463,402,468],{},"etcd, ",[323,464,467],{"href":465,"rel":466},"https:\u002F\u002Fetcd.io\u002Fdocs\u002Fv3.5\u002Fop-guide\u002Fmaintenance\u002F",[343],"maintenance",[323,469,472,473],{"href":470,"rel":471},"https:\u002F\u002Fgithub.com\u002Fetcd-io\u002Fetcd\u002Fblob\u002Fmain\u002Fserver\u002Fembed\u002Fconfig.go",[343],"the default value of ",[100,474,475],{},"--quota-backend-bytes",{"title":477,"searchDepth":478,"depth":478,"links":479},"",2,[480,482,484,485,486],{"id":77,"depth":478,"text":481},"Five failures a tutorial never shows you",{"id":190,"depth":478,"text":483},"The platform keeps changing, and not on your schedule",{"id":218,"depth":478,"text":219},{"id":233,"depth":478,"text":234},{"id":330,"depth":478,"text":331},"md",{},true,{"title":18,"description":19},"news\u002Fblog\u002Fkubernetes-production-lessons",[23,24,25,26],"cMy9yruMc1lZsJ5yRLsJaQ3Y0xXpnhUxuWYWd-5h53A",{"en":495,"fr":496},"kubernetes-production-lessons","kubernetes-en-production",1790796926718]