How Google traced GKE API timeouts to a Linux kernel bug

- Reading
- 5 min
In January, Google teams spent weeks trying to understand why the Kubernetes API on GKE was intermittently hanging. The final cause wasn't in any configuration, but in three chained bugs, one of them inside the Linux kernel itself.
The symptom: the Kubernetes API disappears and comes back on its own
In January, a Google Kubernetes Engine (GKE) customer running multiple clusters with Private Service Connect (PSC) for private access to the control plane started seeing something hard to reproduce: pods trying to talk to the Kubernetes API server were getting intermittent timeouts. It wasn't a constant error, nor was it tied to a specific deploy. It was the kind of failure that shows up, disappears, and leaves any platform team paralyzed.
Packet captures showed something strange: the control plane was responding to requests for a NEW TCP connection with stale ACK packets belonging to connections that had already ended. In other words, the server was "confusing" different TCP sessions. This kind of symptom rarely points to a single cause, and this time was no different: Google Cloud identified three distinct bugs, in three separate network layers, that only caused problems when combined.
The case affected roughly 800 developers who were blocked from deploying to production while the investigation was underway, according to the account published by InfoQ.
Three bugs, three layers, one identical symptom
Akash Thakur, senior technical solutions engineer at Google Cloud, describes the scale of the investigation effort:
In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.
Akash Thakur, senior technical solutions engineer at Google Cloud
The three identified causes were:
- Packet reordering in Private Service Connect, which delivered TCP packets out of the expected order.
- Envoy's half-close behavior, which prevented TCP connections from being closed correctly on both sides.
- A bug in the Linux kernel that dropped certain reordered FIN packets, leaving sockets stuck indefinitely in the
FIN_WAIT_2state.
On its own, each of these behaviors is even reasonable: packet reordering happens on any network, half-close is a legitimate connection strategy, and FIN_WAIT_2 handling is a TCP implementation detail. Together, they formed a failure that none of the three teams could see on its own.
Why this is so hard to debug
What makes this case interesting for people working with infrastructure isn't the bug itself, but the reason it took so long to isolate. Thakur explains the core problem of the investigation:
Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — i/o timeout on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. Every layer had a plausible explanation, and every explanation was incomplete.
Akash Thakur, senior technical solutions engineer at Google Cloud
Rohan Karne, managing director at Google Cloud, sums up the kind of work this required:
This is the unglamorous reality of mission-critical platform engineering—methodically ruling out the obvious suspects and digging into packet captures until you find the kernel root cause.
Rohan Karne, managing director at Google Cloud
The fix: rollback, proxy patch, and finally, the kernel
The solution came in three coordinated stages, over several weeks. First, Google rolled back a recent change in Private Service Connect to restore connectivity immediately. Next, it fixed the proxy protocol handling in Envoy and added regression tests to prevent recurrence. The third stage took the longest: a patch to the Linux kernel itself, authored by Eric Dumazet, a former Google software engineer, that fixed the underlying problem in the handling of reordered FIN packets.
That patch was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches. It reached GKE customer clusters starting with version 1.33.5.
The detail every cluster maintainer should notice
Justin Garrison, field CTO at Sidero, drew attention to a point that often goes unnoticed in bug-fix announcements: the lag between the kernel used in production and the upstream kernel.
At first I was excited to read this article that leads to a kernel patch which are always fun. It turns out GKE just runs a kernel from 3 years ago (6.1) and the bug was already fixed upstream and they had to backport it.
Justin Garrison, field CTO at Sidero
This observation matters for anyone running managed Kubernetes (GKE, EKS, AKS) or LTS kernels on on-premise clusters: the "stable version" doesn't always mean "without this specific bug." The normal process of backporting Linux security and stability fixes can take a while to reach a distribution or a managed service, even when the patch has already existed upstream for months.
What changes for Kubernetes operators
The case isn't cause for widespread panic, but it brings practical lessons for anyone managing clusters with a similar network architecture:
- Anyone using Private Service Connect with GKE who saw intermittent, unexplained API timeouts should check whether the cluster is already on version 1.33.5 or later.
- Teams running Envoy as a proxy for long-lived TCP connections get a reminder that half-close behavior needs to be tested under packet reordering, not just under ideal network conditions.
- Any SRE team debating "which layer the problem is in" should note the point raised by Guangba Yu, a postdoctoral researcher at the Chinese University of Hong Kong:
For AIOps and chaos engineering, the transferable method is organizing simultaneous path observations, protocol state, and rollout events while distinguishing symptom recovery from state-machine correction. This anonymized case supplies no population failure rate and does not explain all API timeouts.
Guangba Yu, postdoctoral researcher at the Chinese University of Hong Kong
In other words: correlating network symptoms with rollout events (what changed, when) was what allowed Google to separate the three bugs. This applies both to those who monitor Kubernetes manually and to those building automated AI-driven root-cause pipelines.
What remains open
The original Google Cloud account doesn't disclose the bug's incidence rate outside the affected customer, nor does it guarantee that the same stuck FIN_WAIT_2 pattern won't appear in other proxy and kernel combinations. For anyone running their own infrastructure on Linux (not just GKE), it's worth checking whether the kernel in use has already received Dumazet's patch, available since version 6.9.7 and backported to 6.6 and 6.1 LTS, before dismissing intermittent timeouts as "unexplained network instability."
Translated from the Brazilian Portuguese original · Read the original
Anthropic's IPO tests the circular financing that powers the AI race
Anthropic's stock market debut, on November 19, returns billions to Amazon, Google, Nvidia, and other suppliers that are also investors in the company. The case exposes a model that AI startups in Brazil replicate on a smaller scale, trading equity for cloud credits.






