Kubernetes 1.37 ends pod resize stuck in Deferred state

In-place pod resize reached GA in Kubernetes v1.35, allowing CPU and memory adjustments for running containers without restarting the pod. Good for anyone running an in-memory database, a web server, or anything that can't blink. But it was born with a hole: if the node didn't have slack (allocatable headroom) to accommodate the increase, the Kubelet marked the request as Deferred and the pod just sat there, waiting for the node to free up space on its own. It could wait forever.

Kubernetes v1.37, according to the official blog announcement, closes that gap with scheduler preemption for in-place pod resize, in alpha behind the InPlacePodVerticalScalingSchedulerPreemption feature gate. The idea: when a high-priority pod needs to grow and the node is full, the scheduler evicts lower-priority pods on that same node to make room, and the resize happens without a restart.

The Deferred problem in practice

When a controller (the Vertical Pod Autoscaler, for example) raises the requests of a running container, the Kubelet checks whether the node has spare allocatable capacity. If it doesn't, the resizeStatus in status.containerStatuses[] becomes Deferred. It's worth distinguishing this from Infeasible: Infeasible is an immediate rejection because the request exceeds the machine's physical limits, the namespace's LimitRange, or the admission quota. Deferred is a valid request that simply can't be executed right now.

The cruel detail is that, before this feature, Deferred could be permanent. A critical application about to hit OOM would sit stuck waiting, and the operator had three options, all bad: manually evict low-priority pods, rely on the cluster autoscaler to bring up a bigger node (which reschedules the pod and violates the very "no restart" promise), or write a custom autoscaler that resizes the node. The kube-scheduler simply couldn't see deferred resizes on already-running pods, so it couldn't use the priority-based preemption it already knows how to do during initial scheduling.

For SRE teams, this was a classic density-versus-reliability dilemma. Packing nodes with batch jobs and best-effort tasks (the famous bin-packing) improves utilization and cuts costs, but turns the slack that critical workloads would need into a risk. Either the cluster ran with idle buffer, or the important application had no room to scale during a traffic spike. You couldn't have both.

How the scheduler now acts

The mechanics integrate into the normal scheduling cycle. Normally, a pod with spec.nodeName filled in is considered placed and doesn't go back into the active queue. With the feature gate enabled, the scheduler intercepts pods carrying the Deferred condition and keeps them under active evaluation specifically to trigger preemption, tracking each one until the Kubelet completes the resize.

Some architectural points that change behavior and matter when operating this:

• Preemption bound to the node. Unlike placement preemption, which scans the whole cluster looking for the best fit, here the search for "victims" is restricted to the node where the deferred pod already runs. If, even after evicting all eligible low-priority pods on that node, capacity is still short, the resize remains Deferred. There's no migration to another node.
• Resource reservation. To avoid a race and double allocation, the scheduler treats the resource requested in the resize as already consumed, which gives the Kubelet the guarantee to execute as soon as preemption frees up space.
• Separation of responsibilities. The Kubelet has a critical Pod admission handler that, at admission time, can locally evict pods to guarantee critical workloads. With the feature gate enabled, that handler no longer does local preemption for resize operations: it defers and hands the decision entirely to the scheduler. A single centralized orchestrator now governs all resize preemption logic, respecting global priorities, Pod Disruption Budgets, and graceful termination.
• Races between resizes. If, during an active preemption cycle, an even higher-priority resize request arrives for another pod on the same node, the Kubelet prioritizes that request and the scheduler triggers a new preemption round if more capacity is needed.

There's also a brake at the node level. A new spec.podPreemptionPolicy field on Node allows disabling resize preemption for specific nodes:

apiVersion: v1
kind: Node
metadata:
  name: batch-workload-node
spec:
  podPreemptionPolicy:
    disableResizePreemption:
    - "cluster-autoscaler.kubernetes.io/disable-preemption"
    - "operator.example.com/policy-override"

The use case cited in the blog is a controller that prefers to scale down other pods or adjust the node's capacity on its own, leaving scheduler preemption as a last resort. It makes sense: evicting workloads is the cheapest solution in terms of latency, but the most disruptive one. Having a switch to turn it off on batch nodes is the kind of explicit trade-off that avoids surprises.

Seeing it work on a local kind cluster

The source includes a reproducible mini-tutorial on a single-node kind cluster with limited CPU. The gate is enabled at cluster creation:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  InPlacePodVerticalScalingSchedulerPreemption: true

kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0

The example node exposes 8 allocatable CPUs. The script creates two PriorityClass objects (high-priority with value 1000000 and low-priority with 1000), spins up a low-priority pod requesting 3 CPUs and a high-priority one requesting 4, consuming 7 of the 8 CPUs and leaving 1 of slack. Then comes the test: a patch raises the high-priority pod's requests/limits from 4 to 6 CPUs, a delta of +2 that doesn't fit in the 1 CPU of slack.

kubectl patch pod high-priority-pod --subresource resize --patch \
 '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'

The event cycle tells the whole story. On the low-priority pod, a Preempted appears followed by Killing. On the high-priority one, the sequence goes ResizeDeferred (with the error OutOfcpu: Node didn't have enough resource: cpu, requested: 6000, used: 3950, capacity: 8000), then ResizeStarted when capacity is freed up, and finally ResizeCompleted. The confirmation:

kubectl get pod high-priority-pod -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}{"\n"}'
# 6

The container went from 4 to 6 CPUs by updating cgroup limits via the container runtime, without restarting. That's the point: a ResizeCompleted without RESTARTS incrementing is what separates this feature from ordinary rescheduling.

What changes for those running Kubernetes clusters in Brazil

In practice, the takeaway is that you can push bin-packing further with less fear. Filling node slack with batch jobs, background processing, and best-effort tasks stops being a bet against the critical workload's peak, because the scheduler now has a way to reopen space when that workload needs to grow. For those paying the bill for idle nodes just to keep a buffer (and cloud exchange rates make that weigh heavily), the promise is higher utilization without sacrificing the responsiveness of sensitive applications.

That said, it's alpha, and alpha doesn't go to production. The feature gate needs to be enabled on kube-apiserver, kube-scheduler, and kubelet, on the control plane and on all workers running v1.37 or later. It's worth understanding the behavior well before trusting it: since preemption is bound to the node, a resize can remain Deferred even when the cluster has space elsewhere, and you need to be sure PDBs and priorities are configured so that the evicted workload really is disposable. The bar stays the same: if you can't measure the effect of preemption and roll back the behavior (via spec.podPreemptionPolicy), it doesn't go on the cluster that matters. The sensible path is the one SIG Scheduling itself asks for: enable it in a test environment, watch the events, and give feedback before any production plan.