update_mode: in_place or
update_mode: in_place_or_recreate, Thoras patches running pods directly
through the Kubernetes resize API instead of recreating them. This page explains
what happens when that resize cannot complete and how Thoras responds depending
on your update policy.
For configuration details, see
Update Policy and the
AIScaleTarget reference.
When a Resize Cannot Complete
After Thoras patches a pod’s resource requests, the kubelet attempts to actuate the change by adjusting the container’s cgroup. Two situations can prevent this:- Insufficient node capacity. An upsize requires more CPU or memory than the node has available. The kubelet cannot expand the cgroup and the resize stalls.
- Memory limit below live usage. A memory limit decrease that would drop below the container’s current working set is accepted by the API server (the patch succeeds), but the kubelet cannot safely shrink the cgroup. It retries on every sync cycle.
PodResizeInProgress
condition on the pod with Reason: Error. The pod continues running with its
previous resource allocation. No data is lost and no containers are restarted by
the stall itself.
Behavior by Update Mode
How Thoras responds to a stalled resize depends onupdate_mode:
in_place
Thoras never evicts or recreates the pod. If the kubelet cannot complete the
resize, the pod stays running with its current resources. The stall resolves on
its own when conditions change (for example, other pods are descheduled and free
up node capacity, or a new forecast lowers the recommendation).
This mode is appropriate for workloads where continuity is more important than
converging to the recommendation quickly.
in_place_or_recreate
Thoras attempts in-place resize first. If a pod’s resize stalls for more than
two minutes, Thoras evicts that specific pod so it can be recreated with the
correct resources by its controller (Deployment, StatefulSet, etc.). Only the
stalled pods are evicted; pods that resized successfully are left alone.
This is the recommended mode for most workloads because it combines the
low-disruption benefits of in-place resize with a reliable fallback.
Safety Controls
Thoras applies several layers of protection when evicting pods as part of the recreate fallback.PodDisruptionBudgets
Evictions use the Kubernetes Eviction API, which enforces your PodDisruptionBudgets. If a PDB would be violated, the API server rejects the eviction and Thoras defers the remaining pods to a later run. This is different from a rollout restart, which replaces the entire ReplicaSet and is not gated by PDBs at the individual pod level.Disruption batching
Evictions are paced using thedisruption settings on the
AIScaleTarget. Thoras applies changes in batches (default: up to 20% of replicas
at a time, with at least 60 seconds between batches) and counts pods that are
already terminating or not-Ready against the budget. This prevents stacking
disruption when a prior batch has not finished rolling.
Circuit breaker
If pods keep coming back stale after eviction for the same recommendation (for example, because the admission webhook is not applying the new values), Thoras stops evicting. The circuit breaker opens after repeated no-progress cycles and backs off exponentially before probing again. This prevents runaway eviction loops.Operational Guidance
Concurrent upsizes on the same node
When multiple pods on the same node receive an upsize at the same time, the node may not have enough capacity for all of them. Thoras handles each pod independently: pods that resize successfully stay in place, and pods that stall follow the fallback path for the configuredupdate_mode.
To reduce the likelihood of node-capacity stalls:
- Reserve headroom on nodes. If you use a node autoscaler like Karpenter, consider configuring a small amount of reserved capacity so upsizes have room to land without waiting for a new node.
- Use
disruptionsettings to pace changes. Smaller batch sizes and longer intervals give the node autoscaler time to respond between batches. - Rely on forecasting. Thoras predicts resource needs ahead of time, so upsizes are typically applied before a spike rather than during one, reducing contention.