> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thoras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# In-Place Resize Behavior

> How Thoras handles in-place pod resizing, what happens when a resize cannot complete, and the safety controls that protect your workloads.

When vertical scaling is configured with `update_mode: in_place` or
`update_mode: in_place_or_recreate`, Thoras patches running pods directly
through the Kubernetes resize API instead of recreating them. This page explains
what happens when that resize cannot complete and how Thoras responds depending
on your update policy.

For configuration details, see
[Update Policy](/guides/vertical-pod-rightsizing#update-policy) and the
[AIScaleTarget reference](/reference/ast-definition#update-policy).

## When a Resize Cannot Complete

After Thoras patches a pod's resource requests, the kubelet attempts to actuate
the change by adjusting the container's cgroup. Two situations can prevent this:

* **Insufficient node capacity.** An upsize requires more CPU or memory than the
  node has available. The kubelet cannot expand the cgroup and the resize
  stalls.
* **Memory limit below live usage.** A memory limit decrease that would drop
  below the container's current working set is accepted by the API server (the
  patch succeeds), but the kubelet cannot safely shrink the cgroup. It retries
  on every sync cycle.

In both cases, Kubernetes reports the stall through a `PodResizeInProgress`
condition on the pod with `Reason: Error`. The pod continues running with its
previous resource allocation. No data is lost and no containers are restarted by
the stall itself.

## Behavior by Update Mode

How Thoras responds to a stalled resize depends on `update_mode`:

### `in_place`

Thoras never evicts or recreates the pod. If the kubelet cannot complete the
resize, the pod stays running with its current resources. The stall resolves on
its own when conditions change (for example, other pods are descheduled and free
up node capacity, or a new forecast lowers the recommendation).

This mode is appropriate for workloads where continuity is more important than
converging to the recommendation quickly.

### `in_place_or_recreate`

Thoras attempts in-place resize first. If a pod's resize stalls for more than
two minutes, Thoras evicts that specific pod so it can be recreated with the
correct resources by its controller (Deployment, StatefulSet, etc.). Only the
stalled pods are evicted; pods that resized successfully are left alone.

This is the recommended mode for most workloads because it combines the
low-disruption benefits of in-place resize with a reliable fallback.

## Safety Controls

Thoras applies several layers of protection when evicting pods as part of the
recreate fallback.

### PodDisruptionBudgets

Evictions use the
[Kubernetes Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/),
which enforces your PodDisruptionBudgets. If a PDB would be violated, the API
server rejects the eviction and Thoras defers the remaining pods to a later run.
This is different from a rollout restart, which replaces the entire ReplicaSet
and is not gated by PDBs at the individual pod level.

### Disruption batching

Evictions are paced using the
[`disruption`](/reference/ast-definition#disruption) settings on the
AIScaleTarget. Thoras applies changes in batches (default: up to 20% of replicas
at a time, with at least 60 seconds between batches) and counts pods that are
already terminating or not-Ready against the budget. This prevents stacking
disruption when a prior batch has not finished rolling.

### Circuit breaker

If pods keep coming back stale after eviction for the same recommendation (for
example, because the admission webhook is not applying the new values), Thoras
stops evicting. The circuit breaker opens after repeated no-progress cycles and
backs off exponentially before probing again. This prevents runaway eviction
loops.

## Operational Guidance

### Concurrent upsizes on the same node

When multiple pods on the same node receive an upsize at the same time, the node
may not have enough capacity for all of them. Thoras handles each pod
independently: pods that resize successfully stay in place, and pods that stall
follow the fallback path for the configured `update_mode`.

To reduce the likelihood of node-capacity stalls:

* **Reserve headroom on nodes.** If you use a node autoscaler like Karpenter,
  consider configuring a small amount of reserved capacity so upsizes have room
  to land without waiting for a new node.
* **Use `disruption` settings to pace changes.** Smaller batch sizes and longer
  intervals give the node autoscaler time to respond between batches.
* **Rely on forecasting.** Thoras predicts resource needs ahead of time, so
  upsizes are typically applied before a spike rather than during one, reducing
  contention.

### Choosing between `in_place` and `in_place_or_recreate`

| Consideration     | `in_place`                                                              | `in_place_or_recreate`                                                                                           |
| ----------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Pod continuity    | Guaranteed. Pods are never evicted by Thoras.                           | Pods may be evicted if resize stalls.                                                                            |
| Convergence speed | Slower. Stalls resolve only when external conditions change.            | Faster. Stalled pods are recreated with correct resources.                                                       |
| Best for          | Stateful workloads, long-running jobs, workloads sensitive to restarts. | General-purpose services where converging to the recommendation matters more than avoiding a single pod restart. |