Skip to main content
When a container is killed due to an out-of-memory (OOM) event, Thoras can automatically increase the container’s memory request and limit to stabilize the workload. This prevents a workload from cycling through repeated OOM kills while waiting for a scheduled forecast to produce a sufficient memory recommendation. OOM remediation requires autonomous vertical scaling to be enabled. It is opt-in per workload.

​
How It Works

When OOM kills are detected on a workload, Thoras:
  1. Bumps memory requests and limits for affected containers by a 1.2x multiplier, computed independently from the average current memory request and limit across all running pods. Both request and limit are raised since the kernel OOM-kills based on the limit, and request guarantees node memory. This applies to regular containers and native sidecar init containers.
  2. Applies the adjustment to running pods and any newly created pods. For running pods, Thoras uses your update_policy: it attempts an in-place resize first (if enabled). When that does not apply and your policy allows recreating pods, Thoras replaces the remaining pods gradually, in batches paced by your disruption settings, rather than restarting the whole workload at once. See Rolling Out the Adjustment. Any pods created or rescheduled in the meantime (e.g., evictions, scale-out) automatically receive the adjusted memory.
  3. Repeats as long as OOM kills continue, compounding each time (1.2x, then 1.44x, then 1.73x, and so on) until OOMs stop. Thoras waits at least 2 minutes between adjustments, and when the adjustment is rolled out in batches it also waits until every pod has the current adjustment before applying the next one. If a memory.upperbound is configured, adjustments are capped at that value (see Upper Bounds).
  4. Holds the memory floor for a configurable stabilization window after the last OOM. During this window, forecasts can raise memory above the floor but cannot lower it. This prevents a new forecast, which may not yet reflect the OOM episode, from reversing the adjustment prematurely.
  5. Returns full control to the forecaster once the stabilization window expires. If OOMs recur, the cycle begins again.
The adjustment is a short-term stabilization measure, not a permanent right-sizing decision. The goal is to keep the workload running long enough for the forecaster to observe the higher memory usage and incorporate it into future recommendations.

​
Rolling Out the Adjustment

New pods pick up the adjusted memory as soon as it is set. For pods that are already running, what Thoras does depends on your update_policy: In-place resize “does not apply” when the resize fails (for example, the node lacks capacity), is not supported by the cluster, or includes a change the cluster can only make by restarting the pod (such as lowering a memory limit). Gradual replacement needs memory in recreate_resources, which is the default. If recreate_resources leaves out memory (for example ["cpu"]), Thoras falls back to a rolling restart of the workload instead, when your update_mode allows recreating pods. When pods are replaced gradually:
  • Batches follow your disruption settings. Thoras evicts at most max_disrupted pods at a time and waits interval between batches. Evictions go through the Kubernetes eviction API, so your PodDisruptionBudgets are honored. Thoras does not modify your Deployment or StatefulSet, so GitOps tools see no drift.
  • Crash-looping pods are replaced first. A pod that is already down because it keeps OOMing goes in the next batch even when max_disrupted is used up, so it cannot stall the rollout.
  • No forecast is needed. The rollout proceeds even when the workload has no usable forecast yet, such as a new workload still collecting metrics.
  • Only pods below the adjusted memory are replaced. Your scale-up threshold does not apply here, and a pod that already has at least the adjusted memory is never replaced for the OOM adjustment.
  • One adjustment at a time. Pods still waiting for their turn may OOM again at the old memory. Those OOMs do not trigger another adjustment until every pod has the current one, so a single episode does not compound on top of itself while the rollout is still in progress.
During a gradual replacement, you can follow progress with the same signals as any other paced rollout: the DisruptionInProgress condition on the AIScaleTarget, which reads Complete once every pod has the adjusted memory, and the thoras_ast_disruption_completions_total metric. See In-Place Resize Behavior for the full list.

​
Enabling OOM Remediation

Add oom_remediation to your spec.vertical configuration:
oom_remediation.enabled must be true and spec.vertical.mode must be autonomous. Workloads in recommendation mode are not affected.

​
Stabilization Window

After Thoras applies an OOM memory adjustment, it holds a memory floor for the duration of the stabilization window. During this window:
  • Incoming forecasts cannot lower memory below the adjusted value.
  • Forecasts can raise memory above the adjusted value if the forecast recommends it.
  • Each new OOM resets the stabilization window, extending it from the time of the most recent adjustment. The workload must be stable for the full window duration before memory may be scaled down.
Once the window expires, Thoras clears the floor and the forecaster resumes full control of memory sizing. The stabilization window should be long enough for the forecaster to observe memory usage at the adjusted level, so future recommendations account for the higher usage that triggered the OOMs. Default duration: max(forecast_interval, 4h). In most configurations, the 4-hour minimum provides sufficient coverage across multiple forecast cycles. To override the default, set stabilization_window explicitly:

​
Upper Bounds

When memory.upperbound is configured, OOM adjustments are capped at that value. If the 1.2x multiplier would push the memory request above the upper bound, Thoras sets the request to the upper bound instead. When the adjustment is capped, Thoras sets an OOMAdjustmentCapped condition on the AIScaleTarget with the message “OOM memory adjustment capped at UpperBound for one or more containers.” This indicates that OOM remediation has reached the configured ceiling and cannot provide more memory. If OOMs continue, consider raising the upper bound to give remediation more headroom. Once the stabilization window expires, the forecaster regains full control and steady-state recommendations are clamped to upperbound as usual.

​
Memory Limits

When an OOM adjustment fires, Thoras raises both the memory request and the memory limit by the same 1.2x multiplier. This preserves the existing request:limit ratio on the way up, giving the workload room to recover without changing its proportions. During the stabilization window, the OOM-adjusted limit acts as a floor. If memory.limit.ratio is configured, Thoras computes the limit from the new request using the ratio and takes the higher of the two values. After the stabilization window expires:
  • With memory.limit.ratio: the limit follows the ratio normally as the forecaster regains control. If you use limit.ratio: "infer", Thoras derives the ratio from your deployment template’s current request:limit ratio. See Vertical Pod Rightsizing for details on configuring limit.ratio.
  • Without memory.limit.ratio: the limit reverts to the value in your workload’s pod template (deployment, statefulset, etc.). If the workload’s steady-state memory usage exceeds that template limit, OOMs will recur each time the limit reverts. The resolution is to raise the limit in the pod template, or to configure memory.limit.ratio so Thoras can manage limits directly.

​
Interaction with Forecasts

During an active stabilization window, Thoras applies max(forecast, floor) to memory when scaling. CPU is always sourced from the forecast unchanged. When a new forecast arrives after the window has expired, the floor is cleared and the forecast value is applied directly. If the workload OOMs again, the remediation cycle restarts from the new forecast baseline.

​
Considerations

Memory may be held above the forecast during the stabilization window. If an OOM was caused by a one-off spike and the workload returns to lower memory usage afterward, requests will remain elevated until the window expires. Thoras favors avoiding another OOM over reclaiming memory immediately, and allows configuring the duration of the stabilization window. New pods after the window expires start at the forecast value. Once the floor clears, pods created from that point forward use the forecaster’s recommendation. If that recommendation is insufficient, another OOM may occur and trigger a new remediation cycle. Partial in-place resize failures. If some pods cannot be resized in place (e.g., the node lacks capacity), Thoras replaces those pods gradually when your update_policy allows recreating them. With update_mode: in_place, they keep their previous memory until a later resize succeeds. Recovery is paced by your disruption settings. Replacing every pod takes roughly replicas / max_disrupted batches, interval apart, and the next adjustment waits until the current one has reached every pod. A pod that a PodDisruptionBudget keeps from being evicted delays the rollout, and the next adjustment, until it can be replaced. For workloads that need to recover faster, raise max_disrupted or shorten interval.

​
Full Example

In this configuration:
  • OOM remediation is enabled with a 6-hour stabilization window
  • Memory requests are capped at 4 GiB (upperbound) during both steady-state and OOM remediation
  • Pods are resized in place where possible; otherwise they are replaced gradually in disruption batches