> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thoras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OOM Remediation

> Automatically recover workloads from out-of-memory kill loops

When a container is killed due to an out-of-memory (OOM) event, Thoras can
automatically increase the container's memory request and limit to stabilize the
workload. This prevents a workload from cycling through repeated OOM kills while
waiting for a scheduled forecast to produce a sufficient memory recommendation.

OOM remediation requires
[autonomous vertical scaling](/guides/vertical-pod-rightsizing) to be enabled.
It is opt-in per workload.

## How It Works

When OOM kills are detected on a workload, Thoras:

1. **Bumps memory requests and limits** for affected containers by a 1.2x
   multiplier, computed independently from the average current memory request
   and limit across all running pods. Both request and limit are raised since
   the kernel OOM-kills based on the limit, and request guarantees node memory.
2. **Applies the adjustment** to running pods and any newly created pods. For
   running pods, Thoras uses your
   [`update_policy`](/reference/ast-definition#update-policy): it attempts an
   in-place resize first (if enabled), and falls back to a rolling restart (if
   enabled) when resize fails or is not supported. Any pods created or
   rescheduled in the meantime (e.g., evictions, scale-out) automatically
   receive the adjusted memory regardless of whether the resize or restart
   succeeds.
3. **Repeats every 2 minutes** as long as OOM kills continue, compounding each
   time (1.2×, then 1.44×, then 1.73×, and so on) until OOMs stop.
4. **Holds the memory floor** for a configurable stabilization window after the
   last OOM. During this window, forecasts can raise memory above the floor but
   cannot lower it. This prevents a new forecast, which may not yet reflect the
   OOM episode, from reversing the adjustment down to a point which would be
   prone to OOMing.
5. **Returns full control to the forecaster** once the stabilization window
   expires. If OOMs recur, the cycle begins again.

The adjustment is a short-term stabilization measure, not a permanent
right-sizing decision. The goal is to keep the workload running long enough for
the forecaster to observe the higher memory usage and incorporate it into future
recommendations.

## Enabling OOM Remediation

Add `oom_remediation` to your `spec.vertical` configuration:

```yaml theme={null}
spec:
  vertical:
    mode: autonomous
    oom_remediation:
      enabled: true
    containers:
      - name: my-container
        memory:
          lowerbound: 256Mi
          upperbound: 4Gi
```

`oom_remediation.enabled` must be `true` and `spec.vertical.mode` must be
`autonomous`. Workloads in `recommendation` mode are not affected.

## Stabilization Window

After Thoras applies an OOM memory adjustment, it holds a **memory floor** for
the duration of the stabilization window. During this window:

* Incoming forecasts **cannot lower** memory below the adjusted value.
* Forecasts **can raise** memory above the adjusted value if the forecast
  recommends it.
* Each new OOM resets the stabilization window, extending it from the time of
  the most recent OOM. The workload must be stable for the configured interval
  before memory may be scaled down from the multiplied value.

Once the window expires, Thoras clears the floor and the forecaster resumes full
control of memory sizing.

**Default duration:** `max(forecast_interval, 1h)`. The window is at minimum one
full forecast cycle, ensuring the forecaster has had at least one opportunity to
observe the workload at the adjusted memory level before the floor is removed.
The stabilization window should be long enough for the forecaster to observe
memory usage at the adjusted level, so future recommendations account for the
higher usage that triggered the OOMs.

To override the default, set `stabilization_window` explicitly:

```yaml theme={null}
oom_remediation:
  enabled: true
  stabilization_window: 2h
```

## Upper Bounds

`memory.upperbound` is a **steady-state** policy and is intentionally bypassed
during OOM remediation. OOM is an emergency response: clamping the adjustment to
`upperbound` would leave the workload exposed to the same kill threshold that
triggered the cycle in the first place. Adjustments may therefore land above the
configured upper bound while the stabilization window is active.

Once the stabilization window expires, the forecaster regains full control and
steady-state recommendations are clamped to `upperbound` as usual.

## Memory Limits

OOM remediation raises both the memory request and the memory limit. How the
limit is determined depends on whether Thoras manages it:

* **Limit managed by Thoras** (`memory.limit.ratio` is set): during the
  stabilization window, the limit is
  `max(adjusted_request × ratio, adjusted_limit)`. The ratio is preserved on the
  way up, and the OOM-adjusted limit acts as a floor so a non-OOM reconcile
  cannot snap the limit back below it mid-episode. After the window expires, the
  limit follows the ratio normally as the forecaster regains control.

* **Limit not managed by Thoras** (no `memory.limit.ratio`): the limit is raised
  by the same multiplier as the request during the stabilization window to give
  the workload room to recover. When the window expires, Thoras reverts the
  limit to the value in your workload's pod template (deployment, statefulset,
  etc.). If your workload's true steady-state memory usage permanently exceeds
  the limit set in your pod template, OOMs will recur each time the limit
  reverts, visible as repeated `OOMKilled` events. The resolution is to raise
  the limit in the workload's pod template, or to configure `memory.limit.ratio`
  so Thoras can manage limits directly.

## Interaction with Forecasts

During an active stabilization window, Thoras applies `max(forecast, floor)` to
memory when scaling. CPU is always sourced from the forecast unchanged.

When a new forecast arrives after the window has expired, the floor is cleared
and the forecast value is applied directly. If the workload OOMs again, the
remediation cycle restarts from the new forecast baseline.

## Considerations

**Memory may be held above the forecast during the stabilization window.** If an
OOM was caused by a one-off spike and the workload returns to lower memory usage
afterward, requests will remain elevated until the window expires. Thoras favors
avoiding another OOM over reclaiming memory immediately, and allows configuring
the duration of the stabilization window.

**New pods after the window expires start at the forecast value.** Once the
floor clears, pods created from that point forward use the forecaster's
recommendation. If that recommendation is insufficient, another OOM may occur
and trigger a new remediation cycle.

**Partial in-place resize failures.** If some pods cannot be resized in place
(e.g., the node lacks capacity), those pods continue at their previous memory
until the node has room or they are rescheduled. The next adjustment cycle reads
a blended average and targets a slightly higher value.

## Full Example

```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
  name: my-service
  namespace: my-namespace
spec:
  scaleTargetRef:
    kind: Deployment
    name: my-service
    apiVersion: apps/v1
  model:
    mode: balanced
    forecast_interval: 1h
    forecast_blocks: 4h
  vertical:
    mode: autonomous
    oom_remediation:
      enabled: true
      stabilization_window: 2h
    update_policy:
      update_mode: in_place_or_recreate
    containers:
      - name: my-service
        memory:
          lowerbound: 256Mi
          upperbound: 4Gi
        cpu:
          lowerbound: 100m
          upperbound: 2
```

In this configuration:

* OOM remediation is enabled with a 2-hour stabilization window
* Steady-state memory recommendations are clamped to 4 GiB (`upperbound`); OOM
  adjustments may exceed this temporarily during stabilization
* Pods are resized in place where possible, with recreation as fallback