# Deprecations
Source: https://docs.thoras.ai/deprecations
Active deprecations in Thoras and their removal timelines.
This page tracks deprecations in Thoras, including the announcement date,
removal date, and migration guidance for each. See the
[Support and Deprecation Policy](/policies/support-policy) for the process and
timelines behind these entries.
## Active deprecations
### `featureFlags.enableSimpleAuthSecret` (Helm chart)
| Announced | Removal |
| - | - |
| Chart 5.0.0 | Chart 6.0.0 |
The toggle for component-to-component bearer authentication moved from
`featureFlags.enableSimpleAuthSecret` to `apiClientSecret.enabled`. The legacy
field continues to work as an alias in chart 5.x. Setting both to conflicting
values fails the chart render.
**Migration:** rename the key in `values.yaml`.
```yaml theme={null}
# Before
featureFlags:
enableSimpleAuthSecret: true
# After
apiClientSecret:
enabled: true
```
### `featureFlags.enableLegacySecretSeeding` and legacy Secrets (Helm chart)
| Announced | Removal |
| - | - |
| Chart 5.0.0 | Chart 6.0.0 |
Chart 5.0 moved credential seeding out of Helm templates and into the in-cluster
config-controller. The chart 4.x `api-client-secret` and
`thoras-timescale-password` Secrets are still rendered when
`featureFlags.enableLegacySecretSeeding` is `true` (the default) so the
config-controller can migrate their values on upgrade. Nothing consumes them
afterwards. The flag and both Secrets are removed in chart 6.0.
**Migration:** set `featureFlags.enableLegacySecretSeeding: false` on fresh
installs, and on existing installs once the first 5.x upgrade has completed and
all pods are healthy. See
[Legacy secret seeding](/installation/credentials#legacy-secret-seeding).
# Compliance & Security
Source: https://docs.thoras.ai/faq/compliance
Learn about Thoras security certifications and data protection practices
## SOC 2 Type 2 Certified
![SOC 2 Certified]()
Thoras is SOC 2 Type 2 certified, demonstrating our commitment to maintaining
the highest standards of security, availability, and confidentiality. Our SOC 2
certification provides assurance that we have established appropriate controls
and safeguards to protect your data and systems.
## Self-Hosted. Your Data Stays With You
Thoras is a self-hosted solution, meaning all your data remains within your own
cluster, never leaving your environment. This approach provides several key
security benefits:
### Complete Data Control
* All metrics, recommendations, and operational data stays within your
Kubernetes clusters
* No data transmission to external third-party services
* Full ownership and control over your sensitive information
### Enhanced Security Posture
* Nothing leaves your environment (unless you enable optional Cloud Sync)
* Eliminates external data exposure risks
* Maintains compliance with your organization's data governance policies
* Reduces attack surface by keeping data within your secured perimeter
### Secure by Default
* Internal API calls between Thoras components are authenticated with a shared
bearer token.
* The dashboard requires authentication out of the box, with single sign-on
available through Okta, Azure Entra ID, or any OIDC-compliant identity
provider.
* Optional least-privilege network policies for every component
### Regulatory Compliance
* Helps meet strict data residency requirements
* Supports GDPR, HIPAA, FedRAMP, and other regulatory compliance frameworks
* Enables [audit trails](/guides/audit-trail) within your controlled environment
## Questions About Security?
For specific security questions or to request additional compliance
documentation, email [support@thoras.ai](mailto:support@thoras.ai).
# Integrations
Source: https://docs.thoras.ai/faq/integrations
How Thoras works alongside KEDA, Cluster Autoscaler, and Karpenter
Thoras is designed to complement your existing Kubernetes autoscaling
infrastructure, not replace it. This guide explains how Thoras integrates with
KEDA, Cluster Autoscaler, and Karpenter.
For HPA-specific setup and configuration, see the
[Predictive HPA guide](../guides/hpa).
## How Thoras finds your HPA
The AIScaleTarget references a Deployment directly. It does not reference an HPA
or KEDA ScaledObject. Thoras automatically discovers the HPA that targets the
same Deployment by looking up HPAs in the same namespace and matching on
`scaleTargetRef`.
This means you do not need to change your existing HPA or KEDA configuration
when adding Thoras. Thoras finds it, reads its configured metrics to know what
to collect, and incorporates its desired replica count into scaling decisions,
all without modifying the HPA itself.
***
## KEDA
[KEDA](https://keda.sh) (Kubernetes Event-Driven Autoscaling) scales workloads
based on external event sources: queue depth, Kafka lag, HTTP request rate, and
more.
**How they work together**
KEDA ScaledObjects create and manage standard Kubernetes HPA objects behind the
scenes. Thoras treats KEDA-managed HPAs identically to manually created HPAs. It
reads KEDA's configured metrics and incorporates KEDA's desired replica count
into its own scaling decisions.
Two things happen at runtime:
1. **Thoras scales proactively.** When a new forecast is ready, Thoras writes
its suggested replica count to the AIScaleTarget. The Thoras operator reads
both Thoras's suggestion and the KEDA-managed HPA's current desired replicas,
then scales the workload to `max(thoras_suggested, keda_desired)`.
2. **Thoras acts as a floor for KEDA's reactive scaling.** When KEDA's HPA
submits a scale request, a Thoras admission webhook intercepts it and ensures
the final replica count is at least `max(keda_proposed, thoras_suggested)`.
This prevents KEDA from scaling below Thoras's predictive floor during quiet
periods.
**What Thoras never does**
Thoras never modifies KEDA ScaledObjects or the HPA objects KEDA manages. KEDA
retains full control over its HPA configuration. Thoras only influences the
replica count at the moment a scale decision is executed.
**Three complementary layers**
With KEDA and Thoras together, you get three scaling layers working in parallel:
KEDA scales based on current event backlog, HPA reacts to live resource
pressure, and Thoras scales ahead of both based on predicted future demand. Each
layer handles a different class of signal, and none replaces the others.
***
## Cluster Autoscaler
[Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
adds and removes nodes based on whether pods can be scheduled.
**How they work together**
Thoras and Cluster Autoscaler operate at different layers and do not communicate
directly. Thoras works at the pod level; Cluster Autoscaler works at the node
level. The integration is through pod resource requests.
Cluster Autoscaler decides how many nodes to provision based on the resource
requests (`resources.requests`) of pending pods. Thoras keeps those requests
accurately sized to actual workload usage through vertical rightsizing. This
gives Cluster Autoscaler a more accurate signal:
* **Under-specified requests** cause Cluster Autoscaler to underestimate how
many nodes are needed, leading to scheduling pressure and delayed scale-out.
* **Over-specified requests** waste node capacity and slow down Cluster
Autoscaler's scale-in decisions (nodes appear utilized even when workloads are
idle).
Right-sized requests let Cluster Autoscaler provision fewer, better-utilized
nodes, reducing infrastructure cost without sacrificing reliability.
***
## Karpenter
[Karpenter](https://karpenter.sh) is a node provisioning tool that bin-packs
pods onto nodes and selects optimal instance types automatically.
**How they work together**
Like Cluster Autoscaler, Karpenter uses pod resource requests to make
provisioning decisions. The same principle applies: accurately sized requests
from Thoras's vertical rightsizing give Karpenter better bin-packing signals,
enabling tighter consolidation and smaller, more cost-effective node selections.
Karpenter's consolidation feature (moving pods to fewer nodes) also benefits
from right-sized requests. When pods request only what they actually use,
Karpenter can consolidate more aggressively without risking resource pressure.
Thoras does not configure or modify Karpenter NodePools or EC2NodeClasses.
***
## Summary
| Tool | Integration type | Thoras role |
| - | - | - |
| HPA | Direct | Reads HPA metrics; intercepts scale requests; never modifies HPA |
| KEDA | Via HPA | Same as HPA; KEDA-managed HPAs treated identically |
| Cluster Autoscaler | Indirect | Right-sized pod requests improve node provisioning signals |
| Karpenter | Indirect | Right-sized pod requests improve bin-packing and consolidation |
# Frequently Asked Questions
Source: https://docs.thoras.ai/faq/introduction
## Where does Thoras store my data?
We have no access to your data. Because Thoras is self-hosted, all of your data
stays with you and never leaves your environment. All data and metrics remain
local to your cluster and are never transmitted beyond your environment's
perimeter.
## What type of data does Thoras use?
The Thoras platform uses historical numeric Kubernetes metrics to train its
models and generate predictions. It collects container performance metrics and
custom metrics via the Kubernetes API.
## How long does Thoras take to install?
Installation takes less than an hour and is done via a Helm chart.
## How are cost, savings, and waste calculated?
Thoras monitors various signals across your cluster to calculate cost, savings,
and waste over time:
* **Controller spec and pod resource requests**: compute resources such as CPUs,
memory, GPUs, etc. allocated to your workloads
* **Pod usage and utilization**: what's actually being consumed versus what's
requested
* **Node-level metrics**: instance types, node counts, and their associated
pricing
All of this data is stored as time-series metrics within your cluster, so Thoras
can track your spend, waste, and savings over time as you optimize with the
platform.
**Important:** All cost calculations and metric storage happen entirely within
your cluster, no data leaves your environment.
## How does Thoras reside in a customer environment?
The platform is lightweight and runs on a per-cluster basis to manage target
workloads using an `AIScaleTarget` custom resource. The `AIScaleTarget` is
configured with just a few lines of YAML.
## Do I have to pick a model to get accurate forecasts?
No. Thoras handles all model training and tuning to ensure you receive the most
accurate forecasts. It leverages a variety of model architectures, parameters,
and features, and continuously scores itself to identify the optimal model for
your workload patterns. Users can customize inputs such as prediction frequency
and operating mode (e.g., cost-saving, balanced, etc.).
## Where do my forecasting models live?
As with all your data, all models reside within your cluster. Training, tuning,
and prediction happen entirely within your environment. Models never leave your
infrastructure.
## How often do forecasts get made?
You're in control. We generally recommend a forecast cadence of 5 to 30 minutes.
More variable workloads often benefit from a faster cadence.
Thoras automatically manages model training and fine-tuning as needed. It
continuously monitors forecast quality and updates models whenever improvements
are possible, so no user intervention is required.
## What if my forecast is wrong?
Thoras will NEVER leave your services under-provisioned. While even the most
accurate forecasting models can't anticipate events like an early marketing
email or a DDoS attack, Thoras is built to handle these situations reliably. If
your service suddenly experiences unexpected demand, Thoras overrides the
forecast and scales in real time to meet current needs.
## How much data does Thoras need to produce accurate forecasts?
Most services will receive highly accurate forecasts within the first 48 hours.
In rare cases, more time may be needed. For example, if your service has a
unique usage pattern that only occurs on Saturdays, Thoras would need to observe
at least one Saturday to 'warm up' and learn that specific trend.
## What types of predictive scaling does Thoras provide? For what use cases is each type of scaling method recommended?
Thoras provides both predictive horizontal pod scaling (adjusting replica count)
and predictive vertical pod rightsizing (adjusting CPU and memory requests).
* **Predictive horizontal pod autoscaling** is recommended for workloads with
variable traffic patterns that need to scale the number of replicas to handle
demand spikes.
* **Predictive vertical pod rightsizing** is ideal for workloads with more
predictable resource patterns that benefit from optimized CPU and memory
requests to improve cluster utilization and reduce costs.
## Can I use both vertical and horizontal modes together?
**Recommendation mode (both directions):** You can have both vertical and
horizontal in `recommendation` mode simultaneously. Thoras will provide
suggestions for both, but these suggestions are mutually exclusive. For example,
if Thoras suggests 1Gi memory (vertical) and 3 pods (horizontal), the workload
can rightsize by either applying the vertical suggestion or the horizontal
suggestion, but not both.
**Autonomous mode (one direction only):** Only one scaling direction can be in
`autonomous` mode at a time. When enabling autonomous mode, you choose whether
Thoras should automatically apply vertical or horizontal scaling. Contact
Thoras' support if you wish to discuss which direction is the best option for
your workloads.
See
[Understanding Vertical and Horizontal Scaling Modes](/guides/configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
for additional details.
## What types of metrics does Thoras support for vertical and horizontal scaling?
Thoras supports CPU, memory, and any custom metrics relevant to your workload.
## Does Thoras support integration with existing autoscalers?
Yes. Thoras works alongside HPA, KEDA, Cluster Autoscaler, and Karpenter without
replacing or modifying any of them. See
[How Thoras integrates with existing autoscalers](./integrations) for a detailed
breakdown of each integration.
## Does Thoras support integration with CI/CD?
Yes! Thoras is compatible with virtually all CI/CD tools and workflows,
including Argo CD, Flux, and Blue/Green deployments. See the
[ArgoCD guide](/guides/argo-cd) for the recommended `Application` configuration.
We can also add support for any unique or unsupported workflows upon request.
## Is the Thoras dashboard protected?
Yes. The dashboard requires a login by default. On first install Thoras
generates a password inside your cluster. For production, you can switch to
single sign-on against Okta, Azure Entra ID, or any OIDC-compliant identity
provider, or disable the built-in layer if you already enforce authentication at
your ingress or service mesh. See
[Dashboard authentication](/guides/dashboard-auth).
## How does Thoras handle credentials?
Any required credentials you don't supply are generated in-cluster on first
bootstrap and stored in a Kubernetes Secret. Alternatively, you can manage the
secrets Thoras needs with your preferred tooling and hand them to the platform
through one of the paths the Helm chart supports. See
[Credentials](/installation/credentials).
# Quickstart
Source: https://docs.thoras.ai/getting-started/quickstart
Deploy Thoras and see Kubernetes resource recommendations in minutes!
## Overview
This guide will walk you through how to quickly install Thoras and get to see
resource recommendations for your workloads in just a few minutes.
## What you'll need
1. Your Thoras license key (email [support@thoras.ai](mailto:support@thoras.ai)
if you don't have one)
2. [Helm](https://helm.sh/)
3. Privileged access to a Kubernetes cluster
4. The cluster should have
[Kubernetes metrics server](https://github.com/kubernetes-sigs/metrics-server)
installed. It probably already has this!
For optimal forecast accuracy and full Autonomous Mode support, Thoras needs a
Kubernetes Persistent Volume (PV) to store historical metrics.
This QuickStart intentionally skips that setup so you can get the product
running quickly and start exploring waste insights right away. If you prefer to
include the PV during initial installation, review the
[Persistent Storage](../installation/persistent-storage) before proceeding with
the [Install Thoras](#install-thoras) steps below. You can also always add a PV
as a follow-on step.
## Install
Let's get Thoras up-and-running!
### Use the Thoras Helm repo
Install the Thoras Helm repo, which will give you access to all of the Thoras
Helm charts.
```
helm repo add thoras https://thoras-ai.github.io/helm-charts
helm repo update thoras
```
### Install Thoras
1. Note the name of your license key file. We'll use `thoras_license.txt` in our
example below.
2. Install Thoras via Helm:
```
helm install thoras thoras/thoras \
--namespace thoras \
--create-namespace \
--set imageCredentials.password="$(cat ./thoras_license.txt)" \
--set featureFlags.enableLegacySecretSeeding=false
```
`featureFlags.enableLegacySecretSeeding` exists only so clusters upgrading
from chart 4.x can migrate their existing credentials. A fresh install has
nothing to migrate, so turning it off keeps the release clean. See
[Upgrading](../installation/upgrading) if you're coming from 4.x.
The chart ships with defaults that should work for most initial deployments. See
[Advanced Setup](../installation/advanced-setup) for the common configuration
knobs, or the
[full values reference](https://github.com/thoras-ai/helm-charts/tree/main/charts/thoras#values)
on GitHub.
Need to set up license secret externally? See
[Advanced Setup](../installation/advanced-setup#reference-an-existing-secret).
### Verify installation
It should only take 1-2 minutes for all Thoras pods to come online. Verify that
all pods are healthy by confirming their `Running` status:
```
kubectl get pods -n thoras
```
On a fresh install, pods briefly report `CreateContainerConfigError` while the
Thoras config-controller generates the credentials they need. This resolves on
its own within a minute. No action is needed.
Once all pods are in `Running` status, congratulations! You've successfully
installed Thoras!
## Get Workload Resource Recommendations
Now that Thoras is up and running, we're just minutes away from getting resource
recommendations that will help our workloads run more efficiently, boosting
reliability and saving on cloud costs.
### View the dashboard
We'll be working out of the Thoras Dashboard from here on out. To access it:
1. Port-forward to the dashboard pod:
```
kubectl port-forward service/thoras-dashboard -n thoras 8080:80
```
2. Pull up the dashboard in your browser: [http://localhost:8080](http://localhost:8080)
3. The dashboard prompts for a username and password. The default username is
`thoras`. Fetch the generated password with:
```
kubectl get secret thoras-config-controller -n thoras \
-o jsonpath='{.data.dashboard-auth-password}' | base64 -d
```
See [Dashboard authentication](/guides/dashboard-auth) for how to change the
credentials, sign in with your identity provider (Okta, Azure Entra ID), or
bring your own auth. Before exposing the dashboard beyond `localhost`,
terminate TLS at your ingress and set
`thorasDashboard.auth.cookieSecure: true`.
Thoras will auto-discover workloads in your cluster and give you the option to
onboard all the workloads in a namespace, with zero configuration needed. This
is a great way to experiment and see the value of Thoras quickly!
4. Select a namespace from the dropdown and click "Enroll Namespace" to enroll
all associated workloads, or alternatively select individual workloads from
the table below and click "Enroll Selected" to specify the workloads you'd
like to target. This will enroll workloads into `Recommendation Mode` so they
may begin receiving resource recommendations. This is a read-only operation.
No changes will take place to your workloads.
![auto-create]()
5. You'll see a new popup where Thoras tells you it's populating
`AIScaleTargets` for you.
6. Once complete, you'll see a list of onboarded targets under the
Recommendation tab on the Home page. Click "View" to see more!
![live-targets]()
7. You should immediately see metrics beginning to collect! Use the slider below
the graph to narrow the time window.
![metrics-collecting]()
8. Predictions will run on a 15m interval. Grab a cup of coffee! ☕
### Replica and resource recommendations
Predictions will become more accurate over time as the Thoras Reasoning Engine
learns more about your environment. But we can still get useful predictions even
on the first forecast! Let's dive in.
#### Workload efficiency at a glance
The `Pod Health` widget will give you a quick overview of replica count
recommendations and workload's overall resource utilization.
![pod-health]()
#### Pod resource recommendations
Check out how much more efficiently, reliably, and cost-effectively your
workloads could be running in the `Resource Allocation Settings` widget.
![resource-allocation]()
## Getting support
For any questions, email [support@thoras.ai](mailto:support@thoras.ai).
# What is Thoras?
Source: https://docs.thoras.ai/getting-started/what-is-thoras
The autonomous control plane for compute efficiency.
Thoras is a self-hosted, AI/ML-powered platform that replaces reactive
Kubernetes autoscaling with predictive, autonomous resource management. It
eliminates waste, prevents capacity outages, and safely optimizes critical
infrastructure. Thoras is designed to earn your trust before running
autonomously.
Reduction in cloud compute waste from CPU and memory over-provisioning.
OOM kill loops resolved automatically. Latency spikes prevented through
rightsized compute.
Continuous right-sizing without manual tuning or on-call escalations.
## Philosophy
**Autonomy is earned, not assumed.** Thoras doesn't arrive in your cluster and
start making decisions. It observes, learns, surfaces recommendations, and only
acts autonomously once you've validated its reasoning. Your workload and
infrastructure context are always respected.
**Performance first, efficiency second.** Thoras reclaims waste from
under-utilized CPU and memory without compromising availability. It knows when
not to scale down as clearly as it knows when to scale up.
## How it works
Thoras ingests real-time telemetry from your existing metric sources
(Prometheus, the Kubernetes metrics server, and any custom or external
metrics you already use) and persists historical usage data inside your
cluster. It builds per-workload demand profiles that account for
seasonality, traffic patterns, and resource trends.
AI/ML models continuously predict horizontal and vertical scaling needs
ahead of time. Forecasts update on configurable intervals (typically minutes
to hours), with model accuracy improving as patterns repeat.
Before any autonomous action, Thoras surfaces predictions and right-sizing
recommendations in a dashboard. Teams validate model accuracy first.
Autonomous mode requires 24 hours of historical data by default, and
benefits from at least 48 hours for workloads with daily seasonality.
Once enabled, Thoras scales pods and replica counts in advance of demand,
remediates OOM kill loops in real time with compounding memory adjustments,
and maintains configurable upper and lower bounds at all times. You can
pause autonomous scaling cluster-wide at any moment.
## Architecture
Thoras installs as a single Helm chart and runs entirely inside your cluster.
Nothing leaves your environment unless you opt in to cloud sync. The platform is
lightweight (a small install runs on about 2 vCPU) and consists of the following
components:
| Component | Role |
| - | - |
| **Operator** | Reconciles `AIScaleTarget` resources and applies scaling decisions. Hosts the admission webhooks that keep other autoscalers from undoing Thoras's work. |
| **Metrics collector** | Gathers pod and node usage into an embedded TimescaleDB (or a database you manage) so models have history to learn from. |
| **Forecast workers** | Train and run the ML models. The pool scales itself as you enroll more workloads. |
| **Worker** | Background jobs: cost refresh, monitors, OOM remediation, and reconciliation. |
| **API server** | Serves forecasts, recommendations, and cost data to the dashboard and to other components, using shared bearer-token authentication. |
| **Dashboard** | The web UI, fronted by an authentication sidecar (generated credentials by default, or single sign-on via your OIDC identity provider). |
| **Config-controller** | Seeds the credentials you don't supply, keeps them in sync, and performs dependency-ordered restarts when they change. |
See [Advanced Setup](/installation/advanced-setup) for how to configure each
component.
## Capabilities
Pre-emptive CPU and memory request optimization. Day-one support for K8s
1.33+ in-place pod resize, with rolling restart fallback.
Replica counts adjusted ahead of demand spikes, not after. Integrates with
your existing HPA.
Detects kill loops and stabilizes workloads with compounding memory
adjustments (1.2× per cycle) until the forecaster catches up.
Purpose-built handling for Java workloads, with heap and GC pressure
awareness.
Real-time waste quantification against node pricing data. Savings estimated
per workload.
`ClusterAIScaleTemplate` applies scaling policies across namespaces without
per-workload config.
## Get started
Install Thoras and see resource recommendations in minutes.
How Thoras works alongside KEDA, Cluster Autoscaler, and Karpenter.
# ArgoCD
Source: https://docs.thoras.ai/guides/argo-cd
ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes applications. This guide covers deploying Thoras with ArgoCD and managing Thoras-targeted workloads alongside it.
## Using ArgoCD to deploy Thoras
The Thoras Helm chart is designed to render deterministically for GitOps tools.
Credentials are never generated at template time: anything you don't supply is
seeded in-cluster by the Thoras config-controller, and anything you pin in
`values.yaml` is rendered verbatim. Two things still need attention:
1. **Disable legacy secret seeding.** Set
`featureFlags.enableLegacySecretSeeding: false`. The flag defaults to `true`
only so clusters upgrading from chart 4.x can migrate their pre-existing
credentials. The legacy Secrets it renders use Helm `lookup`, which Argo CD
cannot evaluate. If you are upgrading from 4.x, complete the upgrade with the
flag on, then set it to `false` once the pods are healthy. During that
upgrade window, keep `jsonPointers: [/data]` `ignoreDifferences` entries for
the `api-client-secret` and `thoras-timescale-password` Secrets so Argo CD
does not overwrite them with freshly-rendered random values before the
config-controller has migrated them. Remove those entries once the flag is
`false`. See [Upgrading](/installation/upgrading).
2. **Ignore fields that are managed in-cluster.** The webhook `caBundle` is
injected after apply (by the chart's cert-generator or by cert-manager), and
the `thoras-forecast-worker` replica count is managed by Thoras's built-in
scaler. Add both to `spec.ignoreDifferences`.
### Example Application config
```yaml theme={null}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: thoras
namespace: argocd
spec:
project: default
source:
chart: thoras
helm:
parameters:
- name: imageCredentials.password
value:
- name: featureFlags.enableLegacySecretSeeding
value: "false"
repoURL: https://thoras-ai.github.io/helm-charts
targetRevision:
destination:
namespace: thoras
server: https://kubernetes.default.svc
syncPolicy:
automated: {}
ignoreDifferences:
# The webhook caBundle is injected in-cluster after apply.
- group: admissionregistration.k8s.io
kind: MutatingWebhookConfiguration
jqPathExpressions:
- ".webhooks[]?.clientConfig.caBundle"
- group: admissionregistration.k8s.io
kind: ValidatingWebhookConfiguration
jqPathExpressions:
- ".webhooks[]?.clientConfig.caBundle"
# Forecast worker replicas are managed by Thoras at runtime.
- jsonPointers:
- /spec/replicas
kind: Deployment
name: thoras-forecast-worker
```
The chart's default webhook certificate flow uses Helm hook Jobs
(`pre-install`, `pre-upgrade`), which GitOps tools do not handle well. If you
run [cert-manager](https://cert-manager.io), set
`thorasOperator.webhookCertGen.certManager.enabled: true` to replace those
Jobs with declarative `Issuer` and `Certificate` resources. See [Webhook TLS
with
cert-manager](/installation/advanced-setup#webhook-tls-with-cert-manager).
## Credential rotation under GitOps
When a credential changes (you rotate a pinned value in `values.yaml`, update a
Secret you manage, or delete a generated key so it is re-seeded), the
config-controller restarts the affected Thoras workloads for you. It does this
by evicting pods rather than patching the Deployment's pod template, so the
restart never shows up as drift in Argo CD. No `kubectl rollout restart` is
needed. See [Credentials](/installation/credentials).
## Flux
The same guidance applies to Flux: set
`featureFlags.enableLegacySecretSeeding: false` and prefer cert-manager for
webhook TLS. Additionally set `flux.enabled: true` so Flux's drift detection
does not revert the operator's own rollout restarts:
```yaml theme={null}
flux:
enabled: true
```
## Using ArgoCD to manage a Thoras horizontally controlled workload
When Thoras manages horizontal scaling for a workload, it uses a validation
webhook to prevent the default Kubernetes Horizontal Pod Autoscaler (HPA) from
reducing replicas during anticipated usage spikes.
By design, this ensures workloads remain scaled up until Thoras determines it's
safe to scale down. However, ArgoCD interprets these blocked scale-down attempts
as HPA errors. This can cause the Application to appear in a degraded state even
though the workload is healthy.
To prevent this, you can customize ArgoCD's health assessment logic for HPAs so
that these Thoras-specific conditions are handled correctly.
### When Do I Need This?
You only need this customization if all of the following are true:
* The workload is horizontally controlled by Thoras.
* You are using ArgoCD to monitor the workload state.
* You notice ArgoCD incorrectly marking the HPA as **Degraded** when Thoras
blocks a scale-down.
If your workloads are **not** using Thoras for horizontal scaling, this
customization is unnecessary.
**1. Edit the ArgoCD ConfigMap**:
Open the `argocd-cm` ConfigMap in the `argocd` namespace:
```bash theme={null}
kubectl edit configmap argocd-cm -n argocd
```
**2. Add the following customizations under** `data`:
```
resource.customizations.useOpenLibs.autoscaling_HorizontalPodAutoscaler: "true"
resource.customizations.health.autoscaling_HorizontalPodAutoscaler: |
hs = {}
if obj.status ~= nil then
if obj.status.conditions ~= nil then
for i, condition in ipairs(obj.status.conditions) do
-- Mark Degraded only for certain conditions
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
-- Ignore AbleToScale=False with FailedUpdateScale reason caused by Thoras
if condition.type == "AbleToScale" and condition.status == "False" and condition.reason == "FailedUpdateScale" and string.find(condition.message or "", "target is already at Thoras desired replicas") ~= nil then
hs.status = "Healthy"
hs.message = "Target is at Thoras desired replicas"
return hs
end
-- Mark Healthy for Ready=True or ScalingActive=True
if (condition.type == "Ready" or condition.type == "ScalingActive") and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for HPA to become healthy"
return hs
```
**3. Restart ArgoCD components** (to apply changes):
```bash theme={null}
kubectl rollout restart deployment argocd-repo-server -n argocd
kubectl rollout restart deployment argocd-applicationset-controller -n argocd
```
# Audit Trail
Source: https://docs.thoras.ai/guides/audit-trail
See who changed scaling behavior, and what changed, from the api-server logs
The api-server logs an audit event for every privileged action it takes:
changing a target's scale mode, pausing or resuming scaling cluster-wide,
cleaning up a removed target, and enrolling new targets. Each event records who
took the action, what changed, and whether it succeeded.
## What's recorded
| Action | Trigger |
| - | - |
| `ast.mode.update` | Changing a target's horizontal or vertical scale mode |
| `system.scaling.update` | Pausing or resuming [autonomous scaling](/guides/pause-scaling) cluster-wide |
| `ast.cleanup` | Cleaning up a removed target's metrics |
| `ast.enroll_candidates` | Enrolling workloads in a namespace (all of them, if none are named) |
| `ast.enroll_all` | Enrolling all namespaces |
Both successful and failed attempts are recorded. A dry-run or count-only
enrollment preview isn't, since nothing is actually changed.
Not every dashboard-reachable change goes through this list yet — cost-savings
settings is the one left out of this first pass.
## Who's recorded
The dashboard forwards the identity of the person who took the action, but only
when [dashboard authentication](/guides/dashboard-auth) can establish one:
| Auth mode | Actor recorded |
| - | - |
| `oidc` | The user's email |
| `htpasswd` | The shared username |
| `auth.enabled: false`, or any direct API call | `unknown` |
If per-person attribution matters to you, use `oidc` mode. `htpasswd` and
disabled auth both share one identity across everyone with dashboard access.
The actor is an attribution hint, not authentication. Anyone holding your
`apiClientSecret` can call the API directly and set it themselves. It's useful
for distinguishing a change made through the dashboard from one made by a
script or another integration, not for proving who made it.
## Reading the log
```bash theme={null}
kubectl logs -n thoras deploy/thoras-api-server-v2 | grep "audit event"
```
Each line carries these fields:
| Field | Meaning |
| - | - |
| `actor`, `actor_source` | Who took the action, and how that was determined |
| `action` | One of the values above |
| `target_kind`, `target_namespace`, `target_name` | What was acted on |
| `old_value` | The state before the action, for actions that have one; not every action does |
| `new_value` | The result, set only on success. Absent on a failed attempt — `details.requested` has what was asked for instead |
| `outcome` | `success` or `failure` |
| `error` | Present only on failure |
| `persisted` | Whether this event also made it into the database (see below); `false` means only this log line exists |
| `details` | Extra context specific to the action — a count, a request ID, or what was asked for versus what changed |
Pod logs are subject to your cluster's log retention, so ship api-server logs to
your log aggregator if you need to keep this history longer than that.
## Is there a durable copy?
Yes — every event is also written to a database table before this log line is
emitted, so a log rotation doesn't lose the history. That table isn't a
supported, documented interface yet: there's no API or dashboard view onto it,
its schema isn't stable across releases, and support for querying it directly
isn't something we can commit to today. Treat the log line above as the
supported way to read this trail; `persisted: false` on a line means even the
database copy is missing for that one event, so the log was all there was.
# AWS Cost Explorer
Source: https://docs.thoras.ai/guides/aws-cost-explorer
Collect your cluster's real billed AWS spend so Thoras reports actual cost instead of estimates.
Thoras estimates cluster cost from a built-in static price list. Those estimates
are directional by design — they don't know about your reserved instances,
savings plans, or negotiated rates. See [Cost Estimates](/guides/cost-estimates)
for how that calculation works.
Enabling AWS Cost Explorer collection gives Thoras the spend AWS actually billed
you, broken down by service and usage type for each day. Cost reporting uses
this in place of the estimates, so the figures you see match your invoice.
Collection is off by default. AWS bills every Cost Explorer API request, so
turning it on is an explicit opt-in.
Cost Explorer collection reads AWS billing, so it applies to EKS clusters.
Thoras skips collection on clusters it detects are running in another cloud
rather than paying for a query that cannot return anything.
## Before you start
You need:
* An EKS cluster with an
[IAM OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)
associated, so the Thoras worker can assume a role through IRSA.
* [Cost Explorer enabled](https://docs.aws.amazon.com/cost-management/latest/userguide/ce-enable.html)
on the account that pays for the cluster.
* The `aws:eks:cluster-name`
[cost allocation tag activated](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/activating-tags.html)
in the Billing console of that account.
Activating the cost allocation tag is the step most often missed. Thoras
filters your account's spend down to this cluster using that tag, and Cost
Explorer only returns tagged spend for tags that have been activated. If the
tag is inactive, every query succeeds and returns nothing. Activation can take
up to 24 hours to take effect, and tag data may not reach all the way back
through the initial 90-day backfill window.
## Grant access to Cost Explorer
The Thoras worker reaches Cost Explorer with whatever AWS credentials its pod
picks up. On EKS, bind an IAM role to the worker's service account with IRSA.
The role needs a single permission, `ce:GetCostAndUsage`. Cost Explorer does not
support resource-level permissions, so the policy is scoped to `*`. Thoras also
resolves your AWS account ID through `sts:GetCallerIdentity`, which requires no
permission of its own.
### Terraform
```hcl theme={null}
data "aws_eks_cluster" "this" {
name = var.cluster_name
}
# Your cluster's existing IAM OIDC provider. Create one with
# aws_iam_openid_connect_provider first if the cluster does not have it.
data "aws_iam_openid_connect_provider" "eks" {
url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}
locals {
# Trust policy conditions are keyed on the issuer host, not the full URL.
oidc_host = replace(
data.aws_eks_cluster.this.identity[0].oidc[0].issuer,
"https://",
"",
)
}
data "aws_iam_policy_document" "thoras_cost_explorer_trust" {
statement {
effect = "Allow"
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [data.aws_iam_openid_connect_provider.eks.arn]
}
# Only the Thoras worker's service account may assume this role. Match
# thorasWorker.serviceAccount.name if you changed it from the default.
condition {
test = "StringEquals"
variable = "${local.oidc_host}:sub"
values = ["system:serviceaccount:${var.thoras_namespace}:thoras-worker"]
}
condition {
test = "StringEquals"
variable = "${local.oidc_host}:aud"
values = ["sts.amazonaws.com"]
}
}
}
data "aws_iam_policy_document" "thoras_cost_explorer" {
statement {
effect = "Allow"
actions = ["ce:GetCostAndUsage"]
# Cost Explorer has no resource-level permissions.
resources = ["*"]
}
}
resource "aws_iam_role" "thoras_cost_explorer" {
name = "thoras-cost-explorer"
assume_role_policy = data.aws_iam_policy_document.thoras_cost_explorer_trust.json
}
resource "aws_iam_role_policy" "thoras_cost_explorer" {
name = "thoras-cost-explorer"
role = aws_iam_role.thoras_cost_explorer.id
policy = data.aws_iam_policy_document.thoras_cost_explorer.json
}
```
Activate the cost allocation tag in the same run. This resource belongs in the
account that pays the bill, which may be a different provider alias than the one
holding the cluster:
```hcl theme={null}
resource "aws_ce_cost_allocation_tag" "eks_cluster_name" {
tag_key = "aws:eks:cluster-name"
status = "Active"
}
```
### Without Terraform
Create a role with this trust policy, substituting your account ID, OIDC issuer
host, and the namespace Thoras is installed in:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:thoras:thoras-worker",
"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
}
}
}
]
}
```
Attach this permission policy to it:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ce:GetCostAndUsage",
"Resource": "*"
}
]
}
```
## Enable collection
Point the worker's service account at the role and turn the collector on:
```yaml theme={null}
thorasWorker:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/thoras-cost-explorer
enableCostAndUsageCollector: true
```
Then `helm upgrade` your release. The worker restarts and begins collecting
immediately.
## How Thoras identifies your cluster's spend
Your AWS account bills for more than one cluster, so Thoras narrows each query
to a cost allocation tag. It reads that tag from your node labels:
| Node label | Cost allocation tag |
| - | - |
| `eks.amazonaws.com/cluster-name` | `aws:eks:cluster-name` |
| `alpha.eksctl.io/cluster-name` | `aws:eks:cluster-name` |
If none of your nodes carry those labels — self-managed nodes often don't —
collection stops and logs the labels it looked for. Set the tag yourself in that
case:
```yaml theme={null}
thorasWorker:
costExplorerFilter: "my-cost-tag=prod-cluster"
```
The value is a `TAG_KEY=TAG_VALUE` pair and replaces the node label lookup
entirely. Use it when your spend is tracked under your own tagging scheme rather
than the AWS-generated cluster tag. Whatever key you choose must be activated as
a cost allocation tag, the same as `aws:eks:cluster-name`.
## What Thoras collects
Each run stores one row per day, per AWS service, per usage type:
* **Billed cost** — the cost as it appears on your invoice.
* **Effective cost** — amortized and commitment-adjusted, so reserved instance
and savings plan fees are spread across the term they cover rather than
landing on the day they were charged.
* **Usage quantity and unit** — how much of that usage type was consumed.
The collector runs every 24 hours. AWS finalizes a day's costs well after the
day ends, so collecting more often would not make the data fresher.
* The first run backfills **90 days**.
* Every run after that re-fetches the **last 7 days**, overwriting what it
already stored. AWS restates recent days as usage finalizes, and this picks up
those corrections.
* The job also runs when the worker starts, but skips if the stored data is less
than 20 hours old, so a rollout or a crash loop doesn't pay for the same days
again.
## What this costs
AWS charges per Cost Explorer API request — \$0.01 per paginated request as of
September 15, 2026. See
[AWS Cost Management pricing](https://aws.amazon.com/aws-cost-management/pricing/)
for current rates.
A nightly run costs a handful of requests. The 90-day backfill on first run is
larger but one-time. Cost scales with how many distinct service and usage type
combinations your account bills for, since that determines how many pages each
query returns.
## Verify it's working
Check the worker logs:
```bash theme={null}
kubectl logs -n thoras deploy/thoras-worker | grep -i "cost and usage"
```
On startup you should see `Cost and usage collector is enabled`, then
`Collecting daily cost and usage` with the date window and the tag Thoras
resolved:
```
Collecting daily cost and usage start=2026-06-17 end=2026-09-14 tagKey=aws:eks:cluster-name tagValue=prod-cluster
```
Confirm `tagValue` matches your cluster. If it doesn't, set `costExplorerFilter`
explicitly.
## Troubleshooting
**`Cost and usage collection returned no rows`** — the query succeeded but
matched no billed resources. Almost always the cost allocation tag isn't
activated, or the `tagKey` and `tagValue` in the log don't match how your
resources are actually tagged. Check the Billing console first, then verify the
tag against a node in the EC2 console.
**`no cost filter could be resolved`** — none of your nodes carry a recognized
cluster name label. The log lists the labels Thoras searched for. Set
`thorasWorker.costExplorerFilter` to your own `TAG_KEY=TAG_VALUE` pair.
**`AccessDeniedException` when calling Cost Explorer** — the worker assumed a
role without `ce:GetCostAndUsage`, or didn't assume one at all. Confirm the
service account carries the annotation:
```bash theme={null}
kubectl get serviceaccount thoras-worker -n thoras -o yaml
```
If the annotation is there but the role isn't being assumed, the trust policy's
`sub` condition likely doesn't match — it must name the namespace Thoras is
installed in and the worker's service account name.
**`Could not build the AWS Cost Explorer provider`** — the worker couldn't read
its AWS configuration. It keeps running and collects nothing rather than failing
startup. Check for a malformed AWS config file or environment.
**`The cost and usage provider does not bill the cloud this cluster runs in`** —
Thoras detected the cluster is running somewhere other than AWS and skipped the
query. Setting `costExplorerFilter` bypasses this check if you're intentionally
attributing non-EKS spend.
# Configuring AIScaleTargets
Source: https://docs.thoras.ai/guides/configuring-asts
To enable Thoras to manage or recommend scaling for your workload, you must
register an `AIScaleTarget` Custom Resource (CRD) in your Kubernetes cluster.
This guide will walk you through doing this by applying an `AIScaleTarget`
resource with `kubectl`.
#### Prerequisites
* A running Kubernetes cluster with
[Kubernetes metrics server](https://github.com/kubernetes-sigs/metrics-server)
installed
* Thoras installed and running (see
[quickstart guide](../getting-started/quickstart))
#### 0. (Optional) Auto-configure `AIScaleTarget`s using the dashboard
On a fresh installation, Thoras can auto-discover workloads and make suggestions
to auto-populate `AIScaleTarget`s in the cluster. This enables metric collection
and resource recommendations for the selected workloads (see
[quickstart guide](../getting-started/quickstart#view-the-dashboard)).
Eventually you will probably want to integrate your `AIScaleTarget`
configurations into your delivery pipelines, but the auto-populate feature is a
particularly useful way to try out the product without having to manage any
configuration.
#### 1. Define Initial `AIScaleTarget` Custom Resource
Create a new file named `my-ast.yaml`
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: {{YOUR_AST_NAME}}
namespace: {{YOUR_NAMESPACE}}
spec:
model:
forecast_blocks: 4h
forecast_interval: 1h
mode: balanced
scaleTargetRef:
kind: Deployment
name: {{YOUR_AST_NAME}}
horizontal:
mode: recommendation
```
* `metadata.name`: The name for your `AIScaleTarget` resource (should match your
workload name for clarity).
* `metadata.namespace`: The namespace where your workload is running.
* `spec.scaleTargetRef`: Points to the Kubernetes resource (usually a
Deployment) you want to scale.
* `spec.model.forecast_blocks`: How far into the future to forecast. Thoras
sizes for the predicted peak within this window.
* `spec.model.forecast_interval`: How often to make a forecast.
#### 2. Apply the `AIScaleTarget` Using the CLI
Use `kubectl apply` to register the `AIScaleTarget` with your cluster:
```sh theme={null}
kubectl apply -f my-ast.yaml
```
You should see output like:
```sh theme={null}
aiscaletarget.thoras.ai/my-name created
```
Replacing an AIScaleTarget that existed before? Create the new one with the
`thoras.ai/inherit-history: "true"` annotation to [keep the earlier
history](/reference/ast-definition#keeping-history-when-an-ast-is-recreated).
### 3. Verify the `AIScaleTarget`
Check that your `AIScaleTarget` was created:
```sh theme={null}
kubectl get aiscaletarget -n my-namespace
```
To see details:
```sh theme={null}
kubectl describe aiscaletarget my-app -n my-namespace
```
Note: An existing `AIScaleTarget` can be exported to a YAML for version control
or editing with:
```sh theme={null}
kubectl get aiscaletarget -n -o yaml > my-ast.yaml
```
## Understanding Vertical and Horizontal Scaling Modes
An `AIScaleTarget` can have both `vertical` and `horizontal` sections
configured, each with its own `mode` setting (`recommendation` or `autonomous`).
However, it is important to understand how these interact.
### Recommendation Mode (Both Directions)
When both vertical and horizontal are in `recommendation` mode, Thoras provides
suggestions for both scaling directions. These suggestions are mutually
exclusive, and should be treated as **one or the other**.
For example, if Thoras suggests 1Gi memory (vertical) and 3 pods (horizontal),
the workload could be rightsized by either:
* Applying the vertical suggestion (1Gi memory), **OR**
* Applying the horizontal suggestion (3 pods)
Applying both suggestions would result in an over-provisioned workload.
### Autonomous Mode (One Direction Only)
Only one scaling direction can be in `autonomous` mode at a time. When enabling
autonomous mode, you choose whether Thoras should automatically scale vertically
or horizontally.
Valid configurations:
| Horizontal Mode | Vertical Mode | Valid |
| - | - | - |
| `recommendation` | `recommendation` | ✓ |
| `autonomous` | `recommendation` | ✓ |
| `recommendation` | `autonomous` | ✓ |
| `autonomous` | `autonomous` | ✗ |
When enrolling a workload into autonomous mode, consider which scaling direction
best fits your workload:
* **Horizontal autonomous** is recommended for workloads with spiky usage that
require rapid scaling to handle traffic bursts.
* **Vertical autonomous** is ideal for workloads that prioritize efficient
cluster bin packing and cost optimization.
Contact Thoras' support with any questions about which scaling direction is the
best option for your workloads and business needs.
## Using Label Selectors to Target Multiple Workloads
Instead of targeting a single workload by name with `scaleTargetRef`, you can
use `selector` to apply the same vertical scaling policy to multiple workloads
that share common pod labels.
### When to Use Label Selectors
Label selectors are useful when you want to:
* Scale a workload that uses a blue/green deployment strategy
* Scale workloads dynamically based on labels without hardcoding specific names
* Manage scaling configurations at a higher level of abstraction
### Example: Targeting Multiple Workloads by Label
Create an `AIScaleTarget` using a label selector:
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: production-services
namespace: {{YOUR_NAMESPACE}}
spec:
selector:
matchLabels:
app: my-service
environment: production
model:
forecast_blocks: 4h
forecast_interval: 1h
vertical:
containers:
- name: {{CONTAINER_NAME}}
cpu:
lowerbound: 20m
upperbound: 1
memory:
lowerbound: 50Mi
upperbound: 2G
mode: recommendation
update_policy:
update_mode: in_place_or_recreate
```
This configuration targets all Deployments, StatefulSets, and Rollouts in the
namespace whose pods have both `app: my-service` and `environment: production`
labels.
**Note:** The `containers[].name` field supports wildcard patterns (e.g., `*`,
`app-*`) to match multiple containers with a single configuration. See
[Container Name Wildcard Matching](/reference/ast-definition#container-name-wildcard-matching)
for examples and matching rules.
### Important Constraints
* `selector` and `scaleTargetRef` are mutually exclusive, use one or the other
* **Horizontal scaling is not supported** with `selector`: omit
`spec.horizontal` entirely
* Only **vertical scaling** is supported when using label selectors
* All targeted workloads must be in the same namespace as the `AIScaleTarget`
* Pods must be managed by a Deployment, StatefulSet, or Argo Rollout. Standalone
pods or other controller types are not supported.
### Behavior in Autonomous Mode
When using `selector` in autonomous mode:
1. Thoras identifies all matching pod controllers in the namespace
2. When the forecaster suggests new resource requests, Thoras updates each
matching controller one by one
3. If `update_policy.update_mode` is set to `in_place` or
`in_place_or_recreate`, pods are resized in place if possible (Kubernetes
1.33+)
4. If `in_place_or_recreate` is specified and resizing is not possible or fails,
then pods are recreated with the new resource requests. If `in_place` is
specified and resizing is not possible or fails, then pods are not created
but any future pods will be right sized.
Additionally, sometimes it can be useful to sync your workload's active resource
requests with the manifests in your source code repo to keep the baseline
workload definition aligned with reality. Email
[support@thoras.ai](mailto:support@thoras.ai) to learn how to automate this.
# Cost Estimates
Source: https://docs.thoras.ai/guides/cost-estimates
Understand how Thoras calculates potential savings and waste across your Kubernetes cluster based on workload resource usage and node pricing data.
Thoras estimates cost savings and waste by analyzing your workload resource
requests against predicted usage patterns. This guide explains how these
calculations work and how to customize pricing data for your environment.
## How Cost Calculations Work
Thoras calculates potential savings by comparing your current resource
allocations with forecasted recommendations. The calculation involves three main
components:
### 1. Determining Average Node Cost
Thoras establishes a baseline "average node cost per hour" for your cluster:
1. **Node Discovery** - Collects node information similar to the output from
`kubectl get nodes -oyaml`
2. **Metadata Extraction** - Parses labels to identify:
* Cloud provider (AWS, GCP, Azure, etc.)
* Instance type (e.g., `m7g.medium`, `n2-standard-4`)
* Capacity type (on-demand or spot)
3. **Price Matching** - Matches node details against pricing data to determine
hourly cost, CPU cores, and memory
4. **Cluster Averaging** - Averages cost, cores, and memory across all nodes to
establish cluster-wide baseline
If pricing data is unavailable for a node (e.g. unknown instance type), that
node is excluded from the cluster's "average node." If no nodes can be priced,
Thoras defaults to 4 cores, 16GB memory, and \$0.19/hour (equivalent to an AWS
`t3.xlarge`).
**Note:** This approach provides ballpark estimates rather than precise billing
figures. The goal is directional guidance on how rightsizing workloads impacts
your overall cloud spend. On EKS, you can have Thoras collect your real billed
spend instead — see [AWS Cost Explorer](/guides/aws-cost-explorer).
### 2. Calculating Workload-Level Savings
For each workload, Thoras compares original controller specifications (e.g.
Deployment, StatefulSet, ArgoRollout) against forecasted resource
recommendations:
**Over-provisioned workloads** (positive delta):
* Current requests exceed predicted usage
* Labeled as **waste** in recommendation mode
* Labeled as **savings** in autonomous mode
**Under-provisioned workloads** (negative delta):
* Current requests fall below predicted usage
* Tracked as **Resources Needed** (recommendation mode) or **Resources Added**
(autonomous mode)
* Thoras recommends increasing resources to improve stability
Thoras tracks the potential reduction (or required increase) in CPU cores and
memory bytes per hour for each workload.
### 3. Cluster-Wide Cost Impact
Thoras aggregates workload-level changes across your entire cluster:
1. Sum wasted or saved CPU and memory across all workloads
2. Calculate how many fewer nodes are needed based on:
* Aggregate CPU core reduction
* Aggregate memory reduction
3. Use the **lower of the two** (CPU or memory determines node count)
4. Multiply by average node hourly cost to estimate total cluster savings
This represents the maximum potential savings if all right sizing
recommendations were applied autonomously by Thoras.
## Sourcing Pricing Data
Because Thoras operates in an air-gapped environment without real-time cloud API
access, pricing data comes from a built-in static price list that is
periodically updated in product releases.
### Handling Spot Pricing
Spot instance pricing fluctuates based on supply and demand. Rather than
tracking real-time spot prices, Thoras uses published average savings
percentages from cloud providers.
For example, according to
[AWS EC2 Spot Instance Advisor](https://aws.amazon.com/ec2/spot/instance-advisor/),
a spot `m7g.medium` provides an average 59% savings over on-demand pricing (as
of February 5, 2026).
**Note:** Your actual spot costs may differ from dashboard estimates because
Thoras uses static averages rather than current market rates.
### Preemptible Instances
Preemptible instances (GCP's predecessor to their spot offering) are priced
using spot pricing data as a proxy. While not exact, the two pricing models are
typically close. Your actual preemptible costs may vary from dashboard
estimates.
## Overriding Average Node Cost
You can manually override the calculated average node cost when:
* Your cluster uses an bespoke instance types
* You have custom pricing agreements or reserved instance discounts
* You want more precise cost calculations based on your actual billing
To override the average node cost, navigate to the cluster settings in the
Thoras dashboard and enter your custom value.
## What's Included (and What's Not)
**Included in cost calculations:**
* CPU and memory resource costs
* Node-level infrastructure pricing
**Not included in cost calculations:**
* Reserved instance or savings plan discounts
* Committed use discounts (GCP)
* Custom pricing agreements
For workloads with significant non-compute costs, treat Thoras savings estimates
as a lower bound on total potential savings.
### Horizontal Scaling Savings and Waste
Vanilla HPA is reactive by design. To protect against sudden traffic spikes,
teams typically run workloads at 20-30% utilization. Thoras's predictive
horizontal scaling allows workloads to run at much higher instantaneous
utilization (80-90%) by scaling proactively based on forecasts.
However, Thoras uses vanilla HPA as a safety backstop, always scaling to the
higher value between the Thoras forecast and vanilla HPA's recommendation.
Realizing savings from this improved utilization requires increasing HPA's
target utilization metric, which is a manual configuration change.
Because savings depend on customer action to adjust HPA settings, cost estimates
exclude horizontal scaling savings.
## Mixed Instance Types
Thoras handles heterogeneous clusters gracefully:
* **Mixed capacity types** - Averages across spot and on-demand instances
* **Multiple instance types** - Blends pricing across different machine types
* **Multi-cloud** - Averages across cloud providers if nodes span multiple
clouds
**Note:** This averaging approach ignores bin-packing optimizations that tools
like Karpenter might achieve. Actual savings may be higher if your autoscaler
can consolidate workloads onto fewer nodes.
## Node Detail Collection
Thoras snapshots node metrics every 5 minutes and rolls up aggregates hourly and
daily. Cost projections are based on these aggregated usage patterns.
# Dashboard authentication
Source: https://docs.thoras.ai/guides/dashboard-auth
How the Thoras dashboard authenticates users, and how to configure the two built-in modes: generated htpasswd credentials or OIDC login against your identity provider.
## Overview
The Thoras dashboard ships with authentication enabled by default. Requests hit
an [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/) sidecar in the
dashboard pod. Only authenticated requests reach the dashboard container.
Two modes are supported, selected via `thorasDashboard.auth.mode`:
* **`htpasswd`** (default): a username and password generated in-cluster on
first install by the Thoras config-controller. No external setup is required.
* **`oidc`**: OIDC login against your identity provider (Okta, Azure Entra ID,
or any OIDC-compliant provider). You provide the client credentials in a
Secret you manage.
Only `oidc` gives you per-person attribution in the
[audit trail](/guides/audit-trail); `htpasswd` shares one identity across
everyone with dashboard access.
```
User -> oauth2-proxy sidecar -> Thoras Dashboard (loopback)
^
htpasswd file OR OIDC IdP
```
Fields under the unused mode's block are ignored, so you can switch modes by
changing `auth.mode`.
Credentials and session cookies are only meaningful behind TLS. Terminate TLS
at your ingress before exposing the dashboard beyond `kubectl port-forward`,
and set `thorasDashboard.auth.cookieSecure: true`. The chart default is
`false` so that port-forward works over plain HTTP. This default is suitable
for evaluation only. Safari drops Secure cookies over plain HTTP, so you
cannot enable `cookieSecure` and use `kubectl port-forward` together from a
Mac.
## Pick a mode
* [Default (htpasswd) auth](/guides/dashboard-auth/htpasswd): log in with the
generated credentials, set your own, or supply your own Secret.
* [Okta](/guides/dashboard-auth/okta): OIDC login against Okta.
* [Azure Entra ID](/guides/dashboard-auth/entra): OIDC login against Entra.
Other OIDC-compliant providers work with `provider: oidc`. The Okta and Entra
guides show the general shape.
## Migrating from a hand-rolled oauth2-proxy sidecar
Before chart 5.0, the documented way to protect the dashboard was to add your
own oauth2-proxy container under `thorasDashboard.extraContainers` and retarget
the Service at port `4180`. The chart now ships that sidecar for you. Migrating
keeps your IdP app registration and your existing `oauth2-proxy-secrets` Secret.
Only `values.yaml` changes.
**Remove** the old wiring, noting the values of your existing oauth2-proxy flags
for the next step:
```yaml theme={null}
thorasDashboard:
service:
targetPort: 4180 # remove
extraContainers: # remove the entire oauth2-proxy container
- name: oauth2-proxy
# ...
```
**Add** the built-in sidecar configuration, substituting the values you noted:
```yaml theme={null}
thorasDashboard:
auth:
mode: oidc
oidc:
provider: oidc # or entra-id; reuse your existing --provider value (Okta uses oidc)
issuerURL:
redirectURL:
emailDomains: [yourcompany.com] # your --email-domain value(s)
existingSecret:
secretName: oauth2-proxy-secrets
```
The chart reads `client-id`, `client-secret`, and `cookie-secret` from the
existing Secret by default (key names are configurable under
`auth.oidc.existingSecret`). The Service goes back to targeting the dashboard's
`containerPort`, now owned by the chart's sidecar, so no `service.targetPort`
override is needed.
## Bring your own auth
Set `thorasDashboard.auth.enabled: false` if you already gate the dashboard
somewhere else, such as an auth plugin at the ingress or gateway, a service-mesh
policy, or a custom SSO sidecar wired through `thorasDashboard.extraContainers`
and `thorasDashboard.service.targetPort`.
```yaml theme={null}
thorasDashboard:
auth:
enabled: false
```
With built-in auth disabled, the dashboard's nginx listens on
`thorasDashboard.containerPort` with nothing in front of it. Any workload in
the cluster that can reach the `thoras-dashboard` Service can then use part of
the Thoras API without authentication, bypassing the shared `apiClientSecret`.
Only disable built-in auth if something external gates every request. The
[audit trail](/guides/audit-trail) also can't attribute actions to a person in
this mode; every action is recorded as `unknown`.
## NetworkPolicy and OIDC
If you enable the chart's
[NetworkPolicy](/installation/advanced-setup#networkpolicy) alongside OIDC mode,
the chart opens egress on port `443` from the dashboard pod so the sidecar can
reach your identity provider. Tighten this with a layered policy scoped to your
IdP's hostnames (for example a `CiliumNetworkPolicy` with `toFQDNs`) if your
security posture requires it.
## Further reading
* [oauth2-proxy documentation home](https://oauth2-proxy.github.io/oauth2-proxy/)
* [Configuration overview / full flag reference](https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview)
* [Session storage options](https://oauth2-proxy.github.io/oauth2-proxy/configuration/session_storage)
* [TLS configuration](https://oauth2-proxy.github.io/oauth2-proxy/configuration/tls)
# Azure Entra OIDC dashboard auth
Source: https://docs.thoras.ai/guides/dashboard-auth/entra
Log into the Thoras dashboard via Azure Entra ID (formerly Azure AD) using the chart's built-in oauth2-proxy sidecar in OIDC mode.
This guide covers the Entra-specific steps for authenticating dashboard users
against Azure Entra ID. See the
[dashboard authentication overview](/guides/dashboard-auth) for how the built-in
sidecar works and how the two auth modes compare.
If you already run your own oauth2-proxy sidecar against Azure Entra ID from a
pre-5.0 install, your app registration and Secret carry over unchanged. See
[Migrating from a hand-rolled oauth2-proxy
sidecar](/guides/dashboard-auth#migrating-from-a-hand-rolled-oauth2-proxy-sidecar).
## Setup
### 1. Register an app in Azure Entra
1. In the Azure Portal, go to **Microsoft Entra ID -> App registrations -> New
registration**.
2. For **Supported account types**, pick **Accounts in this organizational
directory only (single tenant)**. Multi-tenant works but needs extra
configuration (see the `issuerURL` field note below).
3. Set a **Redirect URI** of type "Web":
`https://thoras.yourcompany.com/oauth2/callback`.
4. Note the **Application (client) ID** and **Directory (tenant) ID**. You'll
need both.
5. Under **Certificates & secrets**, create a new client secret and copy its
value immediately (it's only shown once).
6. **API permissions** need no changes. The default `Microsoft Graph User.Read`
permission is enough for sign-in; oauth2-proxy requests the
`openid email profile` scopes at sign-in, and users consent to them on first
login.
![azure-app-registration]()
See oauth2-proxy's own walkthrough of this same registration process:
[Microsoft Entra ID provider config](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/ms_entra_id).
### 2. Create the Kubernetes secret
The sidecar needs three values, read from a Kubernetes Secret you manage rather
than inlined in `values.yaml`. The chart never generates these:
| Key | Purpose |
| - | - |
| `client-id` | The application's client ID |
| `client-secret` | The application's client secret |
| `cookie-secret` | A random value oauth2-proxy uses to encrypt its session cookie |
Generate the cookie secret with:
```bash theme={null}
openssl rand -base64 32 | head -c 32 | base64
```
Create the secret (values below are base64-encoded automatically by `kubectl`
when using `--from-literal`):
```bash theme={null}
kubectl create secret generic oauth2-proxy-secrets \
--namespace thoras \
--from-literal=client-id="" \
--from-literal=client-secret="" \
--from-literal=cookie-secret="$(openssl rand -base64 32 | head -c 32 | base64)"
```
### 3. Switch the chart into OIDC mode
In `values.yaml`, set `thorasDashboard.auth.mode: oidc` and fill in the OIDC
block:
```yaml theme={null}
thorasDashboard:
auth:
mode: oidc
oidc:
provider: entra-id
issuerURL: https://login.microsoftonline.com//v2.0
redirectURL: https://thoras.yourcompany.com/oauth2/callback
emailDomains: [yourcompany.com]
existingSecret:
secretName: oauth2-proxy-secrets
```
#### Field notes
* `provider`: `entra-id` selects Entra-specific handling in oauth2-proxy (group
overage via Microsoft Graph, multi-tenant issuer checks). `oidc` also works
against Entra but drops those Entra-specific features.
* `issuerURL`: points at your Entra tenant's v2.0 OIDC endpoint. Replace
`` with your Directory (tenant) ID. For a multi-tenant app, use
`https://login.microsoftonline.com/common/v2.0` and add
`--insecure-oidc-skip-issuer-verification` to `thorasDashboard.auth.extraArgs`
(oauth2-proxy will otherwise reject the per-tenant issuer returned in tokens).
See the
[Entra ID provider docs](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/ms_entra_id)
for the full multi-tenant setup, including `entra-id-allowed-tenant`.
* `redirectURL`: must be the real external hostname of the dashboard, over
HTTPS, and must exactly match a redirect URI registered on the Entra app
registration.
* `emailDomains`: restricts login to your org's email domain(s). Use `["*"]` to
allow any authenticated user.
* `existingSecret.secretName`: points at the Secret you created in step 2.
### 4. Deploy and verify
1. `helm upgrade` the release with the updated `values.yaml`.
2. Confirm the dashboard pod comes up healthy, and check the sidecar logs if
something goes wrong:
```bash theme={null}
kubectl -n thoras get pods -l app=thoras-dashboard
kubectl -n thoras logs deploy/thoras-dashboard -c oauth2-proxy
```
3. Visit the dashboard URL. You should be redirected to your identity provider's
login page, and land back on the dashboard after authenticating.
4. If the redirect fails or loops, double-check that `redirectURL` exactly
matches a redirect URI registered on your identity provider's app, and that
`issuerURL` is correct.
## Exposing the sidecar externally
The chart's built-in oauth2-proxy sidecar owns the dashboard's `containerPort`,
so the existing `thorasDashboard` Service already routes external traffic
through the auth check. No `service.targetPort` override is needed.
Expose the Service using whichever mechanism the chart already supports:
**Ingress**
```yaml theme={null}
thorasDashboard:
ingress:
enabled: true
hosts:
- host: thoras.yourcompany.com
paths:
- path: /
pathType: Prefix
```
**Gateway API**
```yaml theme={null}
thorasDashboard:
gatewayAPI:
enabled: true
parentRefs:
- name: eg
sectionName: http
hostnames:
- thoras.yourcompany.com
```
Adjust `ingressClassName`/`parentRefs` to match your cluster's actual Ingress
controller or `Gateway`, and terminate TLS there (or upstream at a
cert-manager-issued `Certificate` bound to it). Once DNS for
`thoras.yourcompany.com` resolves and a valid TLS cert is issued, visiting the
dashboard URL should trigger the login flow described above.
## Further reading
* [oauth2-proxy documentation home](https://oauth2-proxy.github.io/oauth2-proxy/)
* [Microsoft Entra ID provider configuration](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/ms_entra_id)
* [Configuration overview / full flag reference](https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview)
* [Session storage options](https://oauth2-proxy.github.io/oauth2-proxy/configuration/session_storage)
* [TLS configuration](https://oauth2-proxy.github.io/oauth2-proxy/configuration/tls)
# Default (htpasswd) dashboard auth
Source: https://docs.thoras.ai/guides/dashboard-auth/htpasswd
Log into the Thoras dashboard with the generated username and password, or provide your own.
## Overview
`htpasswd` is the default authentication mode. On first install, the Thoras
config-controller generates a password and session cookie secret and stores them
in the `thoras-config-controller` Secret. The username is `thoras`. You can set
your own values, pin them for GitOps, or point the chart at a Secret you manage
yourself.
When you're ready to integrate with an identity provider, see the
[dashboard authentication overview](/guides/dashboard-auth) for OIDC options.
## Logging in
The default username is `thoras` (`thorasDashboard.auth.htpasswd.username`).
Fetch the generated password with:
```bash theme={null}
kubectl get secret thoras-config-controller -n thoras \
-o jsonpath='{.data.dashboard-auth-password}' | base64 -d
```
The Secret appears once config-controller has finished its first reconcile,
usually within a minute of install. Until then the dashboard pod reports
`CreateContainerConfigError`. This clears on its own.
## Setting the password explicitly
Set the values in `values.yaml` to pin them. The chart stores pinned values in
the `thoras-helm-values` Secret. That Secret is rendered without random
generation or Helm `lookup`, so it is safe for GitOps tools such as Argo CD and
Flux.
```yaml theme={null}
thorasDashboard:
auth:
htpasswd:
username: thoras
password:
cookieSecret: <16, 24, or 32 byte secret> # e.g. openssl rand -base64 32
```
## Using an existing Secret
Point the dashboard at a Secret you manage yourself, for example one provisioned
by Sealed Secrets, External Secrets, or SOPS.
```yaml theme={null}
thorasDashboard:
auth:
htpasswd:
existingSecret:
secretName: my-dashboard-auth
passwordKey: dashboard-auth-password
cookieSecretKey: dashboard-auth-cookie-secret
```
`existingSecret` takes precedence over `password` and `cookieSecret`.
## Rotating credentials
The generated credentials are not intended to be rotated in place. If you need
control over rotation, take ownership of the values by pinning them in
`values.yaml` or by pointing the chart at an `existingSecret` you manage. The
config-controller watches both and restarts the dashboard automatically when the
value changes, so there is no manual `kubectl rollout restart` step:
* **Pinned in values:** change the value in `values.yaml` and `helm upgrade`.
* **Existing Secret:** update your Secret. The change is picked up within about
a minute.
Rotating the cookie secret logs out every active dashboard session.
## Switching to OIDC
To hand authentication off to your identity provider, see the
[dashboard authentication overview](/guides/dashboard-auth) and pick the guide
for your IdP.
# Okta OIDC dashboard auth
Source: https://docs.thoras.ai/guides/dashboard-auth/okta
Log into the Thoras dashboard via Okta using the chart's built-in oauth2-proxy sidecar in OIDC mode.
This guide covers the Okta-specific steps for authenticating dashboard users
against Okta. See the
[dashboard authentication overview](/guides/dashboard-auth) for how the built-in
sidecar works and how the two auth modes compare.
If you already run your own oauth2-proxy sidecar against Okta from a pre-5.0
install, your app registration and Secret carry over unchanged. See [Migrating
from a hand-rolled oauth2-proxy
sidecar](/guides/dashboard-auth#migrating-from-a-hand-rolled-oauth2-proxy-sidecar).
## Setup
### 1. Register an app in Okta
1. In the Okta Admin Console, go to **Applications -> Applications -> Create App
Integration**.
2. Choose **OIDC - OpenID Connect** as the sign-in method and **Web
Application** as the application type.
3. Set a **Sign-in redirect URI** of
`https://thoras.yourcompany.com/oauth2/callback`.
4. Under **Assignments**, grant access to the group(s) that should be able to
reach the dashboard (or "Everyone" for org-wide access).
5. Once created, note the **Client ID** and **Client secret** from the app's
**General** tab, and your **Okta domain** (e.g. `dev-133337.okta.com`) from
the top of the Admin Console.
Your issuer URL is `https:///oauth2/default` if you're using
the default custom authorization server,
`https:///oauth2/` if you've configured a
custom authorization server under **Security -> API**, or
`https://` if you're using the org authorization server.
**The `default` custom authorization server may not be available or usable.**
Custom authorization servers (including the one named `default`) are part of
Okta's [API Access Management](https://developer.okta.com/docs/concepts/api-access-management/)
add-on. Production orgs that don't have that add-on cannot use `/oauth2/default`
at all, and newly provisioned Integrator Free Plan orgs ship without an Access
Policy on it. Without a matching Access Policy, login fails with `You are not
allowed to access this app` and a `no_matching_policy` error in the Okta system
log. Fix it one of two ways:
* **Add an Access Policy** to the default custom auth server under **Security ->
API -> Authorization Servers -> default -> Access Policies**, then add a rule
that applies to your app.
* **Use the org authorization server instead** by dropping `/oauth2/default`
from the issuer URL. Set the issuer to `https://`. This
works on every Okta org and needs no add-on. See
[Authorization servers (Okta docs)](https://developer.okta.com/docs/concepts/auth-servers/).
See oauth2-proxy's own Okta walkthrough (under the OpenID Connect provider,
which is what Okta uses):
[OpenID Connect - Okta](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/openid_connect#okta),
and Okta's writeup of the same integration:
[Add Auth to Any App with OAuth2 Proxy](https://developer.okta.com/blog/2022/07/14/add-auth-to-any-app-with-oauth2-proxy).
### 2. Create the Kubernetes secret
The sidecar needs three values, read from a Kubernetes Secret you manage rather
than inlined in `values.yaml`. The chart never generates these:
| Key | Purpose |
| - | - |
| `client-id` | The application's client ID |
| `client-secret` | The application's client secret |
| `cookie-secret` | A random value oauth2-proxy uses to encrypt its session cookie |
Generate the cookie secret with:
```bash theme={null}
openssl rand -base64 32 | head -c 32 | base64
```
Create the secret (values below are base64-encoded automatically by `kubectl`
when using `--from-literal`):
```bash theme={null}
kubectl create secret generic oauth2-proxy-secrets \
--namespace thoras \
--from-literal=client-id="" \
--from-literal=client-secret="" \
--from-literal=cookie-secret="$(openssl rand -base64 32 | head -c 32 | base64)"
```
### 3. Switch the chart into OIDC mode
In `values.yaml`, set `thorasDashboard.auth.mode: oidc` and fill in the OIDC
block:
```yaml theme={null}
thorasDashboard:
auth:
mode: oidc
oidc:
provider: oidc
issuerURL: https:///oauth2/default
redirectURL: https://thoras.yourcompany.com/oauth2/callback
emailDomains: [yourcompany.com]
existingSecret:
secretName: oauth2-proxy-secrets
```
#### Field notes
* `provider`: `oidc`. oauth2-proxy has no Okta-specific provider; Okta is served
by the generic OpenID Connect provider, which is what oauth2-proxy's own Okta
example uses. `okta` is not a valid value and the sidecar will fail to start.
* `issuerURL`: points at your Okta authorization server. Use
`https:///oauth2/default` for the default custom
authorization server, `https:///oauth2/` for
another custom one, or `https://` for the org authorization
server (see the Warning above).
* `redirectURL`: must be the real external hostname of the dashboard, over
HTTPS, and must exactly match the sign-in redirect URI registered on the Okta
app.
* `emailDomains`: restricts login to your org's email domain(s). Use `["*"]` to
allow any authenticated user.
* `existingSecret.secretName`: points at the Secret you created in step 2.
### 4. Deploy and verify
1. `helm upgrade` the release with the updated `values.yaml`.
2. Confirm the dashboard pod comes up healthy, and check the sidecar logs if
something goes wrong:
```bash theme={null}
kubectl -n thoras get pods -l app=thoras-dashboard
kubectl -n thoras logs deploy/thoras-dashboard -c oauth2-proxy
```
3. Visit the dashboard URL. You should be redirected to your identity provider's
login page, and land back on the dashboard after authenticating.
4. If the redirect fails or loops, double-check that `redirectURL` exactly
matches a redirect URI registered on your identity provider's app, and that
`issuerURL` is correct.
## Exposing the sidecar externally
The chart's built-in oauth2-proxy sidecar owns the dashboard's `containerPort`,
so the existing `thorasDashboard` Service already routes external traffic
through the auth check. No `service.targetPort` override is needed.
Expose the Service using whichever mechanism the chart already supports:
**Ingress**
```yaml theme={null}
thorasDashboard:
ingress:
enabled: true
hosts:
- host: thoras.yourcompany.com
paths:
- path: /
pathType: Prefix
```
**Gateway API**
```yaml theme={null}
thorasDashboard:
gatewayAPI:
enabled: true
parentRefs:
- name: eg
sectionName: http
hostnames:
- thoras.yourcompany.com
```
Adjust `ingressClassName`/`parentRefs` to match your cluster's actual Ingress
controller or `Gateway`, and terminate TLS there (or upstream at a
cert-manager-issued `Certificate` bound to it). Once DNS for
`thoras.yourcompany.com` resolves and a valid TLS cert is issued, visiting the
dashboard URL should trigger the login flow described above.
## Further reading
* [oauth2-proxy documentation home](https://oauth2-proxy.github.io/oauth2-proxy/)
* [oauth2-proxy OpenID Connect provider - Okta section](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/openid_connect#okta)
* [Add Auth to Any App with OAuth2 Proxy (Okta blog)](https://developer.okta.com/blog/2022/07/14/add-auth-to-any-app-with-oauth2-proxy)
* [Okta authorization servers (org vs. custom)](https://developer.okta.com/docs/concepts/auth-servers/)
* [Configuration overview / full flag reference](https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview)
* [Session storage options](https://oauth2-proxy.github.io/oauth2-proxy/configuration/session_storage)
* [TLS configuration](https://oauth2-proxy.github.io/oauth2-proxy/configuration/tls)
# Diagnostics Export
Source: https://docs.thoras.ai/guides/diagnostics
Download a diagnostics package for troubleshooting and support
A diagnostics package is a zip file containing operational data about your
Thoras deployment. Since Thoras runs self-hosted in your cluster, this is the
primary way to share deployment context with the Thoras team when you need
support or want a configuration review.
It does not include application secrets, credentials, environment variables,
workload application logs, or business data. You can always unzip and inspect it
before sharing.
## What's included
* Cluster info (node types, capacity, availability zones)
* Thoras configuration and workload scaling settings
* Scaling history (suggestions, metrics, resource usage over time)
* Cost and utilization data
* Optionally, Thoras service logs
## Downloading from the dashboard
### Cluster-wide export
From the dashboard homepage, open the **Manage Cluster** dropdown in the header
and select **Download diagnostics**. This exports data for all enrolled
AIScaleTargets.
The dashboard uses a 7-day time range and includes Thoras pod logs by default
(you can uncheck this). Full per-workload metrics are excluded from cluster-wide
exports to keep file size manageable.
### Single-AST export
Navigate to a specific AIScaleTarget page, then open **Manage Cluster** and
select **Download diagnostics**. This exports data for that workload only, with
full metrics included.
To include all ASTs from this page instead, check **Include all AI Scale
Targets** in the export dialog.
## Downloading via API
Port-forward the API server:
```bash theme={null}
kubectl port-forward svc/thoras-api-server-v2 8080:80 -n thoras
```
Then download:
```bash theme={null}
curl -o diagnostics.zip \
"http://localhost:8080/v1/export?start=2026-10-01T00%3A00%3A00Z&end=2026-10-14T00%3A00%3A00Z&include-pod-logs=true&include-metrics=false"
```
### Query parameters
| Parameter | Type | Required | Default | Description |
| - | - | - | - | - |
| `start` | ISO 8601 timestamp | Yes | -- | Start of the time range |
| `end` | ISO 8601 timestamp | Yes | -- | End of the time range |
| `include-pod-logs` | boolean | No | `false` | Include Thoras service pod logs |
| `include-metrics` | boolean | No | `false` | Include full metrics for the time range. When false, only the last 10 minutes of metrics are included per AST. |
| `namespace` | string | No | all | Filter to a specific namespace |
| `targetName` | string | No | all | Filter to a specific AIScaleTarget name (requires `namespace`) |
### Examples
**Cluster-wide with pod logs:**
```bash theme={null}
curl -o cluster-diagnostics.zip \
"http://localhost:8080/v1/export?start=2026-10-01T00%3A00%3A00Z&end=2026-10-14T00%3A00%3A00Z&include-pod-logs=true&include-metrics=false"
```
**Single workload with full metrics:**
```bash theme={null}
curl -o workload-diagnostics.zip \
"http://localhost:8080/v1/export?start=2026-10-01T00%3A00%3A00Z&end=2026-10-14T00%3A00%3A00Z&include-metrics=true&namespace=my-namespace&targetName=my-workload"
```
**All workloads in a namespace:**
```bash theme={null}
curl -o namespace-diagnostics.zip \
"http://localhost:8080/v1/export?start=2026-10-01T00%3A00%3A00Z&end=2026-10-14T00%3A00%3A00Z&namespace=my-namespace"
```
Full metrics can be large for clusters with many ASTs. For cluster-wide
exports, consider leaving `include-metrics=false` and only requesting full
metrics for specific workloads you're investigating.
## Sharing with Thoras
Send your diagnostics package to the Thoras team via your shared Slack channel,
email at [support@thoras.ai](mailto:support@thoras.ai), or any file transfer tool your team prefers (e.g.,
[magic-wormhole](https://github.com/magic-wormhole/magic-wormhole)).
# Enabling Autonomous Mode
Source: https://docs.thoras.ai/guides/enabling-autonomous-mode
Enable predictive scaling with confidence
Following this quick checklist helps ensure your workloads are ready for
predictive autoscaling.
## Autonomous Mode Checklist
1. For autonomously scaling, Thoras requires the ability to persist historical
metrics, which requires a Kubernetes persistent volume. See
[Persistent Storage](../installation/persistent-storage) for a walkthrough
2. Forecast worker capacity scales automatically as you enroll more workloads.
If you need to pin a fixed pool size, see
[Forecast worker capacity](../installation/advanced-setup#forecast-worker-capacity)
3. Deploying Thoras with ArgoCD? Check out the
[ArgoCD config considerations](../guides/argo-cd)
4. \[Optional] `kubectl` port-forwarding is a great way to experiment, but you
may eventually want to expose the dashboard service internally to enable
broader, more stable access. You can do this by enabling `Ingress` via
`$.Values.thorasDashboard.ingress` or `GatewayAPI` via
`$.Values.thorasDashboard.gatewayAPI`. See
[GitHub](https://github.com/thoras-ai/helm-charts/blob/main/charts/thoras/README.md#thoras-dashboard)
for further documentation
5. **Check predictions in the dashboard** - confirm metric suggestions for your
workload are accurate. The Thoras reasoning engine typically needs to see a
pattern twice to learn about it; so model performance can take between a few
minutes to a few days to reach maximum accuracy.
![predictions-vs-usage]()
6. **Wait for sufficient historical data** - Autonomous scaling requires at
least 24 hours of historical data by default before it can be enabled. This
ensures enough data confidence for safe scaling decisions. Configure the
default via the Helm value `thorasForecast.minLookbackToScale`, or override
it per-AST via `model.min_lookback_to_scale` in the
[AST definition](../reference/ast-definition#modelmin_lookback_to_scale).
7. **Understand application behavior** - for vertical scaling, ensure the
workload and its dependencies can handle periodic restarts. See the
[vertical pod rightsizing](../guides/vertical-pod-rightsizing) and
[aiscaletarget](../reference/ast-definition) pages for an overview of scaling
and restart configuration options.
8. **Choose a scaling direction** - only one scaling direction (horizontal or
vertical) can be in autonomous mode at a time. Choose the direction that best
fits your workload:
* **Horizontal** for workloads with variable traffic patterns that need to
scale the number of replicas to handle demand spikes.
* **Vertical** for workloads with more predictable resource patterns that
benefit from optimized CPU and memory requests to improve cluster
utilization and reduce costs. **Note:** before enabling `Autonomous Mode`,
verify no competing vertical autoscaling controllers are modifying pod
resource requests for the same workloads. Running multiple controllers that
attempt to adjust pod resource allocation can lead to unstable behavior,
oscillation, or conflicting decisions.
See
[Understanding Vertical and Horizontal Scaling Modes](./configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
for more details.
# Choosing a Forecast Optimization Strategy
Source: https://docs.thoras.ai/guides/forecast-optimization-strategies
Thoras forecasts future demand for your workload and recommends resource levels
accordingly. You can choose an optimization strategy that reflects your
operational priorities.
All strategies forecast demand and remain peak-aware. The difference is how
closely recommendations track usage versus how much additional margin they
maintain.
Thoras modes span a spectrum from cost optimization to assurance focused:
`undershoot` → `cost_optimized` → `balanced` → `assurance_optimized` →
`overshoot`
Because Thoras is forecasting future demand, no mode eliminates forecast misses
entirely. More aggressive modes are more likely to run closer to real demand,
while more conservative modes maintain more room above it.
`percentile` is an alternative mode that sets the forecast directly from
historical usage rather than predicting future demand.
## Quick Reference
| Strategy | Approach | Recommended For |
| - | - | - |
| `undershoot` | Allows recommendations to run below usage more often | Non-critical or resilient jobs |
| `cost_optimized` | Tracks tightly with minimal headroom | Async, retry-safe workloads |
| `balanced` (default) | Tracks usage with balanced efficiency and assurance | General production workloads |
| `assurance_optimized` | Tracks above usage with moderate headroom | User-facing or consistency-sensitive workloads |
| `overshoot` | Maintains elevated headroom based on historical usage | Critical workloads requiring additional consistency |
| `percentile` | Forecast set at Nth percentile of historical usage | Stable workloads with a specific coverage target |
## Mode Details
### `balanced` (default)
* Balances efficiency and assurance with comfortable headroom
* Provides strong coverage against historically observed demand volatility
* Choose this for production workloads where you want both optimization and
consistency
### `cost_optimized`
* Uses tighter recommendations to reduce unused capacity
* May allow occasional brief resource pressure during rare demand spikes
* Best for workloads that tolerate moderate variability: background jobs, async
services, retry-safe systems
### `undershoot`
* Most aggressive low-headroom strategy
* Intentionally allows recommendations to run below observed demand more often
* Best for non-critical workloads, batch processing, or systems with strong
retry logic and fallback behavior
### `assurance_optimized`
* Tracks above typical usage patterns while remaining responsive to changes
* Stays close to usage for steady workloads, increasing headroom when
variability rises
* Best for workloads where consistency is important without significantly
increasing resource usage
### `overshoot`
* Maintains consistent headroom based on historical usage levels
* Less responsive to short-term changes in usage patterns
* Best for workloads where maintaining additional headroom is preferred over
tighter recommendations
### `percentile`
* Forecast is set to the Nth percentile of observed historical usage
* `percentile_value` (0–100) is required and determines the coverage level: a
value of `80` means 80% of historical data points fell at or below the
forecast value
* Best for workloads where you want to set a specific historical coverage target
rather than rely on demand prediction
```yaml theme={null}
spec:
model:
mode: percentile
percentile_value: 80
```
If not specified, the default is `mode: balanced`.
## Configuring Forecast Mode
Set the optimization strategy in your AIScaleTarget:
```yaml theme={null}
spec:
model:
mode: assurance_optimized
```
You can also configure different modes per metric. See
[model.metrics reference](/reference/ast-definition#model-metrics) for details.
## Fine-Tuning Forecast Behavior
After choosing a forecast optimization strategy, you can further tune forecast
behavior using additional settings. These are best used for workload-specific
adjustments after selecting your primary mode.
### `forecast_buffer_percentage`
Adds a fixed percentage on top of the forecasted recommendation.
**Use this when:**
* The forecast is generally reasonable, but you want it to run slightly higher
or lower overall
**Reference:**
[model.forecast\_buffer\_percentage](/reference/ast-definition#model-forecast-buffer-percentage)
### `forecast_interval`
Controls how often Thoras generates a new forecast.
**Use this when:**
* Workload behavior changes quickly and recommendations need to refresh more
often
**Reference:**
[model.forecast\_interval](/reference/ast-definition#model-forecast-interval)
### `forecast_blocks`
Controls how far into the future each forecast covers. Thoras sizes for the
predicted **peak** inside that window, so a longer window holds capacity for any
peak that falls within it.
**Use this when:**
* You want Thoras to account for demand farther ahead in time
**Note:** `forecast_blocks` should be greater than or equal to
`forecast_interval` to ensure each forecast covers the next update window.
**Reference:**
[model.forecast\_blocks](/reference/ast-definition#model-forecast-blocks)
### Quick Guidance
If you're deciding what to adjust:
* Change **`mode`** when you want forecasts to behave more aggressively or
conservatively
* Change **`forecast_buffer_percentage`** when the forecast shape looks right,
but you want to raise or lower the average prediction
* Change **`forecast_interval`** when you want forecasts to update more or less
frequently
* Change **`forecast_blocks`** when you want to adjust how far ahead forecasts
look (longer horizons are more conservative but may increase overprovisioning)
**In most cases, `model.mode` should be your first tuning lever.** The settings
above are for smaller workload-specific adjustments.
Your model mode also determines default scaling thresholds, controlling how
large a suggested change must be before a scaling action fires. See
[`scaling_behavior`](/reference/ast-definition#scaling_behavior) for details.
# Predictive Horizontal Pod Autoscaling
Source: https://docs.thoras.ai/guides/hpa
This guide explains how to set up Predictive Horizontal Pod Autoscaling in
Kubernetes using Thoras, including the difference between recommendation and
autonomous modes, and how to scale using external or custom metrics.
## What is Traditional Horizontal Pod Autoscaling?
Horizontal Pod Autoscaling (HPA) is a Kubernetes feature that adjusts the number
of pods in a deployment, statefulset, or other scalable resource based on
specified metrics, which can include built-in metrics (CPU, memory) or
external/custom metrics (e.g. queue length, latency). HPA increases or decreases
the number of pods ***after*** metrics cross pre-defined thresholds
**Limitations of Traditional HPA:**
While traditional HPA is powerful, it only reacts to spikes in usage after they
happen. This means your application may scale up too late to handle sudden
surges in traffic, forcing you to keep extra capacity running "just in case",
which can be costly and inefficient.
Another common shortcoming with traditional HPA is that it is prone to
rubber-banding if not carefully managed. For example, a scale down quickly
followed by a scale back up, which can negatively impact both reliability and
cost through pod churn.
**How Thoras Improves Autoscaling:**
Thoras brings Predictive Horizontal Pod Autoscaling to Kubernetes. Instead of
waiting for a spike to occur, Thoras analyzes trends and forecasts future
demand, scaling your pods ***before*** the spike even begins. This proactive
approach gives workloads greater resilience and allows running at much higher
utilization targets, eliminating the need to maintain a large buffer of spare
capacity "just in case". Ultimately, this can significantly reduce cloud waste
while maximizing resilience to increases in usage.
In this guide, we will walk you through how to set up Predictive Horizontal Pod
Autoscaling with Thoras so your pods can scale ahead of demand spikes, giving
you both efficiency and peace of mind.
## Thoras Scaling Modes: Recommendation vs. Autonomous
Thoras supports two modes for managing scaling:
### Recommendation Mode
* Thoras analyzes your workload usage, makes accurate predictions of your
workload's *future usage*, and suggests the optimal number of pods based on
those predictions.
* You (or your automation) can review and apply these recommendations.
### Autonomous Mode
* Thoras actively manages horizontal scaling for your workload by providing your
HPA the ability to scale on future usage in addition to real-time usage. The
number of desired pods for any given scaling decision will always be the
higher number of suggested pods between real-time (provided by HPA) and
future-time (provided by Thoras).
## How Thoras and HPA coordinate
Thoras and HPA run side by side and each contribute to the final replica count.
Thoras never replaces or modifies your HPA.
HPA is reactive: it responds to what is happening right now, scaling when
metrics cross a threshold. Thoras is predictive: it scales based on where your
workload is headed, ahead of demand. Together, HPA handles truly unexpected
spikes while Thoras handles predictable load patterns. Neither operates in
isolation, and neither replaces the other.
## Setting Up Predictive HPA with Thoras
### Prerequisites
* A running Kubernetes cluster with
[Kubernetes metrics server](https://github.com/kubernetes-sigs/metrics-server)
installed
* Thoras installed and running (see
[quickstart guide](../getting-started/quickstart))
### 1. Create an HPA (if not already present)
A single existing HPA is required for each target deployment, which is
turbo-charged with Thoras.
Check whether an HPA exists
```sh theme={null}
kubectl get hpa -n
```
For Thoras to work autonomously, you must have an HPA defined for that workload.
However, if you just want to view recommendations, Thoras can do so without an
HPA by assuming you want suggestions that would scale you to a target of 80%
utilization. Example defining HPA for `AIScaleTarget`:
```sh theme={null}
kubectl autoscale deployment -n --cpu-percent=80 --min=1 --max=10
```
### 2. Register an `AIScaleTarget`
See [Configuring AIScaleTargets](./configuring-asts) for detailed steps on
creating and applying an `AIScaleTarget` manifest.
### 3. Setting Horizontal Mode \[ recommendation | autonomous ]
#### Recommendation Mode
`AIScaleTarget`s should start in `recommendation` mode while the models learn
usage and scaling patterns. To enable `recommendation` mode simply set the
`AIScaleTarget`'s `spec.horizontal.mode` to `recommendation`. For example, in
my-ast.yaml:
```yaml theme={null}
---
spec:
horizontal:
mode: recommendation
```
Apply the update
```sh theme={null}
kubectl apply -f my-ast.yaml
```
#### Autonomous Mode
Workloads can be switched to `autonomous` with as little as 24 hours of data
(the default, controlled by `thorasForecast.minLookbackToScale` in the Helm
chart, or overridden per-AST via `model.min_lookback_to_scale`). However, if
your workload has week-day seasonality, we recommend running in `recommendation`
mode for as much as a week to ensure the models can capture the seasonality.
After models are sufficiently trained in `recommendation` mode, the mode may be
switched to `autonomous` where scaling recommendations are automatically
applied. To enable `autonomous` mode simply set the `AIScaleTarget`'s
`spec.horizontal.mode` to `autonomous`. For example, in my-ast.yaml:
```yaml theme={null}
---
spec:
horizontal:
mode: autonomous
```
Apply the update
```sh theme={null}
kubectl apply -f my-ast.yaml
```
> **Note:** Only one scaling direction (horizontal or vertical) can be in
> `autonomous` mode at a time. See
> [Configuring AIScaleTargets](./configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
> for additional details.
## Constraining Replicas to Fixed Multiples
Some workloads require specific replica counts for even load distribution across
shards or partitions. A Kafka consumer group with 8 partitions, for example,
distributes load unevenly at any replica count that is not a multiple of 8. A
StatefulSet sharded across a fixed topology, where each pod owns a specific
shard, has the same constraint.
Use `multipleOf` on `spec.horizontal` to constrain Thoras to only scale to valid
replica counts:
```yaml theme={null}
---
spec:
horizontal:
mode: autonomous
maxReplicas: 24
multipleOf: 8
```
With this configuration, Thoras will only ever set replica counts of 8, 16,
or 24.
**Note:** if `maxReplicas` is not itself a multiple of `multipleOf`, the
effective scaling ceiling is the largest multiple that fits within
`maxReplicas`. For example, `maxReplicas: 20` with `multipleOf: 8` caps
effective scaling at 16, not 20. Set `maxReplicas` to a multiple of `multipleOf`
to avoid an unreachable ceiling.
See [`horizontal.multipleOf`](/reference/ast-definition#horizontal-multipleof)
in the reference for full configuration details.
## Constraining Replicas to a Fixed Set
`multipleOf` (above) fits workloads whose valid counts are regular multiples of
a single value. Other workloads have an irregular set of valid counts. A topic
with 240 partitions, for example, distributes evenly at any count that divides
240, such as 4, 6, 8, 12, and 16. For cases like this, use `allowedReplicas` on
`spec.horizontal` to list the exact set Thoras may choose from:
```yaml theme={null}
---
spec:
horizontal:
mode: autonomous
minReplicas: 4
maxReplicas: 20
allowedReplicas: [4, 6, 8, 12, 16]
```
Thoras rounds any desired count up to the smallest value in the set that meets
or exceeds it, and never scales beyond the largest value. `allowedReplicas`
cannot be combined with `multipleOf`.
See
[`horizontal.allowedReplicas`](/reference/ast-definition#horizontal-allowedreplicas)
in the reference for full configuration details.
## Using External or Custom Metrics
By default, HPA can scale on CPU and memory. Thoras and Kubernetes also support
scaling on external or custom metrics (e.g., http request rate, queue
writes/depth).
Example: Scaling on an External Metric Add an external metric to your
`AIScaleTarget`:
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: {{YOUR_AST_NAME}}
namespace: {{YOUR_NAMESPACE}}
spec:
scaleTargetRef:
kind: Deployment
name: {{YOUR_AST_NAME}}
horizontal:
mode: autonomous
additional_metrics:
- type: External
external:
metric:
name: {{YOUR_CUSTOM_METRIC}}
selector:
matchLabels:
source: prometheus
target:
type: AverageValue
averageValue: 100
```
* `type: External`: tells HPA to use an external metric.
* `external.metric.name`: the name of the metric (must match what your metrics
adapter provides).
* `external.target.averageValue`: the target value for scaling.
## Verifying HPA and Thoras Integration
* Check HPA status:
```sh theme={null}
kubectl describe hpa -n
```
* Check `AIScaleTarget` status:
```sh theme={null}
kubectl get aiscaletarget -n -o yaml
```
* Monitor pod scaling events:
```sh theme={null}
kubectl get pods -w
```
# In-Place Resize Behavior
Source: https://docs.thoras.ai/guides/in-place-resize
How Thoras handles in-place pod resizing, what happens when a resize cannot complete, and the safety controls that protect your workloads.
When vertical scaling is configured with `update_mode: in_place` or
`update_mode: in_place_or_recreate`, Thoras patches running pods directly
through the Kubernetes resize API instead of recreating them. This page explains
what happens when that resize cannot complete and how Thoras responds depending
on your update policy.
For configuration details, see
[Update Policy](/guides/vertical-pod-rightsizing#update-policy) and the
[AIScaleTarget reference](/reference/ast-definition#update-policy).
## When a Resize Cannot Complete
After Thoras patches a pod's resource requests, the kubelet attempts to actuate
the change by adjusting the container's cgroup. Two situations can prevent this:
* **Insufficient node capacity.** An upsize requires more CPU or memory than the
node has available. The kubelet cannot expand the cgroup and the resize
stalls.
* **Memory limit below live usage.** A memory limit decrease that would drop
below the container's current working set is accepted by the API server (the
patch succeeds), but the kubelet cannot safely shrink the cgroup. It retries
on every sync cycle.
In both cases, Kubernetes reports the stall through a `PodResizeInProgress`
condition on the pod with `Reason: Error`. The pod continues running with its
previous resource allocation. No data is lost and no containers are restarted by
the stall itself.
## Behavior by Update Mode
How Thoras responds to a stalled resize depends on `update_mode`:
### `in_place`
Thoras never evicts or recreates the pod. If the kubelet cannot complete the
resize, the pod stays running with its current resources. The stall resolves on
its own when conditions change (for example, other pods are descheduled and free
up node capacity, or a new forecast lowers the recommendation).
This mode is appropriate for workloads where continuity is more important than
converging to the recommendation quickly.
### `in_place_or_recreate`
Thoras attempts in-place resize first. If a pod's resize stalls for more than
two minutes, Thoras evicts that specific pod so it can be recreated with the
correct resources by its controller (Deployment, StatefulSet, etc.). Only the
stalled pods are evicted; pods that resized successfully are left alone.
This is the recommended mode for most workloads because it combines the
low-disruption benefits of in-place resize with a reliable fallback.
## Safety Controls
Thoras applies several layers of protection when evicting pods as part of the
recreate fallback.
### PodDisruptionBudgets
Evictions use the
[Kubernetes Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/),
which enforces your PodDisruptionBudgets. If a PDB would be violated, the API
server rejects the eviction and Thoras defers the remaining pods to a later run.
This is different from a rollout restart, which replaces the entire ReplicaSet
and is not gated by PDBs at the individual pod level.
### Disruption batching
Evictions are paced using the
[`disruption`](/reference/ast-definition#disruption) settings on the
AIScaleTarget. Thoras applies changes in batches (default: up to 20% of replicas
at a time, with at least 60 seconds between batches) and counts pods that are
already terminating or not-Ready against the budget. This prevents stacking
disruption when a prior batch has not finished rolling.
### Circuit breaker
If pods keep coming back stale after eviction for the same recommendation (for
example, because the admission webhook is not applying the new values), Thoras
stops evicting. The circuit breaker opens after repeated no-progress cycles and
backs off exponentially before probing again. This prevents runaway eviction
loops.
## Observability
The following signals cover the disruption lifecycle discussed on this page:
| Signal | Type | What it tells you |
| - | - | - |
| [`thoras_disruption_circuit_breaker_open`](/guides/metrics#disruption-circuit-breaker) | Prometheus gauge | Breaker is tripped: evictions stopped for this AST |
| [`thoras_ast_disruption_completions_total`](/guides/metrics#ast-disruption-completions) | Prometheus counter | A disruption cycle finished (all stale pods converged) |
| `DisruptionInProgress` condition | AST status | A batch is actively rolling (`BatchStarted`, `BatchEvicted`, or `Complete`) |
| `DisruptionCircuitBreaker` condition | AST status | Breaker state (`Open`, `Closed`, `HalfOpenProbe`) |
Prometheus metrics are detailed on the [Monitoring Thoras](/guides/metrics)
page. AST status conditions are visible via
`kubectl get ast -n -o jsonpath='{.status.conditions}'`.
## Operational Guidance
### Concurrent upsizes on the same node
When multiple pods on the same node receive an upsize at the same time, the node
may not have enough capacity for all of them. Thoras handles each pod
independently: pods that resize successfully stay in place, and pods that stall
follow the fallback path for the configured `update_mode`.
To reduce the likelihood of node-capacity stalls:
* **Reserve headroom on nodes.** If you use a node autoscaler like Karpenter,
consider configuring a small amount of reserved capacity so upsizes have room
to land without waiting for a new node.
* **Use `disruption` settings to pace changes.** Smaller batch sizes and longer
intervals give the node autoscaler time to respond between batches.
* **Rely on forecasting.** Thoras predicts resource needs ahead of time, so
upsizes are typically applied before a spike rather than during one, reducing
contention.
### Choosing between `in_place` and `in_place_or_recreate`
| Consideration | `in_place` | `in_place_or_recreate` |
| - | - | - |
| Pod continuity | Guaranteed. Pods are never evicted by Thoras. | Pods may be evicted if resize stalls. |
| Convergence speed | Slower. Stalls resolve only when external conditions change. | Faster. Stalled pods are recreated with correct resources. |
| Best for | Stateful workloads, long-running jobs, workloads sensitive to restarts. | General-purpose services where converging to the recommendation matters more than avoiding a single pod restart. |
# JVM Workload Scaling
Source: https://docs.thoras.ai/guides/jvm-scaling
How Thoras optimizes Java-based applications
Thoras can predictively tune JVM heap settings (`-Xmx` and `-Xms`) alongside
container cpu and memory requests for Java workloads. Instead of hard-coding
heap sizes or relying on JVM ergonomics, Thoras forecasts heap and non-heap
memory usage and derives optimal values on every pod creation.
## How It Works
### Overview
Traditional JVM memory configuration is static: you set `-Xmx` (maximum heap)
and `-Xms` (initial heap) at deployment time and hope they're sufficient. If
heap usage grows, you get OOM kills. If it shrinks, you waste resources.
Thoras solves this by:
1. **Collecting JVM metrics** (`jvm_memory_used_bytes` for heap and non-heap
areas) directly from your application
2. **Forecasting** future heap and non-heap memory needs using historical
patterns
3. **Computing** optimal `-Xmx`, `-Xms`, and container memory request/limit
values
4. **Injecting** these as environment variables into pods at creation time via a
mutating admission webhook
This happens automatically on each pod creation, so your JVM settings stay in
sync with actual demand.
### The Calculation
Thoras uses three configurable buffers to compute JVM settings from its
forecasts:
| Parameter | Default | Purpose |
| - | - | - |
| `xmx_buffer` | 10% | Headroom above observed/predicted heap usage for `-Xmx` |
| `xms_ratio` | 80% | Ratio of `-Xms` to `-Xmx` (initial heap as a fraction of max heap) |
| `memory_buffer` | 10% | Headroom above total JVM memory (heap + non-heap) for the container memory request |
The calculation follows four steps:
**Step 1: Determine effective heap and non-heap usage**
Thoras takes the **greater** of the forecasted value and current observed value
for both heap and non-heap memory. This ensures the JVM is never sized below
what it's currently using.
```
effective_heap = max(heap_forecast, heap_current)
effective_nonheap = max(nonheap_forecast, nonheap_current)
```
**Step 2: Calculate Xmx (maximum heap)**
The maximum heap size is the effective heap usage plus the `xmx_buffer`:
```
Xmx = effective_heap * (1 + xmx_buffer)
```
If `xmx_bounds` are configured, the value is clamped within those bounds.
**Step 3: Calculate Xms (initial heap)**
The initial heap size is a ratio of the maximum:
```
Xms = Xmx * xms_ratio
```
**Step 4: Calculate container memory request**
The container memory request accounts for the full JVM footprint (heap +
non-heap) plus a buffer for native memory, thread stacks, metaspace, etc.:
```
container_memory = (Xmx + effective_nonheap) * (1 + memory_buffer)
```
### Worked Example
Given:
* Heap forecast: **200 MiB**
* Non-heap forecast: **50 MiB**
* All buffers at defaults (10% `xmx_buffer`, 80% `xms_ratio`, 10%
`memory_buffer`)
| Step | Calculation | Result |
| - | - | - |
| Xmx | 200 MiB \* 1.10 | **220 MiB** |
| Xms | 220 MiB \* 0.80 | **176 MiB** |
| Container memory | (220 MiB + 50 MiB) \* 1.10 | **297 MiB** |
The pod would receive:
* `THORAS_JVM_HEAP_MEMORY_XMX=-Xmx230686720` (220 MiB in bytes)
* `THORAS_JVM_HEAP_MEMORY_XMS=-Xms184549376` (176 MiB in bytes)
* Container memory request: **297 MiB**
### What Gets Injected
Thoras injects two environment variables into the JVM container on pod creation:
| Environment Variable | Format | Description |
| - | - | - |
| `THORAS_JVM_HEAP_MEMORY_XMX` | `-Xmx` | Maximum heap size flag (e.g., `-Xmx230686720`) |
| `THORAS_JVM_HEAP_MEMORY_XMS` | `-Xms` | Initial/minimum heap size flag (e.g., `-Xms184549376`) |
Values are specified in bytes for precision. These variables are prepended to
the container's environment so they're available for reference in `JAVA_OPTS` or
your application's JVM argument configuration.
## Configuration Guide
### Prerequisites
1. **Prometheus JVM metric endpoint**: Your Java application must expose
`jvm_memory_used_bytes` metrics with `area="heap"` and `area="nonheap"`
labels. Most JVM metric exporters (Micrometer, JMX Exporter, etc.) produce
these by default.
2. **Vertical scaling** enabled on the AIScaleTarget in `auto` (autonomous)
mode.
### Step 1: Configure Your Application to Use the Environment Variables
Your application must reference the Thoras-injected environment variables for
its JVM heap settings. The simplest approach is to reference them in `JAVA_OPTS`
or `JAVA_TOOL_OPTIONS`:
```yaml theme={null}
containers:
- name: my-jvm-app
image: my-app:latest
env:
- name: JAVA_TOOL_OPTIONS
value: "$(THORAS_JVM_HEAP_MEMORY_XMX) $(THORAS_JVM_HEAP_MEMORY_XMS)"
```
Or if your entrypoint uses `JAVA_OPTS`:
```yaml theme={null}
env:
- name: JAVA_OPTS
value:
"$(THORAS_JVM_HEAP_MEMORY_XMX) $(THORAS_JVM_HEAP_MEMORY_XMS) -XX:+UseG1GC"
```
Kubernetes will resolve the `$(...)` references at container startup using the
environment variables that Thoras injects.
### Step 2: Create the AIScaleTarget
Configure an AIScaleTarget with JVM options enabled on the target container. The
key requirements are:
* Vertical mode must be set to `auto`
* The container must have `memory.limit.ratio` set (required when JVM
auto-sizing is enabled)
* `jvm_options.enable_auto_heap_size` must be `true`
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: my-jvm-app
namespace: my-namespace
spec:
scaleTargetRef:
kind: Deployment
name: my-jvm-app
apiVersion: apps/v1
model:
mode: balanced
vertical:
mode: auto
containers:
- name: my-jvm-app
memory:
lowerbound: 128Mi
limit:
ratio: "1.0"
jvm_options:
enable_auto_heap_size: true
```
### AIScaleTarget JVM Fields Reference
All JVM configuration lives under `spec.vertical.containers[].jvm_options`:
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `enable_auto_heap_size` | bool | Yes | `false` | Enables automatic JVM heap sizing for this container. |
| `xmx_buffer` | percentage | No | `10%` | Buffer added above observed/predicted heap usage when computing `-Xmx`. Provides headroom to absorb heap growth between forecast cycles. |
| `xms_ratio` | percentage | No | `80%` | Ratio of `-Xms` to `-Xmx`. Controls how much heap the JVM pre-allocates at startup. Higher values reduce GC pressure from heap expansion. |
| `memory_buffer` | percentage | No | `10%` | Buffer added above total JVM memory (Xmx + non-heap) when computing the container memory request. Accounts for native memory, thread stacks, metaspace growth, etc. |
| `xmx_bounds.lowerbound` | resource quantity | No | - | Minimum allowed Xmx value (e.g., `256Mi`). Prevents the heap from being sized too small regardless of forecast. |
| `xmx_bounds.upperbound` | resource quantity | No | - | Maximum allowed Xmx value (e.g., `4Gi`). Caps the heap to prevent runaway growth. |
The parent container spec also requires:
| Field | Relevance |
| - | - |
| `memory.limit.ratio` | **Required** when `enable_auto_heap_size` is `true`. Defines the ratio of memory limit to request (e.g., `"1.0"` sets limit equal to request). |
| `memory.lowerbound` | Minimum container memory request. Acts as a floor. |
| `memory.upperbound` | (Optional) Maximum container memory request. Acts as a steady-state ceiling. Bypassed during opt-in [OOM remediation](/guides/oom-remediation). |
### Full Example with Custom Buffers
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: my-jvm-app
namespace: my-namespace
spec:
scaleTargetRef:
kind: Deployment
name: my-jvm-app
apiVersion: apps/v1
model:
mode: balanced
vertical:
mode: auto
containers:
- name: my-jvm-app
memory:
lowerbound: 256Mi
upperbound: 8Gi
limit:
ratio: "2.0"
jvm_options:
enable_auto_heap_size: true
xmx_buffer: "15%"
xms_ratio: "75%"
memory_buffer: "20%"
xmx_bounds:
lowerbound: 512Mi
upperbound: 4Gi
```
This configuration:
* Adds 15% headroom on heap forecasts when computing Xmx
* Sets Xms to 75% of Xmx
* Adds 20% headroom for the container memory request
* Clamps Xmx between 512 MiB and 4 GiB regardless of forecast
* Keeps the container memory request between 256 MiB and 8 GiB
### Multi-Container Pods
JVM auto-scaling supports **one JVM container per pod**. In multi-container
pods, only the container with `jvm_options.enable_auto_heap_size: true` will
have heap settings managed. Other containers (sidecars, init containers, etc.)
are unaffected:
```yaml theme={null}
vertical:
mode: auto
containers:
- name: my-jvm-app
memory:
lowerbound: 256Mi
limit:
ratio: "2.0"
jvm_options:
enable_auto_heap_size: true
- name: sidecar
memory:
lowerbound: 64Mi
cpu:
lowerbound: 50m
```
In this example, `my-jvm-app` gets Thoras-managed heap settings while `sidecar`
gets standard vertical scaling (or no scaling, depending on configuration).
## Constraints and Considerations
* **One JVM container per pod**: Only one container in the pod can have
`enable_auto_heap_size: true`.
* **Requires `memory.limit`**: The container must have a `memory.limit.ratio`
configured. This ensures the JVM has a bounded memory limit. A ratio of
`"2.0"` is typical for JVM workloads since the JVM manages its own memory
within the heap boundary.
* **Never scales below current usage**: Thoras always takes the maximum of the
forecast and current observed usage to prevent sizing below what the JVM is
actively using.
* **Metrics must be available**: The application must expose
`jvm_memory_used_bytes` with `area="heap"` and `area="nonheap"` labels. If
these metrics are missing, JVM mutation is skipped for that pod.
* **Applied at pod creation**: JVM settings are injected via the admission
webhook when pods are created. Existing running pods are not modified in place
\-- a rollout is required for new settings to take effect.
# Tuning for Large Clusters
Source: https://docs.thoras.ai/guides/large-clusters
Helm chart settings for clusters with 5,000 or more AIScaleTargets.
The chart defaults are sized for small installs. Once a cluster has around 5,000
enrolled `AIScaleTargets` (ASTs), a handful of values need to change to keep
Thoras ahead of the load.
The configuration below is sized for 5,000 ASTs and was validated on a cluster
with over 6,000. Expect Thoras to peak at roughly **100 cores** and **75 GiB of
memory**. Steady-state usage is significantly lower. The peak is a short burst
that occurs once every training interval (12h by default). Below roughly 5,000
ASTs the defaults are sufficient and none of this is required.
Requires chart version `5.5.0`
## Recommended values
Apply the following helm chart values.
```yaml theme={null}
featureFlags:
thorasManageThoras:
mode: autonomous
metricsCollectorFollowsMode: false
thorasWorker:
replicas: 2
queriesPerSecond: "100"
postgresql:
maxConns: 40
maxConnLifetime: 15m
thorasApiServerV2:
replicas: 3
postgresql:
maxConnLifetime: 15m
thorasForecast:
trainingJitterMinutes: 120
worker:
scaler:
maxReplicas: 150
metricsCollector:
persistence:
# Size at roughly 200 MiB per AST; see "Storage" below. 1Ti covers
# ~5,000 ASTs.
pvcStorageRequestSize: "1Ti"
timescale:
requests:
# TS_TUNE_NUM_CPUS below reads this value, so it must be a whole core
# count, not millicores.
cpu: "8"
memory: 30Gi
limits:
memory: 30Gi
env:
- name: TS_TUNE_NUM_CPUS
valueFrom:
resourceFieldRef:
containerName: timescaledb
resource: requests.cpu
divisor: "1"
- name: TS_TUNE_MAX_CONNS
value: "500"
```
## Applying to an existing install
The database tunes itself only when its volume is first initialized, so
deploying the new values alone will not apply the `TS_TUNE_*` settings to an
existing install. Once the new values have rolled out, rerun the tuner and
restart the database pod. Replace `thoras` with your release namespace if it
differs.
```sh theme={null}
kubectl -n thoras exec deploy/metrics-collector -c timescaledb -- \
/docker-entrypoint-initdb.d/001_timescaledb_tune.sh
kubectl -n thoras delete pod -l app=metrics-collector
```
The database is unavailable for a short time while the pod restarts. Neither
step is needed on a fresh install.
## Thoras components
Autonomous mode lets Thoras rightsize its own components as load grows, so you
do not have to hand-tune their requests.
The database is the exception. `metricsCollectorFollowsMode: false` keeps it in
observation mode, and its resources are set explicitly under
`metricsCollector.timescale` above. Changing its resources requires a restart,
which is disruptive to the database and to the Thoras platform.
## Kubernetes API load
Kubernetes API load from the worker is about 2.4 requests per AST per minute
(\~200/s at 5,000 ASTs, scaling linearly). Budget for it if you run a fixed-size,
self-managed control plane.
## Storage
At scale, plan on roughly **200 MiB per AST**, or about 1 TiB for 5,000 ASTs.
This is deliberately lower than the 1 GB-per-workload guidance on the
[Persistent Storage](/installation/persistent-storage#understanding-usage)
page. That rule is sized for small installs, where oversizing is harmless and
buys IOPS on cloud block storage. At thousands of ASTs it is excessive (it
would call for 5 TiB or more at 5,000 ASTs) and incurs real cost.
See [Persistent Storage](/installation/persistent-storage) for storage class,
EFS, and existing-volume configuration.
## Questions?
For help, email [support@thoras.ai](mailto:support@thoras.ai).
# Monitoring Thoras
Source: https://docs.thoras.ai/guides/metrics
Thoras exposes native Prometheus metrics that offer real-time visibility into the actions and health of the Thoras system. This empowers your teams with deeper insights, transparency, and observability.
## Overview
### Scraping metrics with Prometheus
Every Thoras component that emits Prometheus metrics has a `prometheus.enabled`
toggle (default `true`) and a fixed `prometheus.port`. Scrape those ports
however your Prometheus already discovers targets.
If you use the Prometheus Operator, the chart can emit a `ServiceMonitor` for
every Thoras component:
```yaml theme={null}
serviceMonitor:
enabled: true
interval: "30s" # optional; empty uses the Prometheus default
additionalLabels: # optional; useful for Prometheus discovery
release: kube-prometheus-stack
```
### Action Metrics
See Thoras in action. These metrics provide insight into how Thoras actively
manages your application resources across your environment. They track when
Thoras takes autonomous action to scale horizontally or vertically, when
replicas or resource allocations differ from Thoras' desired values, and how
each `AIScaleTarget` is currently operating. Together, these metrics give teams
the visibility they need to understand scaling behavior, make informed
decisions, and maintain efficient, well-performing applications.
| Metric Name | Type | Description | Labels |
| - | - | - | - |
| [`thoras_horizontal_scale_total`](#horizontal-scale-total) | Counter | Counts horizontal scaling actions in autonomous mode | `ai_scale_target`, `scale_metric`, `namespace` |
| [`thoras_vertical_scale_total`](#vertical-scale-total) | Counter | Counts vertical scaling adjustments in autonomous mode | `ai_scale_target`, `namespace` |
| [`thoras_recommendation`](#recommendation) | Gauge | Current scaling recommendation per `AIScaleTarget` | `ai_scale_target`, `resource`, `namespace`, `container`, `unit` |
| [`thoras_scale_targets`](#scale-targets) | Gauge | The count of `AIScaleTargets` in various operational modes (autonomous or recommendation) | `vertical`, `horizontal` |
| [`thoras_provisioning_ratio`](#provisioning-ratio) | Gauge | Ratio of current resource usage to the predicted maximum usage within the forecasted period. Values > 1 indicate current usage exceeding predicted maximum, values \< 1 indicate usage below the predicted maximum. | `ai_scale_target`, `namespace`, `scaling_mode`, `resource`, `container`, `mode` |
| [`thoras_provisioning_delta`](#provisioning-delta) | Gauge | Absolute difference between current resource usage and predicted maximum usage within the forecasted period. Units depend on resource type (bytes for memory, cores for CPU). | `ai_scale_target`, `namespace`, `scaling_mode`, `resource`, `container`, `mode` |
| [`thoras_workload_oom_killed_total`](#workload-oom-killed) | Counter | Total OOM kill events observed per workload | `ast_name`, `namespace` |
| [`thoras_ast_active_suggestion`](#ast-active-suggestion) | Gauge | 1 if the `AIScaleTarget` has an active suggestion at last check, 0 if missing. Only set for auto-mode ASTs past their startup grace period. | `ai_scale_target`, `namespace`, `uid` |
| [`thoras_ast_metric_integrity`](#ast-metric-integrity) | Gauge | 1 if metrics for the `AIScaleTarget` were collected in the last check interval, 0 if stale. Workloads at zero replicas for the entire window count as healthy. ASTs inside the creation grace period are absent. | `ai_scale_target`, `namespace`, `uid` |
| [`thoras_ast_disruption_completions_total`](#ast-disruption-completions) | Counter | Counts completed disruption cycles per `AIScaleTarget`, incremented once every pod is back in sync with the latest recommendation. Counts cycles, not pods, and marks completion rather than the start of the resize. | `ai_scale_target`, `namespace`, `reason` |
| [`thoras_disruption_circuit_breaker_open`](#disruption-circuit-breaker) | Gauge | 1 for each `AIScaleTarget` whose eviction circuit breaker is currently tripped. Absence means the breaker is closed. | `ai_scale_target`, `namespace` |
### System Health Metrics
Observe the state of Thoras' system health. These metrics offer a window into
the health and performance of the Thoras platform. They can also be easily
integrated into popular observability tools like Datadog, Grafana, giving your
team the flexibility to monitor Thoras wherever you already track system
performance.
| Metric Name | Type | Description | Labels |
| - | - | - | - |
| [`thoras_api_http_response_total`](#api-http-response-total) | Counter | Tracks total HTTP responses from internal API; spikes may indicate issues | `path`, `method`, `code` |
| [`thoras_api_http_request_duration_seconds`](#api-http-request-duration) | Gauge | Captures average internal API response time; highlights potential slowdowns | `path`, `method` |
| [`thoras_forecast_queue_pending_count`](#forecast-queue-pending-count) | Gauge | Number of pending forecast jobs currently in the queue | `action` |
| [`thoras_forecast_queue_pending_oldest_duration_seconds`](#forecast-queue-pending-oldest) | Gauge | Duration in seconds that the oldest forecast job has been pending in the queue | |
| [`thoras_forecast_queue_status_count`](#forecast-queue-status-count) | Counter | Number of forecast jobs by status | `status` |
| [`thoras_job_completions_total`](#job-completions-total) | Counter | Total number of completed jobs by status and kind | `status`, `kind` |
| [`thoras_job_duration_seconds`](#job-duration-seconds) | Gauge | Duration of jobs in seconds by status and kind | `status`, `kind` |
## Advanced Usage
### Provisioning Ratio
**Metric:** `thoras_provisioning_ratio`
The provisioning ratio compares your current resource usage to the Thoras
recommended request. The Thoras recommended request is based on the predicted
maximum usage during the forecasted period.
**Interpretation:**
Note: Recommendations are based on the predicted maximum usage over the forecast
window. Utilization ratio lowers when preemptive up-scaling occurs.
* `ratio > 1.0`: Current usage exceeding predicted maximum
* `ratio < 1.0`: Current usage below predicted maximum
**Calculation:**
* **Vertical scaling**: `current_avg_request / recommended_request`
* **Horizontal scaling**: `current_total_usage / forecasted_value`
**Example Queries & Dashboards:**
```promql theme={null}
# Find workloads with current usage more than 20% above suggested request (under predicted).
thoras_provisioning_ratio{scaling_mode="horizontal"} > 1.2
# Count workloads with current usage more than 20% above suggested request (under predicted).
count (thoras_provisioning_ratio > 1.2) # Over-provisioned
# View provisioning ratio for a specific AIScaleTarget
thoras_provisioning_ratio{ai_scale_target="cartservice"}
```
### Provisioning Delta
**Metric:** `thoras_provisioning_delta`
The provisioning delta shows the absolute difference between your current
resource usage and Thoras recommended request. The Thoras recommended request is
based on the predicted maximum usage during the forecasted period.
**Units:**
* Memory: bytes
* CPU: cores
**Calculation:**
* `abs(current_value - recommended_value)`
**Example Queries & Dashboards:**
```promql theme={null}
# Find memory deltas exceeding 1GB
thoras_provisioning_delta{resource="memory", unit="bytes"} > 1073741824
# View CPU delta for a specific workload
thoras_provisioning_delta{ai_scale_target="cartservice", resource="cpu"}
# Memory waste from over-provisioning (in GB) - combines ratio + delta
(thoras_provisioning_delta{resource="memory", unit="bytes"}
* on(ai_scale_target, namespace) group_left()
(thoras_provisioning_ratio > 1)) / 1073741824
```
### Horizontal Scale Total
**Metric:** `thoras_horizontal_scale_total`
Tracks the total number of horizontal scaling actions Thoras has performed in
autonomous mode. Each increment represents a scaling event where Thoras adjusted
replica counts.
**Example Queries & Dashboards:**
```promql theme={null}
# Total horizontal scaling actions across all targets
sum(thoras_horizontal_scale_total)
# Scaling actions per AIScaleTarget
sum by (ai_scale_target) (thoras_horizontal_scale_total)
# Scaling actions per namespace
sum by (namespace) (thoras_horizontal_scale_total)
# Rate of scaling actions over the last hour
rate(thoras_horizontal_scale_total[1h])
```
### Vertical Scale Total
**Metric:** `thoras_vertical_scale_total`
Tracks the total number of vertical scaling actions Thoras has performed in
autonomous mode. Each increment represents a scaling event where Thoras adjusted
resource requests or limits.
**Example Queries & Dashboards:**
```promql theme={null}
# Total vertical scaling actions across all targets
sum(thoras_vertical_scale_total)
# Scaling actions per AIScaleTarget
sum by (ai_scale_target) (thoras_vertical_scale_total)
# Scaling actions per namespace
sum by (namespace) (thoras_vertical_scale_total)
# Rate of scaling actions over the last hour
rate(thoras_vertical_scale_total[1h])
# Compare horizontal vs vertical scaling activity
sum(thoras_horizontal_scale_total) / sum(thoras_vertical_scale_total)
```
### Recommendation
**Metric:** `thoras_recommendation`
Shows Thoras' current scaling recommendation for each `AIScaleTarget`. The value
and unit depend on the resource type being recommended.
**Units:**
* Memory: bytes
* CPU: cores
**Example Queries & Dashboards:**
```promql theme={null}
# Current memory recommendations (in GB)
thoras_recommendation{resource="memory", unit="bytes"} / 1073741824
# Current CPU recommendations
thoras_recommendation{resource="cpu"}
# Recommendations for a specific AIScaleTarget
thoras_recommendation{ai_scale_target="cartservice"}
# Compare recommendations across containers in a workload
thoras_recommendation{ai_scale_target="frontend"} by (container, resource)
```
### Scale Targets
**Metric:** `thoras_scale_targets`
Reports the count of `AIScaleTargets` currently managed by Thoras, broken down
by operational mode (autonomous or recommendation) and scaling dimension
(vertical or horizontal).
**Example Queries & Dashboards:**
```promql theme={null}
# Total AIScaleTargets managed by Thoras
sum(thoras_scale_targets)
# Count of vertical scaling targets
thoras_scale_targets{vertical="true"}
# Count of horizontal scaling targets
thoras_scale_targets{horizontal="true"}
# Targets with both vertical and horizontal scaling enabled
thoras_scale_targets{vertical="true", horizontal="true"}
```
### Workload OOM Killed
**Metric:** `thoras_workload_oom_killed_total`
Counts the total number of OOM (Out of Memory) kill events observed per
workload. Each increment represents a container that was terminated due to
exceeding its memory limit. Use this metric to identify workloads that are
frequently running out of memory and may need higher memory recommendations.
**Example Queries & Dashboards:**
```promql theme={null}
# Total OOM kills across all workloads
sum(thoras_workload_oom_killed_total)
# OOM kills per workload
sum by (ast_name) (thoras_workload_oom_killed_total)
# OOM kills per namespace
sum by (namespace) (thoras_workload_oom_killed_total)
# Rate of OOM kills over the last hour
rate(thoras_workload_oom_killed_total[1h])
# Workloads with the most OOM kills
topk(10, sum by (ast_name, namespace) (thoras_workload_oom_killed_total))
# Alert on OOM kills in the last 15 minutes
increase(thoras_workload_oom_killed_total[15m]) > 0
```
### AST Active Suggestion
**Metric:** `thoras_ast_active_suggestion`
Indicates whether each auto-mode `AIScaleTarget` has an active suggestion at the
time of the last periodic check. The value is `1` when a suggestion exists and
`0` when no suggestion is present. ASTs still within their startup grace period
are excluded.
**Example Queries & Dashboards:**
```promql theme={null}
# ASTs currently missing a suggestion
thoras_ast_active_suggestion == 0
# Count of ASTs missing a suggestion
count(thoras_ast_active_suggestion == 0)
# Alert when any AST has been missing a suggestion for 15m
max_over_time(thoras_ast_active_suggestion[15m]) == 0
```
### AST Metric Integrity
**Metric:** `thoras_ast_metric_integrity`
Note: Requires `thorasWorker.enableMetricIntegrityWorker: true` in `values.yaml`
(default `true`).
Reports whether Thoras has successfully collected CPU/memory metrics for each
`AIScaleTarget` in the most recent check interval. The value is `1` when the AST
is healthy and `0` when metrics are stale (the workload was running but no
CPU/memory samples were collected). Workloads that sat at zero replicas for the
entire window count as healthy, since no metrics are expected. ASTs still within
their creation grace period, as well as orphaned ASTs whose scale target no
longer exists, are absent from the metric entirely.
Use this metric to detect a wedged collector or broken metrics pipeline for a
specific workload.
**Example Queries & Dashboards:**
```promql theme={null}
# ASTs with stale metrics
thoras_ast_metric_integrity == 0
# Count of ASTs with stale metrics
count(thoras_ast_metric_integrity == 0)
# Alert when any AST has been stale for 60m
max_over_time(thoras_ast_metric_integrity[60m]) == 0
```
### AST Disruption Completions
**Metric:** `thoras_ast_disruption_completions_total`
Counts completed disruption cycles per `AIScaleTarget`. When Thoras applies a
new vertical recommendation, it resizes or replaces the workload's pods in
batches; once every pod is back in sync with the recommendation, the cycle
completes and this counter increments.
Use this metric to correlate Thoras resize activity against your own application
signals — error rates, latency, or restart counts — without having to infer
timing from resource requests changing on the workload.
Two things to know about what it counts:
* **It counts cycles, not pods.** A cycle that resized four pods increments by
one, the same as a cycle that resized one. Cycles resolved entirely by
[in-place resize](/guides/vertical-pod-rightsizing) — with no pod restarts at
all — are counted too, as are cycles that end early because the update policy
changed mid-flight.
* **It marks completion, not the start.** Thoras records the cycle as complete
on the check that observes every pod back in sync, so the timestamp trails the
actual pod changes, typically by up to one disruption interval. When lining
this up against an error-rate spike, expect the spike to precede the
increment.
A series appears for an `AIScaleTarget` only after its first completed cycle, so
absence means no cycle has completed yet rather than zero.
**Example Queries & Dashboards:**
```promql theme={null}
# Disruption cycles completed per workload over the last hour
sum by (ai_scale_target, namespace) (increase(thoras_ast_disruption_completions_total[1h]))
# Overlay marker: did a cycle complete for this workload in the last 5m?
increase(thoras_ast_disruption_completions_total{ai_scale_target="my-workload"}[5m]) > 0
# Busiest workloads by disruption activity
topk(10, sum by (ai_scale_target, namespace) (increase(thoras_ast_disruption_completions_total[24h])))
# Cluster-wide disruption rate
sum(rate(thoras_ast_disruption_completions_total[30m]))
```
### API HTTP Response Total
**Metric:** `thoras_api_http_response_total`
Tracks the total number of HTTP responses from Thoras' internal API. This
counter increments for every API response and can help identify unusual traffic
patterns or issues.
**Example Queries & Dashboards:**
```promql theme={null}
# Total API responses
sum(thoras_api_http_response_total)
# Response rate over the last 5 minutes
rate(thoras_api_http_response_total[5m])
# Responses by status code
sum by (code) (thoras_api_http_response_total)
# Error rate (4xx and 5xx responses)
sum(rate(thoras_api_http_response_total{code=~"4..|5.."}[5m]))
# Requests by endpoint
sum by (path, method) (thoras_api_http_response_total)
# Success rate percentage
(sum(rate(thoras_api_http_response_total{code=~"2.."}[5m]))
/ sum(rate(thoras_api_http_response_total[5m]))) * 100
```
### API HTTP Request Duration
**Metric:** `thoras_api_http_request_duration_seconds`
Measures the average response time for Thoras' internal API requests in seconds.
Rising values may indicate performance degradation or capacity issues.
**Example Queries & Dashboards:**
```promql theme={null}
# Response time by endpoint
avg by (path) (thoras_api_http_request_duration_seconds)
# Average response time over the last hour
avg_over_time(thoras_api_http_request_duration_seconds[1h])
```
### Forecast Queue Pending Count
**Metric:** `thoras_forecast_queue_pending_count`
**Labels:** `action` (`forecast` or `train`)
Shows the current number of pending jobs waiting in the queue, broken down by
action type. High values indicate that requests are backing up, which may signal
worker capacity issues or processing bottlenecks.
**Example Queries & Dashboards:**
```promql theme={null}
# Current pending count by action
thoras_forecast_queue_pending_count
# Pending forecast jobs only
thoras_forecast_queue_pending_count{action="forecast"}
# Pending train jobs only
thoras_forecast_queue_pending_count{action="train"}
# Total pending jobs across all actions
sum(thoras_forecast_queue_pending_count)
# Maximum pending count over the last hour
max_over_time(thoras_forecast_queue_pending_count[1h])
# Alert on high queue depth (> 10 jobs total)
sum(thoras_forecast_queue_pending_count) > 10
```
### Forecast Queue Pending Oldest
**Metric:** `thoras_forecast_queue_pending_oldest_duration_seconds`
Tracks how long the oldest forecast job has been waiting in the queue. Rising
values indicate jobs are not being processed quickly enough.
**Example Queries & Dashboards:**
```promql theme={null}
# Current oldest job wait time
thoras_forecast_queue_pending_oldest_duration_seconds
# Maximum wait time over the last day
max_over_time(thoras_forecast_queue_pending_oldest_duration_seconds[24h])
# Average wait time over the last day
avg_over_time(thoras_forecast_queue_pending_oldest_duration_seconds[24h])
```
### Forecast Queue Status Count
**Metric:** `thoras_forecast_queue_status_count`
Tracks the total number of forecast jobs processed, categorized by status. This
counter helps monitor forecast job lifecycle and identify issues with job
processing.
**Example Queries & Dashboards:**
```promql theme={null}
# Total forecast jobs by status
sum by (status) (thoras_forecast_queue_status_count)
# Rate of forecast jobs by status over the last day
rate(thoras_forecast_queue_status_count[24h])
# Failed forecast jobs
sum(thoras_forecast_queue_status_count{status="failed"})
# Success rate percentage
(sum(rate(thoras_forecast_queue_status_count{status="completed"}[24h]))
/ sum(rate(thoras_forecast_queue_status_count[24h]))) * 100
# Total forecast jobs processed
sum(thoras_forecast_queue_status_count)
```
### Job Completions Total
**Metric:** `thoras_job_completions_total`
Tracks the total number of completed jobs, categorized by status (success,
failure, etc.) and job kind. This counter helps identify job failure rates and
patterns across different job types.
**Example Queries & Dashboards:**
```promql theme={null}
# Total job completions
sum(thoras_job_completions_total)
# Job completions by status
sum by (status) (thoras_job_completions_total)
# Job completions by kind
sum by (kind) (thoras_job_completions_total)
# Failed jobs only
sum(thoras_job_completions_total{status="failure"})
# Job completion rate over the last hour
rate(thoras_job_completions_total[1h])
# Failure rate percentage
(sum(rate(thoras_job_completions_total{status="failure"}[24h]))
/ sum(rate(thoras_job_completions_total[24h]))) * 100
```
### Job Duration Seconds
**Metric:** `thoras_job_duration_seconds`
Measures the duration of completed jobs in seconds, broken down by status and
job kind. Use this to identify slow jobs or performance regressions.
**Example Queries & Dashboards:**
```promql theme={null}
# Average duration by job kind
avg by (kind) (thoras_job_duration_seconds)
# Jobs taking longer than 30 seconds
thoras_job_duration_seconds > 30
```
## Grafana Dashboard
We provide a pre-built Grafana dashboard that visualizes Thoras metrics out of
the box. It includes panels for scaling activity, provisioning efficiency,
forecast pipeline health, API performance, and worker job status.
![Thoras Grafana Dashboard]()
![Thoras Grafana Dashboard]()
### Importing the Dashboard
1. Download the
[Thoras Grafana Dashboard](./static/thoras-grafana-dashboard.json)
2. In Grafana, go to **Dashboards > New > Import**
3. Click **Upload dashboard JSON file** and select the downloaded file
4. When prompted, select your Prometheus datasource that scrapes Thoras metrics
5. Click **Import**
### Prerequisites
* Thoras must be installed and exposing its Prometheus metrics endpoint
* A Prometheus instance must be scraping Thoras metrics
* Grafana must have that Prometheus instance configured as a datasource
# OOM Remediation
Source: https://docs.thoras.ai/guides/oom-remediation
Automatically recover workloads from out-of-memory kill loops
When a container is killed due to an out-of-memory (OOM) event, Thoras can
automatically increase the container's memory request and limit to stabilize the
workload. This prevents a workload from cycling through repeated OOM kills while
waiting for a scheduled forecast to produce a sufficient memory recommendation.
OOM remediation requires
[autonomous vertical scaling](/guides/vertical-pod-rightsizing) to be enabled.
It is opt-in per workload.
## How It Works
When OOM kills are detected on a workload, Thoras:
1. **Bumps memory requests and limits** for affected containers by a 1.2x
multiplier, computed independently from the average current memory request
and limit across all running pods. Both request and limit are raised since
the kernel OOM-kills based on the limit, and request guarantees node memory.
2. **Applies the adjustment** to running pods and any newly created pods. For
running pods, Thoras uses your
[`update_policy`](/reference/ast-definition#update_policy): it attempts an
in-place resize first (if enabled). When that does not apply and your policy
allows recreating pods, Thoras replaces the remaining pods gradually, in
batches paced by your [`disruption`](/reference/ast-definition#disruption)
settings, rather than restarting the whole workload at once. See
[Rolling Out the Adjustment](#rolling-out-the-adjustment). Any pods created
or rescheduled in the meantime (e.g., evictions, scale-out) automatically
receive the adjusted memory.
3. **Repeats as long as OOM kills continue**, compounding each time (1.2×, then
1.44×, then 1.73×, and so on) until OOMs stop. Thoras waits at least 2
minutes between adjustments, and when the adjustment is rolled out in batches
it also waits until every pod has the current adjustment before applying the
next one.
4. **Holds the memory floor** for a configurable stabilization window after the
last OOM. During this window, forecasts can raise memory above the floor but
cannot lower it. This prevents a new forecast, which may not yet reflect the
OOM episode, from reversing the adjustment down to a point which would be
prone to OOMing.
5. **Returns full control to the forecaster** once the stabilization window
expires. If OOMs recur, the cycle begins again.
The adjustment is a short-term stabilization measure, not a permanent
right-sizing decision. The goal is to keep the workload running long enough for
the forecaster to observe the higher memory usage and incorporate it into future
recommendations.
## Rolling Out the Adjustment
New pods pick up the adjusted memory as soon as it is set. For pods that are
already running, what Thoras does depends on your
[`update_policy`](/reference/ast-definition#update_policy):
| `update_mode` | In-place resize applies | In-place resize does not apply |
| - | - | - |
| `in_place_or_recreate` | Pods are resized | Remaining pods are replaced gradually in [`disruption`](/reference/ast-definition#disruption) batches |
| `recreate` (or no `update_mode`) | Not attempted | Pods are replaced gradually in [`disruption`](/reference/ast-definition#disruption) batches |
| `in_place` | Pods are resized | Pods are never recreated; Thoras retries the resize in [`disruption`](/reference/ast-definition#disruption) batches until it succeeds or the stabilization window ends |
In-place resize "does not apply" when the resize fails (for example, the node
lacks capacity), is not supported by the cluster, or includes a change the
cluster can only make by restarting the pod (such as lowering a memory limit).
Gradual replacement needs `memory` in
[`recreate_resources`](/reference/ast-definition#update_policy), which is the
default. If `recreate_resources` leaves out `memory` (for example `["cpu"]`),
Thoras falls back to a rolling restart of the workload instead, when your
`update_mode` allows recreating pods.
When pods are replaced gradually:
* **Batches follow your `disruption` settings.** Thoras evicts at most
`max_disrupted` pods at a time and waits `interval` between batches. Evictions
go through the Kubernetes eviction API, so your PodDisruptionBudgets are
honored. Thoras does not modify your Deployment or StatefulSet, so GitOps
tools see no drift.
* **Crash-looping pods are replaced first.** A pod that is already down because
it keeps OOMing goes in the next batch even when `max_disrupted` is used up,
so it cannot stall the rollout.
* **No forecast is needed.** The rollout proceeds even when the workload has no
usable forecast yet, such as a new workload still collecting metrics.
* **Only pods below the adjusted memory are replaced.** Your scale-up threshold
does not apply here, and a pod that already has at least the adjusted memory
is never replaced for the OOM adjustment.
* **One adjustment at a time.** Pods still waiting for their turn may OOM again
at the old memory. Those OOMs do not trigger another adjustment until every
pod has the current one, so a single episode does not compound on top of
itself while the rollout is still in progress.
During a gradual replacement, you can follow progress with the same signals as
any other paced rollout: the `DisruptionInProgress` condition on the
AIScaleTarget, which reads `Complete` once every pod has the adjusted memory,
and the `thoras_ast_disruption_completions_total` metric. See
[In-Place Resize Behavior](/guides/in-place-resize) for the full list.
## Enabling OOM Remediation
Add `oom_remediation` to your `spec.vertical` configuration:
```yaml theme={null}
spec:
vertical:
mode: autonomous
oom_remediation:
enabled: true
containers:
- name: my-container
memory:
lowerbound: 256Mi
upperbound: 4Gi
```
`oom_remediation.enabled` must be `true` and `spec.vertical.mode` must be
`autonomous`. Workloads in `recommendation` mode are not affected.
## Stabilization Window
After Thoras applies an OOM memory adjustment, it holds a **memory floor** for
the duration of the stabilization window. During this window:
* Incoming forecasts **cannot lower** memory below the adjusted value.
* Forecasts **can raise** memory above the adjusted value if the forecast
recommends it.
* Each new OOM resets the stabilization window, extending it from the time of
the most recent OOM. The workload must be stable for the configured interval
before memory may be scaled down from the multiplied value.
Once the window expires, Thoras clears the floor and the forecaster resumes full
control of memory sizing.
**Default duration:** `max(forecast_interval, 1h)`. The window is at minimum one
full forecast cycle, ensuring the forecaster has had at least one opportunity to
observe the workload at the adjusted memory level before the floor is removed.
The stabilization window should be long enough for the forecaster to observe
memory usage at the adjusted level, so future recommendations account for the
higher usage that triggered the OOMs.
To override the default, set `stabilization_window` explicitly:
```yaml theme={null}
oom_remediation:
enabled: true
stabilization_window: 2h
```
## Upper Bounds
`memory.upperbound` is a **steady-state** policy and is intentionally bypassed
during OOM remediation. OOM is an emergency response: clamping the adjustment to
`upperbound` would leave the workload exposed to the same kill threshold that
triggered the cycle in the first place. Adjustments may therefore land above the
configured upper bound while the stabilization window is active.
Once the stabilization window expires, the forecaster regains full control and
steady-state recommendations are clamped to `upperbound` as usual.
## Memory Limits
OOM remediation raises both the memory request and the memory limit. How the
limit is determined depends on whether Thoras manages it:
* **Limit managed by Thoras** (`memory.limit.ratio` is set): during the
stabilization window, the limit is
`max(adjusted_request × ratio, adjusted_limit)`. The ratio is preserved on the
way up, and the OOM-adjusted limit acts as a floor so a non-OOM reconcile
cannot snap the limit back below it mid-episode. After the window expires, the
limit follows the ratio normally as the forecaster regains control.
* **Limit not managed by Thoras** (no `memory.limit.ratio`): the limit is raised
by the same multiplier as the request during the stabilization window to give
the workload room to recover. When the window expires, Thoras reverts the
limit to the value in your workload's pod template (deployment, statefulset,
etc.). If your workload's true steady-state memory usage permanently exceeds
the limit set in your pod template, OOMs will recur each time the limit
reverts, visible as repeated `OOMKilled` events. The resolution is to raise
the limit in the workload's pod template, or to configure `memory.limit.ratio`
so Thoras can manage limits directly.
## Interaction with Forecasts
During an active stabilization window, Thoras applies `max(forecast, floor)` to
memory when scaling. CPU is always sourced from the forecast unchanged.
When a new forecast arrives after the window has expired, the floor is cleared
and the forecast value is applied directly. If the workload OOMs again, the
remediation cycle restarts from the new forecast baseline.
## Considerations
**Memory may be held above the forecast during the stabilization window.** If an
OOM was caused by a one-off spike and the workload returns to lower memory usage
afterward, requests will remain elevated until the window expires. Thoras favors
avoiding another OOM over reclaiming memory immediately, and allows configuring
the duration of the stabilization window.
**New pods after the window expires start at the forecast value.** Once the
floor clears, pods created from that point forward use the forecaster's
recommendation. If that recommendation is insufficient, another OOM may occur
and trigger a new remediation cycle.
**Partial in-place resize failures.** If some pods cannot be resized in place
(e.g., the node lacks capacity), Thoras replaces those pods gradually when your
`update_policy` allows recreating them. With `update_mode: in_place`, they keep
their previous memory until a later resize succeeds.
**Recovery is paced by your `disruption` settings.** Replacing every pod takes
roughly `replicas ÷ max_disrupted` batches, `interval` apart, and the next
adjustment waits until the current one has reached every pod. A pod that a
PodDisruptionBudget keeps from being evicted delays the rollout, and the next
adjustment, until it can be replaced. For workloads that need to recover faster,
raise `max_disrupted` or shorten `interval`.
## Full Example
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: my-service
namespace: my-namespace
spec:
scaleTargetRef:
kind: Deployment
name: my-service
apiVersion: apps/v1
model:
mode: balanced
forecast_interval: 1h
forecast_blocks: 4h
vertical:
mode: autonomous
oom_remediation:
enabled: true
stabilization_window: 2h
update_policy:
update_mode: in_place_or_recreate
containers:
- name: my-service
memory:
lowerbound: 256Mi
upperbound: 4Gi
cpu:
lowerbound: 100m
upperbound: 2
```
In this configuration:
* OOM remediation is enabled with a 2-hour stabilization window
* Steady-state memory recommendations are clamped to 4 GiB (`upperbound`); OOM
adjustments may exceed this temporarily during stabilization
* Pods are resized in place where possible; otherwise they are replaced
gradually in [`disruption`](/reference/ast-definition#disruption) batches
# Pausing Autonomous Scaling
Source: https://docs.thoras.ai/guides/pause-scaling
Temporarily halt all autonomous scaling actions cluster-wide
The scaling pause is a system-wide toggle that halts all autonomous scaling
actions across the entire cluster. Thoras continues collecting metrics,
generating predictions, and producing recommendations. No data or model state is
lost while paused.
Only [Autonomous mode](/guides/enabling-autonomous-mode) targets are affected.
Workloads in `recommendation` mode are unaffected.
## Example Use Cases
* **Planned maintenance**: prevent Thoras from modifying resource requests
during a maintenance window
* **Incident response**: freeze current allocations while investigating a
production issue
* **Baseline validation**: observe workload behavior without scaling, such as
during load testing
## Pausing and Resuming
Open the **Manage Cluster** dropdown in the dashboard header and select **Pause
autonomous scaling**. A flyout displays the count of autonomous targets that
will be affected and asks for confirmation before applying.
To resume, open the same dropdown, selects **Resume autonomous scaling**, then
confirm.
The pause UI is gated behind `thorasDashboard.extras.enable_scaling_pause` in
the Helm chart (default `true`). If you've turned it off, use the manual
ConfigMap toggle described under [Advanced](#advanced).
## Pod Behavior
**When you pause scaling**, workloads' running pods retain the resource requests
last applied by Thoras. If a pod restarts while scaling is paused, it starts
with the requests defined in the deployment spec, and Thoras will not apply any
adjustments until scaling is resumed.
**When you resume scaling** (un-pause), the Thoras operator restarts then
reconciles autonomous targets, setting requests to the latest suggestion for
pods which need resizing.
## Visual Indicators
### Banner
When paused, a persistent amber banner appears at the top of the dashboard:
> Autonomous scaling actions are paused. Targets will not be right-sized.
The banner links directly to the resume flyout so you can re-enable scaling
without navigating away.
### Scaling History
A target's scaling and suggestions timeline indicates which suggestions were
made while the system was in the scaling paused state, and which, if any,
actions would have been applied had scaling not been paused.
## Persistence
The pause state is stored in the `thoras-operator-system-config` ConfigMap in
the Thoras namespace. If the ConfigMap does not exist, the system defaults to
**not paused**.
Toggling the state triggers a rolling restart of both the operator and API
server to load the new configuration. There is a brief period of unavailability
for the API server and operator as they restart.
Because the state lives in a ConfigMap, **it survives pod restarts and
upgrades**. The cluster remains paused until explicitly resumed.
## Advanced
As an alternative to using the dashboard, scaling pause state can be toggled
manually by modifying the ConfigMap `thoras-operator-system-config` then
restarting `thoras-api-server` and `thoras-operator`.
# Stopping Autonomous Scaling
Source: https://docs.thoras.ai/guides/stopping-autoscaling
Options for halting Thoras autoscaling, from cluster-wide to per-workload
There are three ways to stop Thoras from autonomously scaling workloads, ordered
from broadest to narrowest scope. Choose the option that matches the blast
radius you need.
| Approach | Scope | Restarts Pods | Workload requests | Model & data retention |
| - | - | - | - | - |
| System-wide pause | All autonomous ASTs | No | Running pods keep the last applied suggestion; restarted pods come up with the controller spec | Thoras keeps forecasting and producing suggestions |
| `recommendation` mode | One AST | Yes | Reverts to the controller spec | Thoras keeps forecasting and producing suggestions |
| Unenroll (delete AST) | One AST | Yes | Reverts to the controller spec | The workload is no longer managed or observed by Thoras. A recreated AST starts with empty history unless it [inherits it](/reference/ast-definition#keeping-history-when-an-ast-is-recreated) |
## 1. System-wide pause
Halts all autonomous scaling actions across the entire cluster. Thoras continues
collecting metrics and producing recommendations; only the apply step is
suspended. Workloads in `recommendation` mode are unaffected.
Pausing does **not** restart workload pods. Running pods keep the resource
requests Thoras last applied to them. When new pods are created they will use
the requests defined in the controller spec.
**When to use:** planned maintenance windows, incident response, or baseline
validation where you want to freeze allocations cluster-wide without changing
any `AIScaleTarget` definitions.
**How to do it:**
1. Open the **Manage Cluster** dropdown in the dashboard header.
2. Select **Pause autonomous scaling**.
3. Confirm in the flyout, which shows the count of autonomous targets that will
be affected.
To resume, open the same dropdown and select **Resume autonomous scaling**.
The pause state persists across pod restarts and upgrades via the
`thoras-operator-system-config` ConfigMap. See
[Pausing Autonomous Scaling](/guides/pause-scaling) for pod behavior during
pause, visual indicators, and the ConfigMap-based advanced workflow.
## 2. Switch the AST from autonomous to recommendation mode
Stops autonomous scaling for a single workload while keeping the `AIScaleTarget`
enrolled. Thoras continues to forecast and surface suggestions in the dashboard,
but no scaling actions are applied.
Unlike a system-wide pause, switching to `recommendation` mode restarts the
workload's pods and reverts it to the requests defined in the controller spec.
**When to use:** you want to keep the workload monitored and the model warm (for
example, to compare suggestions against actual usage), but you don't want Thoras
applying changes for that specific workload.
**How to do it:**
Edit the `AIScaleTarget` and set the active scaling direction's `mode` to
`recommendation`:
```sh theme={null}
kubectl edit ast {{YOUR_AST_NAME}} -n {{YOUR_NAMESPACE}}
```
For a vertically autonomous workload:
```yaml theme={null}
spec:
vertical:
mode: recommendation
```
For a horizontally autonomous workload:
```yaml theme={null}
spec:
horizontal:
mode: recommendation
```
Recommendations continue to appear in the dashboard. To re-enable autonomous
scaling later, set `mode` back to `autonomous`. See
[Understanding Vertical and Horizontal Scaling Modes](/guides/configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
for how the two directions interact, and the
[AIScaleTarget reference](/reference/ast-definition) for the full mode
specification.
## 3. Unenroll the workload (delete the AST)
Removes the workload from Thoras entirely. The workload is restarted and its
pods come up with the resource requests defined in the controller spec.
**Deleting an AST is permanent.** Its model state is gone, and Thoras stops
using its suggestions and utilization data, which age out under normal
retention. Re-creating an AST with the same name later produces a new AST with
empty history, unless you create it with the `thoras.ai/inherit-history`
annotation to [keep the retained
history](/reference/ast-definition#keeping-history-when-an-ast-is-recreated).
If you want to stop scaling actions while keeping history intact, use option 1
or 2 above.
**When to use:** the workload should no longer be managed or observed by Thoras
at all. For example, when the workload is being decommissioned, moved to a
different scaling system, or you want a clean re-enrollment later.
**How to do it:**
```sh theme={null}
kubectl delete ast {{YOUR_AST_NAME}} -n {{YOUR_NAMESPACE}}
```
If the AIScaleTarget is managed by a `ClusterAIScaleTemplate` (it has the
`thoras.ai/managed-by-template` label), deleting it isn't enough — the
template will recreate it on the next reconcile. Annotate the workload with
`thoras.ai/skip-template: "true"` first, then delete the AIScaleTarget. See
[Opting a Workload
Out](/reference/cluster-aiscale-template#opting-a-workload-out) for details.
# Predictive Vertical Pod Rightsizing
Source: https://docs.thoras.ai/guides/vertical-pod-rightsizing
Predictive Vertical Pod Rightsizing is the process of preemptively optimizing the CPU and memory resource requests and limits for Kubernetes pods to ensure they have just enough resources to run efficiently without over or under provisioning. Unlike horizontal scaling, which adjusts the number of pod replicas, vertical rightsizing focuses on tuning individual pod configurations to better match their actual usage patterns. This helps reduce wasted resources, lower cloud costs, and improve cluster utilization, while also minimizing the risk of resource starvation or out-of-memory (OOM) errors.
**In-place pod resize on Kubernetes 1.33+.** When running on Kubernetes 1.33
or later, Thoras applies CPU and memory adjustments without restarting pods,
eliminating the rollout churn historically required by vertical autoscalers.
Resizes are still rolled out in paced batches across the workload rather than
all at once. Thoras falls back to a rolling restart automatically on older
clusters or when an in-place resize is not supported (for example, a QoS class
change). See [Update Policy](#update-policy) for details.
## Prerequisites
Before you begin, make sure:
* Thoras is installed and running in your Kubernetes cluster.
* The
[Kubernetes Metrics Server](https://github.com/kubernetes-sigs/metrics-server)
is installed in your cluster.
* Your workloads are instrumented with resource requests and have some usage
history.
## Sample Configuration
Here's an example `AIScaleTarget` that configures rightsizing.
```yaml ast.yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: {{YOUR_AST_NAME}}
namespace: {{YOUR_NAMESPACE}}
spec:
scaleTargetRef:
kind: Deployment
name: {{YOUR_DEPLOYMENT_NAME}}
model:
forecast_blocks: 4h
forecast_interval: 1h
forecast_buffer_percentage: 0%
mode: balanced
vertical:
mode: recommendation
containers:
- name: {{YOUR_CONTAINER_NAME}}
cpu:
lowerbound: 20m
upperbound: 1
memory:
lowerbound: 50Mi
limit:
ratio: 1.5
update_policy:
update_mode: in_place_or_recreate
```
## Vertical Breakdown
* `spec.vertical.mode` set to `recommendation` ensures Thoras suggests optimal
requests and limits but does not apply them.
* `spec.vertical.containers[].name` should be set to the name of the container
in the pods for the workload you are targeting with this `AIScaleTarget`. This
field supports wildcard patterns (e.g., `app-*`, `*-sidecar`) to match
multiple containers. See
[Container Name Wildcard Matching](/reference/ast-definition#container-name-wildcard-matching)
for details.
* `spec.vertical.containers[].RESOURCE.lowerbound` is the lowest Thoras is
allowed to set the request for that resource and is a *required* field.
* `spec.vertical.containers[].RESOURCE.upperbound` is the highest Thoras is
allowed to set the request for that resource and is an optional field.
* In `autonomous` mode, if you change `lowerbound` or `upperbound` and live pod
requests fall outside the new range, Thoras applies the updated bounds
immediately according to [`update_policy`](#update-policy) (in-place resize
and/or recreate). With `update_mode: initial`, bounds still apply only when
pods are next created.
* `spec.vertical.containers[].RESOURCE.limit` allows thoras to modify the
container's limit along with the request for this resource and is optional. If
unset, Thoras will *not* modify limits for this resource.
* `spec.vertical.containers[].RESOURCE.limit.ratio` the ratio used to keep the
limit inline with the suggested request for this resource. E.g. for a ratio
`2.5` Thoras will make the limit `2.5G` if the suggestion for the request was
for `1G`.
## How Limits Work With Predictive Right-Sizing
When Thoras right-sizes your pods, it respects existing resource limits to
ensure safe operation. Here's how limits interact with Thoras recommendations:
### Limit Enforcement Rules
Thoras enforces the following maximum constraints when right-sizing:
* **AIScaleTarget `upperbound`**: If you define an `upperbound` in your
AIScaleTarget, Thoras will never set requests to above that value during
steady-state operation.
* **Pod spec `limit`**: If a `limit` exists on the pod spec, Thoras will never
right-size a pod above that limit during steady-state operation.
When both a pod `limit` and an AIScaleTarget `upperbound` are set, Thoras uses
the lower of the two as the effective maximum value for right-sizing.
`upperbound` is a steady-state constraint. The one exception is
[OOM remediation](/guides/oom-remediation), an opt-in emergency response that
intentionally bypasses `memory.upperbound` so it can lift a workload out of an
out-of-memory kill loop. Steady-state recommendations are clamped to
`upperbound` as usual once the remediation's stabilization window expires.
Changing `lowerbound` or `upperbound` on an autonomous AIScaleTarget also
re-enforces those bounds on running pods that are outside the new range. See
[Vertical Breakdown](#vertical-breakdown) and [`update_policy`](#update-policy).
### Scaling Limits Proportionally
You can configure Thoras to automatically scale your resource limits along with
your requests using the `limit.ratio` setting. This keeps limits proportional to
requests as Thoras adjusts resource allocations.
For example, setting `.spec.vertical.containers[].RESOURCE.limit.ratio: 1.5`
means that if Thoras recommends a request of `1G`, the limit will be set to
`1.5G`.
See the
[AIScaleTarget reference documentation](/reference/ast-definition#vertical) for
more details on configuring the `limit.ratio` field.
## Update Policy
The `update_policy` field controls pod update behavior during vertical scaling
operations (such as restarts).
**Configuration:**
* `update_mode` - Determines how pod updates are applied:
* `recreate` - Pods are recreated when resource changes are applied
* `in_place` - Changes applied without pod recreation (requires Kubernetes
1.33+)
* `in_place_or_recreate` - Attempts in-place updates with fallback to
recreation (recommended)
* `initial` - Recommendations apply only during pod creation
* `recreate_resources` (optional; defaults to `memory` only) - Array specifying
which resources (`memory`, `cpu`) trigger pod recreation when using recreate
or fallback modes.
**Example configurations:**
For in-place updates with fallback to recreation:
```yaml theme={null}
update_policy:
update_mode: in_place_or_recreate
```
For in-place-only updating, no fallback to recreation:
```yaml theme={null}
update_policy:
update_mode: in_place
recreate_resources: ["memory"]
```
For in-place updates with fallback to recreation if either `memory` or `cpu`
require right sizing:
```yaml theme={null}
update_policy:
update_mode: in_place_or_recreate
recreate_resources: ["memory", "cpu"]
```
> **Note:** Enabling in-place resizing does not guarantee that pods will never
> restart during resizing operations. Pods may still be evicted and restarted if
> the resize would change the pod's
> [QoS class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) or if
> the cluster does not support the specific resize operation.
**How it works:**
1. When `update_mode` is `in_place` or `in_place_or_recreate`, Thoras attempts
to resize pods in place using the Kubernetes resize API. Resizes are rolled
out in paced batches across the workload rather than all at once, governed by
[`disruption`](/reference/ast-definition#disruption). If the Kubernetes API
rate-limits these requests, Thoras backs off and resumes the remaining pods
on a later run
2. When `update_mode` is `in_place_or_recreate` and the pods
[QoS class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
would change or resize is not supported, Thoras falls back to recreating the
pod
3. Evicted pods are also rolled out gradually to minimize service disruption
For details on what causes a resize to stall and how each mode responds, see
[In-Place Resize Behavior](/guides/in-place-resize).
**Benefits:**
* **Reduced disruption**: Pods continue running during resource adjustments
* **Faster scaling**: Changes take effect immediately without pod recreation
* **Better node utilization**: Enables more efficient bin-packing and cost
savings, especially when combined with node autoscaling tools like Karpenter
* **Graceful fallback**: Automatic eviction when in-place resizing isn't
possible
* **Fine-grained control**: Specify which resources trigger pod recreation
**Requirements:**
* Kubernetes 1.33+
## Additional Details
* Thoras recommendations are based on historical usage data collected from the
Kubernetes Metrics Server.
* Recommendations are updated on the interval defined by
`spec.model.forecast_interval`.
* If resource usage patterns change significantly, Thoras will adjust its
suggestions accordingly.
* You can monitor the impact of rightsizing by reviewing resource utilization
and cost metrics in the Thoras Dashboard.
* For workloads with multiple containers, you can specify vertical settings for
each container individually under `spec.vertical.containers`.
* `spec.vertical.containers[].RESOURCE` can be either `cpu` or `memory`. If
either is unset for a container, Thoras will not scale that resource for that
container.
## Viewing Recommendations
Once the `AIScaleTarget` is applied, The Thoras Reasoning Engine will begin
analyzing resource usage and generate vertical sizing recommendations on the
interval defined by `spec.model.forecast_interval`.
You can view the suggested requests and their projected effects on your workload
in the Thoras dashboard under Resource Allocation Settings.
![resource-allocation]()
## Next Steps
* Change `spec.vertical.mode` to `autonomous` if you want Thoras to
automatically apply recommendations.
* Add optional `scaling_behavior` if you want to fine-tune the thresholds Thoras
uses to determine if a recommendation is worth restarting the entire workload.
```yaml ast.yaml theme={null}
...
vertical:
mode: autonomous
# Only restart workload if a new resource recommendation is at least 50% greater or lower than the current pod resource allocation
scaling_behavior:
scale_up:
type: percent
percent: 50
scale_down:
type: percent
percent: 50
...
```
## Autonomous Mode
The Thoras Reasoning Engine will make usage predictions on the interval set by
`spec.model.forecast_interval` for the length of `spec.model.forecast_blocks`.
If no `spec.vertical.scaling_behavior` is set then Thoras will check all
resource requests to determine if they match the current suggestion. If any
resource request differs from the current suggestion, Thoras triggers a rollout
restart of the workload. As the pods restart, their resource requests and limits
will be updated to match the suggestion made by Thoras.
> **Note:** Thoras mutates the resource requests and limits of running pods
> directly. It does **not** modify the Deployment or StatefulSet object itself.
> **Note:** Only one scaling direction (horizontal or vertical) can be in
> `autonomous` mode at a time. See
> [Configuring AIScaleTargets](./configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
> for additional details.
## Identifying Thoras-Managed Pods
When the admission webhook sets a pod's resources in `autonomous` mode, it
annotates the pod with `thoras.ai/managed: "true"`, `thoras.ai/scaler`, and
`thoras.ai/admitted-at`. This lets external tooling identify pods Thoras has
mutated without diffing against the workload's controller spec. See
[Pod Annotations](/reference/pod-annotations) for the full reference, including
limitations around pods that predate autonomous mode and pods that have since
been reverted to template values.
# Advanced Setup
Source: https://docs.thoras.ai/installation/advanced-setup
This page collects the Helm chart knobs you're most likely to touch beyond a
default install. For a quicker, higher-level guide to install and try out
Thoras, check out the [QuickStart Guide](../getting-started/quickstart).
Several install-time topics live on their own pages:
* [Persistent Storage](/installation/persistent-storage) – configuring the
metrics-collector PV and, optionally, an external TimescaleDB.
* [Credentials](/installation/credentials) – how to supply and rotate the
credentials Thoras uses.
* [Private Container Registry](/installation/private-registry) – pulling images
from your own registry, including air-gapped installs.
## Installing the Thoras license
Thoras will provide you a license key which should be made available to your
installation.
You have two options for managing this value:
1. Pass the value into the Helm release via
`$.Values.imageCredentials.password`, or
2. Reference an existing secret
Let's review both options, choose whichever works best for you!
### Option 1: Pass license in as Helm chart value
Pass the content of your license key file to
`$.Values.imageCredentials.password` in your helm release
### Reference an existing secret
If you'd prefer to reference a pre-existing secret object instead of passing
values into your helm release, this option is for you!
Ensure a secret of type `docker-registry` exists in the `thoras` namespace and
that it's structured correctly:
1. An example of how to create the secret manually:
```sh theme={null}
kubectl create secret docker-registry thoras-license \
--docker-server=us-east4-docker.pkg.dev/thoras-registry/platform \
--docker-username=_json_key_base64 \
--docker-password="{{ YOUR_LICENSE_KEY_CONTENT_HERE }}" \
-n thoras
```
2. In the helm values file, reference this secret by setting
`.Values.imageCredentials.secretRef: "thoras-license"`.
### Confirm license installed successfully
You can confirm the license has been installed successfully if the pods in the
`thoras` namespace come up into `Running` status after one or two minutes. Pod
error statuses such as `ImagePullBackOff` might indicate an issue.
## Forecast worker capacity
Thoras runs a pool of forecast workers that produce predictions for your
enrolled workloads. By default the chart runs a built-in scaler that grows the
pool automatically as you enroll more `AIScaleTargets`, so most installs need no
configuration here.
**How the built-in scaler works**
* Scaler settings live under `thorasForecast.worker.scaler`:
* `minReplicas: 1`: the floor.
* `maxReplicas: 100`: the ceiling.
* `astsPerReplica: 67`: the target ratio. The scaler adds a worker replica for
roughly every 67 `AIScaleTargets`.
* `interval: "1m"`: how often the scaler reconciles.
* `thorasForecast.worker.replicas` is intentionally **unset** by default so the
scaler is in charge.
**Pinning a fixed number of workers**
Setting `thorasForecast.worker.replicas` to an explicit number disables the
built-in scaler and pins the pool at that size. Use this only if you
specifically want a static count (for example, to match a capacity plan or work
around a scaler issue):
```yaml theme={null}
thorasForecast:
worker:
replicas: 3
```
## Configuring Webhook Namespace Exclusions
By default, Thoras webhooks exclude the `kube-system` and `kube-node-lease`
namespaces. You can customize which namespaces are excluded, or disable
exclusions entirely, using the `thorasOperator.webhooks.namespaceSelector` Helm
value.
### Default Behavior
Out of the box, Thoras excludes system namespaces from webhook enforcement:
```yaml theme={null}
thorasOperator:
webhooks:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- kube-system
- kube-node-lease
```
### Customizing Excluded Namespaces
To exclude additional namespaces (e.g. a monitoring namespace), extend the
`values` list:
```yaml theme={null}
thorasOperator:
webhooks:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- kube-system
- kube-node-lease
- monitoring
```
### Disabling Namespace Exclusions
If you use an admission enforcer such as OPA Gatekeeper that manages namespace
selection independently, you can disable the default exclusions by setting the
selector to `null` so the chart omits it entirely:
```yaml theme={null}
thorasOperator:
webhooks:
namespaceSelector: null
```
## Webhook TLS with cert-manager
By default the chart provisions its own webhook TLS material via a bundled
cert-generator Job. If you already run [cert-manager](https://cert-manager.io)
in your cluster, you can hand that job off to cert-manager instead:
```yaml theme={null}
thorasOperator:
webhookCertGen:
certManager:
enabled: true
```
With this enabled, cert-manager provisions the `thoras-tls-certs` Secret and
injects the CA bundle into the webhook configurations. The bundled
kube-webhook-certgen Jobs, along with their RBAC and network policy, are
skipped. This is the recommended setting for GitOps tools such as Argo CD and
Flux, and for any `helm template` based workflow, since Helm hook Jobs are
handled poorly outside of `helm install`.
The chart uses a long-lived CA (10 years by default) that signs the webhook leaf
cert, so routine leaf renewals don't rotate the injected CA bundle. Adjust the
CA lifetime via `thorasOperator.webhookCertGen.certManager.ca.duration` if your
policy requires it.
## Scheduling
The chart exposes standard Kubernetes scheduling primitives at both a global
level and per-component. Component-specific settings take precedence over global
ones.
**Global (applied to every component and to the chart's hook Jobs):**
```yaml theme={null}
nodeSelector:
kubernetes.io/os: linux
tolerations:
- key: dedicated
operator: Equal
value: thoras
effect: NoSchedule
# Global affinity rules that components can opt into via useGlobalAffinity: true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node.kubernetes.io/instance-type
operator: In
values: [m5.large, m5.xlarge]
priorityClassName: thoras-high-priority
# Applied to all components; a component-specific list fully replaces this.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
```
**Per-component overrides** exist under each component block (`thorasOperator`,
`thorasApiServerV2`, `thorasWorker`, `thorasForecast`, `metricsCollector`,
`thorasDashboard`, `thorasConfigController`) as `affinity`, `priorityClassName`,
and `topologySpreadConstraints`. Each component also has a `useGlobalAffinity`
switch (default `true`) that controls whether the global `affinity` block is
merged in on top of the component's own.
## NetworkPolicy
The chart can emit a least-privilege network policy for every Thoras component,
in one of two flavors:
```yaml theme={null}
networkPolicy:
enabled: true
# kubernetes (default): standard networking.k8s.io/v1 NetworkPolicy, works
# with any NetworkPolicy-capable CNI.
# cilium: CiliumNetworkPolicy (cilium.io/v2). Requires Cilium.
flavor: kubernetes
```
**Kubernetes API server ports.** Standard `NetworkPolicy` is evaluated after
kube-proxy translates the `kubernetes` Service address to a real endpoint, so
the `kubernetes` flavor needs to know the port the API server actually listens
on (`networkPolicy.apiServerPorts`, default `[443, 6443]`). Check with
`kubectl get endpoints kubernetes -n default` and add the port if it's
non-standard. Minikube, for example, uses `8443`. The `cilium` flavor targets
the API server by identity and ignores this list.
**Extending the policies.** Each component block accepts `extraIngressRules` and
`extraEgressRules`, appended verbatim to the generated policy in either flavor.
**OIDC egress.** When [dashboard OIDC auth](/guides/dashboard-auth) is enabled,
the chart opens egress on port `443` from the dashboard pod so the sidecar can
reach your identity provider. Tighten with a layered policy scoped to your IdP's
hostnames if required.
## Egress proxy
If Thoras components must dial an outbound HTTP/HTTPS proxy, for example to
reach `console.thoras.ai` for cloud sync, set the standard proxy environment
variables globally:
```yaml theme={null}
proxy:
httpProxy: "http://proxy.example.com:3128"
httpsProxy: "http://proxy.example.com:3128"
noProxy: ".svc.cluster.local,10.0.0.0/8"
```
The values are injected into every Thoras component as `HTTP_PROXY`,
`HTTPS_PROXY`, and `NO_PROXY`.
## Prometheus scraping
Every Thoras component that emits Prometheus metrics has a `prometheus.enabled`
toggle (default `true`) and a fixed `prometheus.port`. The chart can also emit a
`ServiceMonitor` for the Prometheus Operator via the `serviceMonitor` values.
See [Monitoring Thoras](/guides/metrics) for the full scraping configuration and
the list of metrics.
# Credentials
Source: https://docs.thoras.ai/installation/credentials
How Thoras handles the credentials it needs, and how to supply your own.
Thoras needs a handful of credentials to run: an internal API token, the
dashboard login, and the bundled database password. You don't have to provide
any of them. On first install, Thoras generates whatever you didn't supply and
stores it in the `thoras-config-controller` Secret in the release namespace. If
a credential later changes, Thoras restarts the components that use it so you
don't have to.
On a fresh install, workload pods briefly report `CreateContainerConfigError`
while Thoras performs its first reconcile. This clears on its own.
## Supplying your own credentials
Each credential resolves in this order of precedence:
1. **A Secret you manage**, referenced via `*.existingSecret` or `*SecretRef*`
values.
2. **A value pinned in `values.yaml`**.
3. **Generated by Thoras** into the `thoras-config-controller` Secret (default).
For example, to point the internal API token at your own Secret:
```yaml theme={null}
apiClientSecret:
existingSecret:
secretName: my-thoras-api-client
secretKey: api-client-secret
```
The Helm chart README lists every credential and the values keys that back it.
See
[Secrets](https://github.com/thoras-ai/helm-charts/tree/main/charts/thoras#secrets)
for the full table.
## Rotating a credential
Generated values are not intended to be rotated in place. To control rotation,
take ownership of the value by pinning it in `values.yaml` or by pointing the
chart at a Secret you manage.
* **Existing Secret:** update the value in your Secret. Thoras picks up the
change within about a minute.
* **Pinned value:** change it in `values.yaml` and run `helm upgrade`.
Two caveats:
* Rotating the dashboard cookie secret signs every active dashboard session out.
* The bundled TimescaleDB password has no rotation path. Use
[external TimescaleDB](/installation/persistent-storage#external-timescaledb-byodb)
if you need to rotate the database password.
For the exact rotation semantics under GitOps, see
[Rotating secrets](https://github.com/thoras-ai/helm-charts/tree/main/charts/thoras#rotating-secrets)
in the chart README.
## Legacy secret seeding
`featureFlags.enableLegacySecretSeeding` (default `true`) keeps rendering the
chart 4.x `api-client-secret` and `thoras-timescale-password` Secrets so their
values can be migrated on upgrade. Nothing consumes them afterwards.
* **Fresh installs:** set it to `false`.
* **Upgrading from 4.x:** leave it on for the upgrade, then set it to `false`
once the pods are healthy.
* **GitOps and `helm template`:** set it to `false`. The legacy Secrets use Helm
`lookup`, which those workflows cannot evaluate.
The flag and the legacy Secrets are removed in chart 6.0. See
[Deprecations](/deprecations).
## Further reading
* Chart README:
[Secrets](https://github.com/thoras-ai/helm-charts/tree/main/charts/thoras#secrets),
[Rotating secrets](https://github.com/thoras-ai/helm-charts/tree/main/charts/thoras#rotating-secrets),
and
[Thoras Config Controller values](https://github.com/thoras-ai/helm-charts/tree/main/charts/thoras#thoras-config-controller).
* [Default (htpasswd) dashboard auth](/guides/dashboard-auth/htpasswd)
* [ArgoCD](/guides/argo-cd)
# Persistent Storage
Source: https://docs.thoras.ai/installation/persistent-storage
Configure the persistent volume Thoras uses to retain historical usage metrics, and optionally point Thoras at an external TimescaleDB.
To produce production-grade predictions, Thoras needs durable storage for
historical usage metrics. The chart supports two independent pieces of state:
1. A Kubernetes
[persistent volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
(PV) attached to the metrics collector.
2. A TimescaleDB instance for time-series storage. The chart deploys one
in-cluster by default; you can point it at an external database instead.
## Understanding Storage Requirements
You'll want to ensure Thoras has the disk space it needs now and in the future.
This may or may not require configuration based on your backend storage
provider.
### Understanding Usage
The storage requirement for a managed workload is `1GB`. So for a cluster with
twenty managed workloads, Thoras requires `20GB` of disk space.
### Understanding Kubernetes Storage Backend
In general there are two patterns for Kubernetes storage backends with regard to
disk space configuration:
#### Growable Storage Backends
These are PV storage backends that don't enforce a fixed size, they dynamically
grow as-needed and require no Thoras configuration to specify volume size.
Examples include:
* `AWS Elastic File System (EFS)`
* `Google Cloud Filestore (enterprise tier)`
* `Azure Files`
* `CephFS`
**Action:** If you use one of these backends, you don't need to allocate disk
space in the Thoras configuration, the disk will automatically grow to meet your
needs.
#### Fixed-Size Storage Backends
These are PV storage backends that require explicit provisioning of a fixed
capacity and enforce the size specified in the `PersistentVolumeClaim`. These
backends require you to specify the desired size of the disk, which you should
configure as a Helm chart value (read on for details on how to do that).
Examples include:
* `AWS Elastic Block Store (EBS)`
* `Google Persistent Disk (GCP PD)`
* `Azure Disk Storage`
* `VMware vSphere Volumes`
**Action:** If you use a fixed-size storage backend, you'll need to specify the
volume size as a Helm value:
1. Determine how much total size will be needed using
[guidance above](#understanding-usage)
2. Configure the storage size by setting the following helm value:
```yaml theme={null}
metricsCollector:
persistence:
# Or whatever size needed based on your math from step #1
pvcStorageRequestSize: "100Gi"
```
## Managing Persistent Volumes
You may have an established workflow for provisioning Kubernetes Persistent
Volumes (PV). That's great! Here's how to get up and running depending on your
preferred approach:
### Referencing an existing `StorageClass` provisioner
If you have a Kubernetes `StorageClass` that you'd like to use to dynamically
provision your PV (such as an
[EFS StorageClass](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/dynamic_provisioning/specs/storageclass.yaml)).
In that case, simply reference the storage class name in your Thoras helm chart
`values.yaml`:
```yaml theme={null}
metricsCollector:
persistence:
storageClassName: "{{ YOUR_STORAGE_CLASS_NAME }}"
```
`metricsCollector.persistence.enabled` defaults to `null`, which auto-enables
persistence as soon as you set any of `storageClassName`,
`pvcStorageRequestSize`, or `createEFSStorageClass.fileSystemId`. Setting
`enabled: true` is only necessary if you want to force persistence on without
configuring any of the above.
### Referencing an existing PV
If you have a workflow for provisioning a PV and would like to simply reference
the PV directly by name, add the following config block to your Helm
`values.yaml`:
```yaml theme={null}
metricsCollector:
persistence:
enabled: true
volumeName: "{{ YOUR_VOLUME_NAME_HERE }}"
```
## External TimescaleDB (BYODB)
The chart deploys an in-cluster TimescaleDB by default. If you'd rather manage
TimescaleDB yourself, for backups, HA, or shared infrastructure, point Thoras at
an external database instead:
```yaml theme={null}
externalTimescale:
# Full postgres DSN, including the database name.
dsn: "postgres://user:password@host:5432/tsdb?sslmode=require"
```
Or reference an existing Secret containing the DSN:
```yaml theme={null}
externalTimescale:
secretRefName: my-timescale-dsn
secretRefKey: dsn
```
When either is set, the chart skips deploying its bundled TimescaleDB and wires
all components to the external database. The TimescaleDB extension must be
pre-installed on the external database.
See [Credentials](/installation/credentials) for how the DSN interacts with the
other credentials Thoras manages.
## Questions?
For help, email [support@thoras.ai](mailto:support@thoras.ai).
# Private Container Registry
Source: https://docs.thoras.ai/installation/private-registry
Pull Thoras container images from your own private registry instead of the public Thoras registry.
If you need to pull Thoras container images from your own private registry
instead of the public Thoras registry, you can override the default registry
location using the `$.Values.imageCredentials.registry` value.
## Prerequisites
1. Access to download Thoras container images (email
[support@thoras.ai](mailto:support@thoras.ai) for assistance)
2. A private container registry accessible from your Kubernetes cluster
3. Container images pushed to your private registry
## Configuration
Set the private registry hostname value in your Helm `values.yaml`:
```yaml theme={null}
imageCredentials:
registry: "your-private-registry.example.com/thoras"
```
## Copying Multi-Platform Images
Thoras container images are built for multiple platforms (`linux/amd64` and
`linux/arm64`) to support multiple CPU architectures. When copying images from
the Thoras registry to your private registry, it's a good idea to preserve the
multi-platform manifest to ensure compatibility across different CPU
architectures.
### Generate a list of images to copy
```sh theme={null}
helm repo add thoras https://thoras-ai.github.io/helm-charts
helm template testit thoras/thoras \
--set featureFlags.enableLegacySecretSeeding=false \
| awk '/image:/ {print $NF}' | sort -u
```
The list will include Thoras images (`services`, `thoras-dashboard-v2`,
`thoras-forecast`, `nginx`, `timescaledb`) as well as `oauth2-proxy` and `httpd`
(the dashboard auth sidecar and its init container) and `kube-webhook-certgen`
(webhook TLS, unless you use cert-manager). All are hosted on the Thoras
registry, so air-gapped installs only need to mirror from the Thoras registry.
No separate upstream mirror step is needed.
If you deploy with `helm template` rather than `helm install`, set
`featureFlags.enableLegacySecretSeeding: false` in your values (see [Legacy
secret seeding](/installation/credentials#legacy-secret-seeding)) and consider
[cert-manager for webhook
TLS](/installation/advanced-setup#webhook-tls-with-cert-manager) so no Helm
hook Jobs are involved.
### Using Docker Buildx
The recommended approach is to use `docker buildx imagetools create`, which
preserves the multi-platform manifest. Substitute `` with the tag that
`helm template` shows for the release you're deploying:
```sh theme={null}
# Authenticate to both registries
docker login us-east4-docker.pkg.dev/thoras-registry/platform
docker login your-private-registry.example.com
# Copy the image with all platforms preserved
docker buildx imagetools create \
--tag your-private-registry.example.com/thoras/services: \
us-east4-docker.pkg.dev/thoras-registry/platform/services:
```
### Verifying Multi-Platform Support
After copying, verify that both platforms are available:
```sh theme={null}
docker buildx imagetools inspect your-private-registry.example.com/thoras/services:
```
You should see both `linux/amd64` and `linux/arm64` in the output.
## Questions?
For help downloading container images or configuring your private registry,
email [support@thoras.ai](mailto:support@thoras.ai).
# System Requirements
Source: https://docs.thoras.ai/installation/system-requirements
* **Kubernetes**. It doesn't matter which cloud, managed vs unmanaged, or
on-prem. If it's Kubernetes, Thoras has your back! Thoras supports the five
most recent upstream Kubernetes minor versions. See the
[support policy](/policies/support-policy#kubernetes-version-support).
* **Kubernetes metrics server**. Thoras reads live resource usage from the
[metrics server](https://github.com/kubernetes-sigs/metrics-server). Most
clusters already have it.
* **A Persistent Volume**. For production-grade predictions, Thoras requires a
[Kubernetes persistent volume](./persistent-storage). Note that a PV is not
required for a demo or trial context!
Thoras is extremely lightweight and a small install runs on about 2 vCPU.
Forecast capacity scales itself as you enroll workloads. See
[Forecast worker capacity](./advanced-setup#forecast-worker-capacity) for
details. Clusters with 5,000 or more enrolled workloads should review
[Tuning for Large Clusters](/guides/large-clusters).
## Optional integrations
* **cert-manager**, to provision the admission webhook certificates
declaratively instead of via Helm hook Jobs. Recommended for GitOps. See
[Webhook TLS with cert-manager](./advanced-setup#webhook-tls-with-cert-manager).
* **Cilium**, if you want the chart's network policies rendered as
`CiliumNetworkPolicy`. Standard `NetworkPolicy` works with any capable CNI.
See [NetworkPolicy](./advanced-setup#networkpolicy).
* **An OIDC identity provider** (Okta, Azure Entra ID, or any OIDC-compliant
provider) for dashboard single sign-on. See
[Dashboard authentication](/guides/dashboard-auth).
* **A private container registry**, if you're running in an air-gapped
environment or prefer to mirror Thoras images internally. See
[Private Container Registry](./private-registry).
# Upgrading
Source: https://docs.thoras.ai/installation/upgrading
How Thoras Helm chart versions map to upgrade effort, and what changed in chart 5.x.
Thoras ships continuously. Within a single major version (for example `5.x.x`),
minor and patch releases are routine and should not require any special steps.
A major chart version bump (for example `4.x.x` to `5.x.x`) signals a breaking
change that may need human intervention. Only one major version can be crossed
at a time (for example, `4.x.x` to `6.x.x` is not supported).
The authoritative, step-by-step migration notes for every major version live in
the chart repository:
Detailed upgrade procedures for each breaking chart release.
## Chart 5.x at a glance
Chart `5.0` adds dashboard authentication and moves credential management into
the cluster. Most `4.x` installs upgrade with no changes to `values.yaml`.
* **The dashboard requires a login.** An oauth2-proxy sidecar now fronts the
dashboard, using a generated username and password by default or your identity
provider via OIDC. See [Dashboard authentication](/guides/dashboard-auth).
* **Credentials are managed in-cluster.** A new `thoras-config-controller`
component seeds any credential you don't supply, migrates your existing 4.x
credentials automatically, and restarts dependent workloads when a credential
changes. See [Credentials](/installation/credentials).
* **`featureFlags.enableSimpleAuthSecret` is renamed** to
`apiClientSecret.enabled`. The old name still works as an alias. See
[Deprecations](/deprecations).
## Does anything apply to me?
Work through whichever of these match your deployment:
| Your 4.x setup | What to do |
| - | - |
| Dashboard reached via `kubectl port-forward` or an ingress with no auth in front | Nothing before upgrading. Afterwards, sign in with the generated credentials. See [Default (htpasswd) auth](/guides/dashboard-auth/htpasswd). |
| Your own oauth2-proxy under `thorasDashboard.extraContainers` | Move to the built-in sidecar. Your IdP app and Secret carry over. See [Migrating from a hand-rolled sidecar](/guides/dashboard-auth#migrating-from-a-hand-rolled-oauth2-proxy-sidecar). |
| Auth already enforced at the ingress, gateway, or mesh | Set `thorasDashboard.auth.enabled: false`. See [Bring your own auth](/guides/dashboard-auth#bring-your-own-auth). |
| `featureFlags.enableSimpleAuthSecret` in `values.yaml` | Rename to `apiClientSecret.enabled`. |
| Deployed with Argo CD, Flux, or `helm template` | During the upgrade, keep `jsonPointers: [/data]` `ignoreDifferences` entries on the `api-client-secret` and `thoras-timescale-password` Secrets so Argo doesn't overwrite them before config-controller migrates their values. Once the pods are healthy, set `featureFlags.enableLegacySecretSeeding: false` and remove those entries. See [ArgoCD](/guides/argo-cd). |
If you use the chart's legacy secret seeding (the 4.x default), complete the
upgrade with `featureFlags.enableLegacySecretSeeding` left at its default of
`true` so the config-controller can migrate your existing credentials. Once
all pods are healthy, set it to `false`.
# Support and Deprecation Policy
Source: https://docs.thoras.ai/policies/support-policy
Thoras support and deprecation policy: supported versions, compatibility commitments, Kubernetes version support, and breaking change notification.
This policy describes how Thoras supports customers, communicates changes, and
maintains stability across releases. The model is designed so that staying up to
date is safe and straightforward: every release contains the latest fixes,
upgrades within a major version do not require migration work, and any change
that would require migration is communicated well in advance.
## Supported versions
Thoras ships continuously. Bug fixes and security patches are included in every
release, and customers receive them by upgrading to the latest version. Because
Thoras maintains strict compatibility within a major version (described below),
upgrades within a major are designed to be safe and routine: no migration work,
no breaking changes, no surprises. Previous release branches are not maintained.
## Compatibility within a major version
Within a major version, Thoras does not introduce breaking changes to:
* `AIScaleTarget`, `ClusterAIScaleTemplate`, and other documented Custom
Resource Definitions
* Helm chart values (no renames or removals without the deprecation process
described below)
Behavior improvements that do not alter the interfaces listed above (forecasting
accuracy, internal defaults, performance) may ship in any minor or patch
release.
This compatibility commitment is the foundation of the support model: because
nothing breaks between releases within a major version, customers can
confidently apply the latest release whenever they need a fix.
## APIs
Thoras APIs are internal. They are not part of the public product surface, are
not covered by this policy, and may change in any release without prior notice.
Customers who require a stable API should contact their Thoras Product Adoption
Engineer. Any API stability commitment requires a separate written agreement and
is not implied by this policy.
## Breaking changes
Breaking changes to the interfaces listed under "Compatibility within a major
version" are reserved for major version releases. Before a major release, Thoras
will provide:
* At least 90 days written notice via release notes, the documentation site, and
email to listed customer contacts
* A migration guide
The 90-day notice period gives customers time to plan and execute the migration.
When the new major reaches general availability, support consolidates on the new
version. For migrations that need additional time or coordination, customers
should engage their Product Adoption Engineer; Thoras works directly with
customers to make these transitions successful.
## Kubernetes version support
Thoras supports the five most recent upstream Kubernetes minor versions. When a
new upstream minor reaches general availability, the oldest supported version is
dropped at the next Thoras release. The current list of supported versions is
published in each Thoras release notes. Older versions may continue to work but
are not part of the Thoras test matrix and compatibility is not guaranteed.
## Deprecations
Any field or behavior marked deprecated within the interfaces listed under
"Compatibility within a major version" will continue to function for at least 90
days from the deprecation announcement before removal. Deprecations are listed
in release notes at the time of announcement and tracked at
[docs.thoras.ai/deprecations](/deprecations), giving customers lead time to
adjust.
## Out of scope
* Alpha and beta features carry no compatibility or deprecation guarantee. They
may change or be removed at any time.
* Behaviors not described in the public documentation are not covered by this
policy.
## Questions
For questions about this policy or about a specific upgrade or migration, email
[support@thoras.ai](mailto:support@thoras.ai) or contact your Thoras Product
Adoption Engineer.
# AIScaleTarget
Source: https://docs.thoras.ai/reference/ast-definition
The specification describes the `AIScaleTarget` Custom Resource that defines how
Thoras scales workloads.
The following is a sample `AIScaleTarget` definition:
```yaml ast.yaml theme={null}
apiVersion: thoras.ai/v1
kind: AIScaleTarget
metadata:
name: "{{ YOUR_AST_NAME }}"
namespace: "{{ YOUR_NAMESPACE }}"
spec:
scaleTargetRef:
kind: Deployment
name: "{{ YOUR_AST_NAME }}"
model:
forecast_blocks: 4h
forecast_buffer_percentage: 0%
forecast_interval: 1h
mode: balanced
horizontal:
mode: recommendation
scaling_behavior:
scale_up:
type: percent
percent: 50
scale_down:
type: percent
percent: 50
vertical:
containers:
- name: "{{ CONTAINER_NAME }}"
cpu:
lowerbound: 20m
upperbound: 1
memory:
lowerbound: 50Mi
upperbound: 2G
mode: recommendation
scaling_behavior:
scale_up:
type: percent
percent: 50
scale_down:
type: percent
percent: 50
update_policy:
update_mode: in_place_or_recreate
```
## `metadata`
```yaml ast.yaml theme={null}
metadata:
name: {{YOUR_AST_NAME}}
namespace: {{YOUR_NAMESPACE}}
```
It's recommended that `metadata.name` matches the name of the workload being
scaled.
### Keeping history when an AST is recreated
Requires chart version `5.6.0`
Deleting an AIScaleTarget and creating it again, for example during a migration
or a reinstall, gives it a new identity, and it starts with empty history. To
keep the history of the AIScaleTarget it replaces, add the
`thoras.ai/inherit-history` annotation when you create it:
```yaml ast.yaml theme={null}
metadata:
name: {{YOUR_AST_NAME}}
namespace: {{YOUR_NAMESPACE}}
annotations:
thoras.ai/inherit-history: "true"
```
The new AIScaleTarget then reads the history of every earlier AIScaleTarget with
the same namespace and name: its metric history and charts, forecast performance
history, and cost history. The earlier data isn't copied or changed. Thoras only
links it to the new AIScaleTarget.
Current state still starts fresh: the latest forecast signals, current metrics,
and readiness for autonomous mode come from the new AIScaleTarget's own data.
A few rules apply:
* **Set it at creation.** The annotation only takes effect on an AIScaleTarget
that is less than 10 minutes old. On an older AIScaleTarget, Thoras reports
`skipped-not-new` and inherits nothing.
* **Only retained history is inherited.** A deleted AIScaleTarget's data is kept
until it ages out under normal retention. Detailed metrics and forecast data
are kept for 30 days, so only what's left of them comes back. Daily usage and
cost history is kept longer.
* **Undo is permanent.** Setting the annotation to `"false"` removes the link to
the earlier history, which stays untouched. Setting it back to `"true"`
afterwards doesn't restore it.
* **Templates don't set it.** AIScaleTargets created by a
[ClusterAIScaleTemplate](/reference/cluster-aiscale-template) don't carry the
annotation, so they start with empty history.
Thoras reports progress in the `thoras.ai/inherit-history-status` annotation.
Its value starts with one of the statuses below, followed by the AIScaleTarget's
UID:
```sh theme={null}
kubectl get ast {{YOUR_AST_NAME}} -n {{YOUR_NAMESPACE}} \
-o jsonpath='{.metadata.annotations.thoras\.ai/inherit-history-status}'
```
| Status | Meaning |
| - | - |
| `queued` | Thoras is linking the earlier history. |
| `done` | The earlier history is linked. `inherited=` is the number of earlier ASTs. |
| `nothing-to-inherit` | No earlier AIScaleTarget with this name has retained data. |
| `skipped-not-new` | The annotation was added after the first 10 minutes, so nothing was inherited. |
| `removed` | The annotation was set to `"false"` and the link was removed. |
| `failed` | Linking failed after several retries. The AIScaleTarget works with its own history. |
## `scaleTargetRef`
```yaml ast.yaml theme={null}
scaleTargetRef:
kind: Deployment
name: {{YOUR_AST_NAME}}
namespace: {{YOUR_NAMESPACE}}
apiVersion: apps/v1
```
`scaleTargetRef` is the reference to a specific workload that Thoras will scale
up or down. The workload must reside in the same namespace as the
`AIScaleTarget`.
`apiVersion` specifies the API group and version that defines the resource,
while `kind` identifies the type of resource (e.g., `Deployment`, `StatefulSet`,
`Rollout`, etc).
**Note:** `scaleTargetRef` and `selector` are mutually exclusive. Use
`scaleTargetRef` to target a single workload by name, or use `selector` to
target multiple workloads by label.
## `selector`
```yaml ast.yaml theme={null}
selector:
matchLabels:
app: my-service
environment: production
```
`selector` allows you to target multiple workloads by
[pod labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
instead of specifying a single workload by name. This is useful when you want to
apply the same scaling policy to multiple workloads that share common labels.
**Important constraints:**
* `selector` and `scaleTargetRef` are mutually exclusive, you must use one or
the other, not both.
* When using `selector`, horizontal scaling is **not supported**. You must leave
`spec.horizontal` empty or omit it entirely.
* Pods must be managed by a Deployment, StatefulSet, or Argo Rollout. Standalone
pods or other controller types are not supported.
**How it works:**
When a `selector` is defined, Thoras identifies all pod controllers
(Deployments, StatefulSets, Rollouts) in the same namespace whose pods match the
label selector.
In `autonomous` mode:
* Thoras restarts each matching pod controller one by one when a resource
request suggestion comes in from the forecaster.
* If `update_policy.update_mode` is set to `in_place` or `in_place_or_recreate`,
pods are resized in place without restarts (when possible).
## `model`
```yaml ast.yaml theme={null}
model:
forecast_blocks: 4h
forecast_buffer_percentage: 0%
forecast_interval: 1h
```
## `model.forecast_blocks`
The window of future time each forecast covers. Thoras predicts the **peak**
(maximum) load over that window and recommends resources or replicas for that
peak, not for the average or the load at the end of the window.
For example, with `forecast_blocks: 4h` and `forecast_interval: 1h`, every hour
Thoras predicts the highest load expected in the next 4 hours and sizes the
workload for it. The recommendation only drops once the predicted peak is more
than 4 hours away.
**Choosing a value:**
* **Shorter windows** track demand more closely, so the workload scales down
sooner after a peak passes. Lowest cost, least headroom.
* **Longer windows** hold capacity for any peak that falls inside the window.
More headroom, more overprovisioning between peaks.
**Cost of changing it:** a longer window adds no meaningful load to Thoras.
Changing the value retrains the workload's model once, and a model needs at
least 4x `forecast_blocks` of metric history to train (e.g. `24h` needs 4 days).
Until enough history exists, Thoras sizes for the peak of observed data.
**Important:** `forecast_blocks` must be greater than or equal to
`forecast_interval`. For example, if forecasts run every `1h`, `forecast_blocks`
must be at least `1h`, otherwise there would be stretches of time no forecast
covers.
You will need to specify either the "m" (for minutes) or "h" (for hours) unit
for the value of [model.forecast\_blocks](#model-forecast-blocks).
## `model.forecast_buffer_percentage`
Defines an additional buffer applied on top of the forecasted resource usage to
reduce the risk of under-provisioning.
For example, if the forecasted CPU usage is 500m and
`forecast_buffer_percentage` is set to 10%, the final recommended value will be
550m. Setting this to 0% means no buffer will be added.
## `model.forecast_interval`
Controls how frequently Thoras generates forecasts and evaluates scaling.
Accepts a plain duration string (e.g., `"15m"`, `"1h"`). Shorter intervals can
lead to more accurate forecasts but may result in more frequent scaling.
`forecast_interval` controls how often forecasts are made, while
[model.forecast\_blocks](#model-forecast-blocks) defines the time window each
forecast covers.
## `model.mode`
Defines the optimization strategy that controls how tightly forecast
recommendations track expected usage.
Thoras modes span a spectrum from efficiency-focused to assurance-focused:
`undershoot` → `cost_optimized` → `balanced` → `assurance_optimized` →
`overshoot`
Available modes:
* `undershoot` - Most aggressive strategy, allowing recommendations to run below
usage more often
* `cost_optimized` - Tighter recommendations with increased efficiency focus
* `balanced` (default) - Balanced reliability and efficiency for general
production workloads
* `assurance_optimized` - More conservative recommendations that track above
typical usage patterns
* `overshoot` - Most conservative strategy, maintaining additional headroom
based on historical usage
* `percentile` - Deterministic history-based forecast set at a specific
percentile of observed usage. Requires `percentile_value` (0–100). An integer
or quoted decimal is accepted (e.g. `75` or `"99.9"`).
If not specified, the default is `balanced`.
**Visit
[Choosing a Forecast Optimization Strategy](/guides/forecast-optimization-strategies)
for detailed guidance on selecting the right mode for your workload.**
## `model.metrics`
```yaml ast.yaml theme={null}
model:
mode: balanced
metrics:
cpu:
mode: undershoot
memory:
mode: balanced
forecast_buffer_percentage: 10%
```
Per-metric overrides allow you to configure different forecast settings for CPU
and memory independently, overriding the global `model.mode` and
`model.forecast_buffer_percentage` settings.
**Common use cases:**
* Apply aggressive cost optimization to CPU while maintaining conservative
memory forecasts
* Add extra buffer to memory forecasts to prevent OOM errors while keeping CPU
forecasts tight
* Test different optimization strategies per resource type
**Configuration:**
Each metric override supports:
* `mode` - Override the global forecast mode for this metric. Same values as the
top-level `model.mode`: `balanced`, `cost_optimized`, `undershoot`,
`overshoot`, `assurance_optimized`, or `percentile`.
* `forecast_buffer_percentage` - Override the global buffer percentage for this
metric
**Example: Mixed strategy**
```yaml ast.yaml theme={null}
model:
mode: balanced # Global default
forecast_buffer_percentage: 0%
metrics:
cpu:
mode: undershoot # More aggressive for CPU
memory:
mode: balanced
forecast_buffer_percentage: 10% # Extra buffer for memory
```
In this configuration:
* CPU uses `undershoot` mode (ignoring global `balanced`)
* Memory uses `balanced` mode with 10% buffer (ignoring global 0%)
If a metric is not specified in `model.metrics`, it inherits the global
`model.mode` and `model.forecast_buffer_percentage` settings.
## `model.min_lookback_to_scale`
Minimum lookback window required before autonomous scaling is enabled.
Autonomous scaling will only occur if historical data spans at least this
duration.
**Default:** `24h` (inherited from helm chart value
`thorasForecast.minLookbackToScale` if not specified)
**Format:** Duration string (e.g., `"24h"`, `"7h30m"`). Cannot exceed metrics
retention window (typically 14 days).
**Example:**
```yaml ast.yaml theme={null}
model:
min_lookback_to_scale: "24h" # Override default: require 24 hours before autonomous scaling
```
This setting ensures sufficient historical data exists before allowing
autonomous scaling decisions. Only applies when `mode` is set to `autonomous` -
has no effect in recommendation mode.
## `horizontal`
```yaml ast.yaml theme={null}
horizontal:
mode: recommendation
minReplicas: 2
maxReplicas: 20
```
Thoras horizontal scaling adjusts the number of replicas based on forecasted or
observed workload demands.
The `mode` field controls whether Thoras provides recommendations or actively
scales your workload:
In `autonomous` mode:
* Horizontally-scaled workloads **must** have CPU and/or memory requests defined
in the pod spec if you plan to scale on `averageUtilization`.
* **The target deployment of your `AIScaleTarget` must have a single existing
[Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).**
> **Note:** Only one scaling direction (horizontal or vertical) can be in
> `autonomous` mode at a time. See
> [Understanding Vertical and Horizontal Scaling Modes](/guides/configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
> for additional details.
In `recommendation` mode :
* System does **not** require an existing HPA. Thoras will assume an 80%
utilization target for CPU and memory when generating scaling suggestions.
Thoras does **not** replace your HPA, it works alongside it by feeding
forecasted metrics into your existing HPA configuration. You can customize
behavior by:
* Opting in additional metrics using `spec.horizontal.additional_metrics`
* Opting out of default metrics using `spec.horizontal.exclude_metrics`
This setup gives you full control over how scaling decisions are made while
benefiting from Thoras' predictive intelligence.
To enable horizontal scaling recommendations, set `spec.horizontal.mode` to
`recommendation`. It is recommended to start in `recommendation` mode to allow
the model time to learn workload patterns and validate scaling suggestions. Once
the recommendations align with your performance expectations, you can switch to
`autonomous` mode for automated scaling.
## `horizontal.minReplicas`
The lower bound for the number of replicas Thoras will scale down to. When set,
this value takes precedence over the `minReplicas` configured on the target HPA.
If not set, Thoras falls back to the HPA's `minReplicas` value.
**Note:** `minReplicas` must not be greater than `maxReplicas` when both are set
, the CRD will reject the configuration if this constraint is violated.
## `horizontal.maxReplicas`
The upper bound for the number of replicas Thoras will scale up to. When set,
this value takes precedence over the `maxReplicas` configured on the target HPA.
If not set, Thoras falls back to the HPA's `maxReplicas` value.
**Note:** `maxReplicas` must not be less than `minReplicas` when both are set ,
the CRD will reject the configuration if this constraint is violated.
**Example: Constraining replicas independently of HPA**
```yaml ast.yaml theme={null}
horizontal:
mode: autonomous
minReplicas: 2
maxReplicas: 20
```
This gives teams fine-grained control over Thoras scaling behavior independent
of HPA configuration. The HPA bounds remain in place and continue to govern
Kubernetes' own scaling, but Thoras will only apply replica counts within the
AST-defined range.
**Visit [Predictive Horizontal Pod Autoscaling with Thoras Guide](/guides/hpa)
for more info.**
## `horizontal.multipleOf`
Constrains autonomous replica scaling to exact multiples of this value. When
set, Thoras rounds any desired replica count **up** to the nearest multiple
before applying it.
Must be a positive integer. The CRD rejects configurations where `multipleOf`
exceeds `maxReplicas`.
**Example: StatefulSet sharded across 8 partitions**
```yaml ast.yaml theme={null}
horizontal:
mode: autonomous
maxReplicas: 24
multipleOf: 8
```
Thoras will only set replica counts of 8, 16, or 24.
**Visit
[Constraining Replicas to Fixed Multiples](/guides/hpa#constraining-replicas-to-fixed-multiples)
for more info.**
## `horizontal.allowedReplicas`
Constrains autonomous replica scaling to an explicit set of valid replica
counts. When set, Thoras rounds any desired count **up** to the smallest value
in the set that meets or exceeds it, and never scales beyond the largest value.
Must be a non-empty list of unique positive integers (up to 64 values). Every
value must fall within `minReplicas` and `maxReplicas` when those are set, and
`allowedReplicas` cannot be combined with `multipleOf`. The CRD rejects
configurations that violate these constraints.
**Example: a topic sharded across divisors of its partition count**
```yaml ast.yaml theme={null}
horizontal:
mode: autonomous
minReplicas: 4
maxReplicas: 20
allowedReplicas: [4, 6, 8, 12, 16]
```
Thoras will only set replica counts from this set. For example, a forecast of 10
scales to 12.
**Visit
[Constraining Replicas to a Fixed Set](/guides/hpa#constraining-replicas-to-a-fixed-set)
for more info.**
## `vertical`
```yaml ast.yaml theme={null}
vertical:
containers:
- name: {{CONTAINER_NAME}}
cpu:
lowerbound: 20m # Mandatory
upperbound: 1 # Optional
memory:
lowerbound: 50Mi # Mandatory
upperbound: 2G # Optional
mode: recommendation
update_policy:
update_mode: in_place_or_recreate
```
The Thoras vertical scaler adjusts container-level CPU and memory resource
requests based on forecasted utilization.
The `mode` field controls whether Thoras provides recommendations or actively
scales your workload:
To define a vertical scaling policy, you'll want to set the following fields:
* `vertical.mode: recommendation` - we recommend running Thoras in
`recommendation` mode for at least a day for the model to train before
enabling `autonomous`.
* `vertical.containers[0].memory.lowerbound` is always required and
`vertical.containers[0].memory.upperbound` is optional if you had a preference
for a memory floor and ceiling for your target workload. In `autonomous` mode,
changing these bounds re-enforces them immediately on live pods that fall
outside the new range, according to `update_policy`. With
`update_mode: initial`, bounds apply only when pods are next created.
* `vertical.containers[].RESOURCE.limit` allows Thoras to modify the container's
limit along with the request for this resource and is optional. If unset,
Thoras will *not* modify limits for this resource.
* `vertical.containers[].RESOURCE.limit.ratio` the ratio used to keep the limit
inline with the suggested request for this resource. E.g. for a ratio `2.5`
Thoras will make the limit `2.5G` if the suggestion for the request was for
`1G`.
### Limit Enforcement
Thoras respects resource limits to ensure safe right-sizing operations:
* **AIScaleTarget `upperbound`**: Thoras won't right-size a pod above the
configured `upperbound` during steady-state operation. The one exception is
[OOM remediation](/guides/oom-remediation), an opt-in emergency response that
intentionally bypasses `memory.upperbound` to recover a workload from an
out-of-memory kill loop; the bound is re-applied once its stabilization window
expires.
* **Bound changes in autonomous mode**: If you update `lowerbound` or
`upperbound` and live requests are outside the new range, Thoras applies the
new bounds right away per `update_policy` (see
[Predictive Vertical Pod Rightsizing](/guides/vertical-pod-rightsizing#vertical-breakdown)).
* **Pod spec `limit`** is also enforced, Thoras will never right-size a pod
above its defined limit
* **Combined enforcement**: When both `upperbound` and pod `limit` are set,
Thoras uses the lower value as the maximum for right-sizing
> **Note:** Only one scaling direction (horizontal or vertical) can be in
> `autonomous` mode at a time. See
> [Understanding Vertical and Horizontal Scaling Modes](/guides/configuring-asts#understanding-vertical-and-horizontal-scaling-modes)
> for details on how vertical and horizontal modes interact.
**Visit
[Predictive Vertical Pod Rightsizing Guide](/guides/vertical-pod-rightsizing)
for more info.**
### Container Name Wildcard Matching
The `vertical.containers[].name` field supports wildcard patterns using `*` to
match multiple containers with a single configuration. This is useful when you
have multiple containers with similar naming patterns or when you want to apply
the same resource bounds to all containers.
**Important:** Container configurations are matched in order from top to bottom.
The first matching pattern is used. Place more specific patterns before generic
ones.
**Matching Rules:**
* **Exact match**: Pattern matches the container name exactly
* **Wildcard `*`**: Matches any sequence of characters (including empty string)
* **Prefix match**: `app-*` matches any container starting with `app-`
* **Suffix match**: `*-sidecar` matches any container ending with `-sidecar`
* **Contains match**: `*cache*` matches any container containing `cache`
* **Multiple wildcards**: `app-*-prod-*` matches containers with `app-` prefix
and `-prod-` in the middle
**Important:** If a pattern doesn't start with `*`, it must match from the
beginning. If it doesn't end with `*`, it must match to the end.
**Examples:**
```yaml ast.yaml theme={null}
vertical:
containers:
# Specific patterns first - these will match before the wildcard
# Match containers starting with "app-"
- name: "app-*"
cpu:
lowerbound: 100m
upperbound: 2
memory:
lowerbound: 128Mi
upperbound: 4G
# Match containers ending with "-sidecar"
- name: "*-sidecar"
cpu:
lowerbound: 10m
upperbound: 500m
memory:
lowerbound: 32Mi
upperbound: 512Mi
# Match containers containing "cache"
- name: "*cache*"
cpu:
lowerbound: 50m
upperbound: 1
memory:
lowerbound: 256Mi
upperbound: 8G
# Match specific pattern with multiple wildcards
- name: "app-*-prod-*"
cpu:
lowerbound: 200m
upperbound: 4
memory:
lowerbound: 512Mi
upperbound: 16G
# Generic wildcard last - catches any remaining containers
# This will only match containers that didn't match patterns above
- name: "*"
cpu:
lowerbound: 20m
upperbound: 1
memory:
lowerbound: 50Mi
upperbound: 2G
```
**Pattern Matching Examples:**
| Pattern | Container Name | Matches | Reason |
| - | - | - | - |
| `*` | `any-container` | ✅ | Wildcard matches everything |
| `app-*` | `app-frontend` | ✅ | Starts with `app-` |
| `app-*` | `my-app-frontend` | ❌ | Doesn't start with `app-` |
| `*-sidecar` | `logging-sidecar` | ✅ | Ends with `-sidecar` |
| `*-sidecar` | `sidecar-proxy` | ❌ | Doesn't end with `-sidecar` |
| `*cache*` | `redis-cache-primary` | ✅ | Contains `cache` |
| `*cache*` | `redis-primary` | ❌ | Doesn't contain `cache` |
| `app-*-prod` | `app-frontend-prod` | ✅ | Matches pattern |
| `app-*-prod` | `app-frontend-prod-v2` | ❌ | Has extra content after `prod` |
### `update_policy`
```yaml theme={null}
vertical:
update_policy:
update_mode: in_place_or_recreate
recreate_resources: ["memory"]
```
The `update_policy` field controls pod update behavior during vertical scaling
operations.
**Prerequisites for in-place updates:**
* Kubernetes 1.33+
**Configuration:**
* `update_mode` (string): Determines how pod updates are applied:
* `recreate` - Pods are recreated when resource changes are applied
* `in_place` - Changes applied without pod recreation (requires Kubernetes
1.33+)
* `in_place_or_recreate` - Attempts in-place updates with fallback to
recreation (recommended)
* `initial` - Recommendations apply only during pod creation
* `recreate_resources` (optional array): Specifies which resources (`memory`,
`cpu`) trigger pod recreation when using recreate or fallback modes
**How it works:**
1. When `update_mode` is `in_place` or `in_place_or_recreate`, Thoras attempts
to resize pods in place using the Kubernetes resize API. Resizes are applied
in batches rather than to the whole workload at once — the batch size and
cadence are controlled by [`disruption`](#disruption)
2. When `update_mode` is `in_place_or_recreate` and the pod's
[QoS class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
would change or the resize operation is not supported, Thoras falls back to
evicting the pod
3. Evicted pods are rolled out gradually rather than all at once, minimizing
service disruption. The eviction batch size and cadence are also controlled
by [`disruption`](#disruption)
For details on what causes a resize to stall and how each mode responds, see
[In-Place Resize Behavior](/guides/in-place-resize).
**Benefits:**
* Reduced disruption: Pods are not recreated unless necessary when using
in-place updates
* Faster scaling: Resource adjustments take effect immediately
* Better node utilization: Enables more efficient bin-packing and cost savings,
especially when combined with node autoscaling tools like Karpenter
* Graceful fallback: Automatic eviction when in-place resizing is not possible
* Fine-grained control: Specify which resources trigger pod recreation
### `oom_remediation`
```yaml theme={null}
vertical:
mode: autonomous
oom_remediation:
enabled: true
stabilization_window: 2h
```
Enables automatic memory adjustment when OOM kills are detected. Only applies
when `vertical.mode` is `autonomous`.
**Fields:**
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `enabled` | bool | Yes | `false` | Activates OOM remediation for this workload. |
| `stabilization_window` | duration | No | `max(forecast_interval, 1h)` | How long the adjusted memory is held as a floor after an OOM. Forecasts can raise memory above the floor but cannot lower it until the window expires. Resets on each new OOM adjustment. |
**Visit [OOM Remediation Guide](/guides/oom-remediation) for detailed
configuration guidance.**
### `disruption`
```yaml ast.yaml theme={null}
vertical:
disruption:
interval: 60s
max_disrupted: "20%"
```
`disruption` controls how aggressively Thoras applies pod changes (in-place
resize or eviction) when applying vertical recommendations. Whether Thoras
resizes a pod in place or has to recreate (evict) it to apply a recommendation,
these settings pace those changes in batches so the workload is not disrupted
all at once.
This setting applies only when vertical scaling is in `autonomous` mode. It has
no effect in `recommendation` mode. It governs both in-place resizes and
evictions (see [`update_policy`](#update_policy)) — an entire workload is never
resized in one shot. If `disruption` is omitted, Thoras applies the defaults
below.
**Configuration:**
* `interval` (string, default `60s`): The minimum time Thoras waits after a
batch of pod changes (resize or eviction) before starting the next batch. Must
be a positive duration.
* `max_disrupted` (integer or percentage, default `20%`): The maximum number of
pods that may be disrupted at once across the workload, counting both pods
resized in place and pods evicted in the batch. Accepts an absolute integer
(e.g. `2`) or a percentage of the workload's desired replicas (e.g. `"25%"`).
Must be a positive integer or a percentage between 1% and 100%.
**How it works:**
* Thoras applies changes to stale pods in batches — resizing in place where
possible, evicting otherwise. After issuing a batch it waits `interval` before
issuing the next one, so changes roll out gradually instead of all at once.
* A pod counts as "disrupted" if it is terminating or its `Ready` condition is
not `True`. Each batch is sized against the remaining headroom
(`max_disrupted` minus the pods already disrupted), so batches that haven't
finished rolling shrink the next batch.
* Unlike a `PodDisruptionBudget`'s `maxUnavailable`, this cap is enforced by
Thoras itself: pods that are unavailable for reasons unrelated to Thoras (e.g.
failing readiness probes) still consume the budget.
* If the Kubernetes API rate-limits Thoras's resize requests, Thoras stops the
current batch and resumes the remaining pods on a later run rather than
continuing to hammer the API.
* Thoras still honors your `PodDisruptionBudget`s. If an eviction is blocked by
a PDB, Thoras defers the remaining evictions to a later run rather than
forcing them.
## `additional_configuration`
### `scaling_behavior`
Controls scaling thresholds to prevent unnecessary scaling actions. Thresholds
can be configured independently for `scale_up` and `scale_down`. Both
**percentage** and **absolute** thresholds can be used for horizontal and
vertical scaling.
If `scaling_behavior` is not configured, defaults are applied automatically
based on your `model.mode`. Explicit configuration here overrides those
defaults. The default mode is `balanced`, so if no `model.mode` is set, the
5%/5% thresholds apply.
**Default thresholds by model mode:**
| Mode | Scale up threshold | Scale down threshold |
| - | - | - |
| `undershoot` | 20% | 5% |
| `cost_optimized` | 10% | 5% |
| `balanced` | 5% | 5% |
| `assurance_optimized` | 5% | 10% |
| `overshoot` | 5% | 20% |
Cost-leaning modes (`undershoot`, `cost_optimized`) have a higher scale-up
threshold and a lower scale-down threshold: they scale down readily and require
a larger change before scaling up. Assurance-leaning modes
(`assurance_optimized`, `overshoot`) do the opposite. See
[`model.mode`](#model-mode) for mode details.
**Percentage Thresholds** - Trigger scaling when change exceeds a percentage of
current usage (works for both horizontal and vertical):
```yaml theme={null}
horizontal:
scaling_behavior:
scale_up:
type: percent
percent: 20 # Scale up when increase is at least 20%
scale_down:
type: percent
percent: 15
vertical:
scaling_behavior:
scale_up:
type: percent
percent: 20 # Scale up when increase is at least 20%
scale_down:
type: percent
percent: 15
```
**Absolute Thresholds** - Trigger scaling when change exceeds fixed amounts
(works for both horizontal and vertical):
For horizontal scaling, use `pods`:
```yaml theme={null}
horizontal:
scaling_behavior:
scale_up:
type: absolute
absolute:
pods: "5" # Scale up only if adding at least 5 pods
scale_down:
type: absolute
absolute:
pods: "3" # Scale down only if removing at least 3 pods
```
For vertical scaling, use `memory` and `cpu`:
```yaml theme={null}
vertical:
scaling_behavior:
scale_up:
type: absolute
absolute:
memory: "100Mi" # Scale up only if memory increase is at least 100Mi
cpu: "500m" # Scale up only if CPU increase is at least 500m
scale_down:
type: absolute
absolute:
memory: "50Mi" # Scale down only if memory decrease is at least 50Mi
cpu: "250m" # Scale down only if CPU decrease is at least 250m
```
**Best Practices:**
* Use absolute thresholds for small workloads (percentage changes can be
misleading)
* Use percentage thresholds for large workloads (scales naturally with
deployment size)
* Set asymmetric thresholds (conservative scale-up, aggressive scale-down)
* If you are unsure where to start, rely on the mode defaults and only override
if you observe unwanted scaling behavior
### `additional_metrics` and `exclude_metrics`
```yaml ast.yaml theme={null}
horizontal:
mode: recommendation
additional_metrics:
- external:
metric:
name: {{EXTERNAL_METRIC_NAME}}
selector:
matchLabels:
app: {{APP_NAME}}
target:
averageValue: "1"
type: AverageValue
type: External
exclude_metrics:
- name: cpu
type: Resource
```
When you enable horizontal scaling in Thoras, it defaults to using the same
metrics as your HPA (Horizontal Pod Autoscaler) to guide predictive scaling,
whether they're resource, external or custom metrics. However, there may be
cases where you want Thoras to:
* **Exclude certain metrics** (e.g., CPU) from its predictions, even if they are
used by the HPA.
* **Add new metrics** for prediction that are not part of the HPA.
### Example 1: **Exclude CPU from Predictive Scaling**
This configuration tells Thoras **not to use CPU** for predictive scaling, even
if it's included in the HPA:
```yaml ast.yaml theme={null}
spec:
horizontal:
exclude_metrics:
- name: cpu
type: Resource
```
### Example 2: **Add Custom Metric for Predictive Scaling**
In this example, custom metric is configured to **predictively scale on CPU** by
explicitly including it in `additional_metrics`. Thoras will predictively scale
on CPU even though it is not defined in the HPA.
```yaml ast.yaml theme={null}
spec:
horizontal:
additional_metrics:
- external:
metric:
name: {{EXTERNAL_METRIC_NAME}}
selector:
matchLabels:
app: {{APP_NAME}}
target:
averageValue: "1"
type: AverageValue
type: External
```
# ClusterAIScaleTemplate
Source: https://docs.thoras.ai/reference/cluster-aiscale-template
ClusterAIScaleTemplate is a cluster-scoped custom resource that automatically creates and manages AIScaleTargets for matching workloads across multiple namespaces. This feature eliminates the need to manually create individual AIScaleTargets for each workload, enabling fleet-wide scaling policies.
## Overview
ClusterAIScaleTemplate watches for Deployments, StatefulSets, and Argo Rollouts
that match a label selector, and automatically creates corresponding
[AIScaleTargets](/reference/ast-definition) with a predefined configuration.
When workloads are added, updated, or removed, Thoras automatically manages the
lifecycle of the associated AIScaleTargets.
## Use Cases
* Apply AI-powered scaling to all workloads with a specific label (e.g.,
`thoras.ai/enabled: "true"`)
* Standardize scaling policies across multiple microservices
* Automatically enable AI scaling for new workloads as they're deployed
* Manage scaling configuration for entire application tiers or environments
## Basic Example
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
name: production-workloads
spec:
# Match workloads with specific labels
selector:
matchLabels:
env: production
thoras.ai/enabled: "true"
# Template to apply to matching workloads
template:
model:
forecast_interval: 1h
forecast_blocks: 4h
forecast_buffer_percentage: 10%
mode: balanced
vertical:
mode: auto
update_policy:
update_mode: in_place_or_recreate
containers:
- name: "*"
cpu:
lowerbound: "10m"
memory:
lowerbound: 0
```
This template will automatically create AIScaleTargets for all Deployments,
StatefulSets, and Argo Rollouts in any namespace that have the labels
`env: production` and `thoras.ai/enabled: "true"`.
## Specification
### ClusterAIScaleTemplateSpec
| Field | Type | Required | Description |
| - | - | - | - |
| `selector` | `metav1.LabelSelector` | Yes | Matches workloads to apply this template to. Supports `matchLabels` and `matchExpressions`. |
| `template` | `AIScaleTemplateSpec` | Yes | Defines the AIScaleTarget configuration to apply. |
| `policy` | `TemplatePolicy` | No | Controls template application behavior. |
### AIScaleTemplateSpec
The template spec accepts the full
[AIScaleTarget spec](/reference/ast-definition), minus the `scaleTargetRef`
(which is automatically populated based on the matched workload). Any field you
can set on an AIScaleTarget can be set on the template, including
`horizontal.multipleOf`, `horizontal.allowedReplicas`, `vertical.disruption`,
`vertical.update_policy`, `vertical.oom_remediation`, and per-metric mode
overrides.
Top-level fields:
| Field | Type | Required | Description |
| - | - | - | - |
| `model` | `ModelSpec` | Yes | Model configuration (mode, [forecast settings](/reference/forecasting), buffer percentage). |
| `vertical` | `VerticalSpec` | No | [Vertical scaling](/guides/vertical-pod-rightsizing) configuration. |
| `horizontal` | `HorizontalSpec` | No | [Horizontal scaling](/guides/hpa) configuration. |
See the [AIScaleTarget reference](/reference/ast-definition) for the
field-by-field detail of each sub-spec.
### TemplatePolicy
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `conflictResolution` | `string` | No | `skip` | How to handle conflicts with existing AIScaleTargets. Options: `skip`, `override`. |
| `includedNamespaces` | `[]string` | No | - | Allowlist of namespaces to target. Mutually exclusive with `excludedNamespaces`. |
| `excludedNamespaces` | `[]string` | No | - | Denylist of namespaces to skip. Mutually exclusive with `includedNamespaces`. |
## Behavior
### Automatic AIScaleTarget Creation
When a workload (Deployment, StatefulSet, or Argo Rollout) matches the
template's selector:
1. Thoras creates an AIScaleTarget with the name `-ast`
2. The AIScaleTarget is labeled with
`thoras.ai/managed-by-template: `
3. The AIScaleTarget has an owner reference to the workload (so it's deleted
when the workload is deleted)
4. The `scaleTargetRef` is automatically populated with the workload's details
### Lifecycle Management
* **Workload Created**: AIScaleTarget is automatically created
* **Workload Updated**: If labels change and no longer match, the AIScaleTarget
is deleted
* **Workload Opted Out**: If the workload is annotated with
`thoras.ai/skip-template: "true"`, the template stops managing it. Any
existing AIScaleTarget is retained, not deleted; see
[Opting a Workload Out](#opting-a-workload-out)
* **Workload Deleted**: AIScaleTarget is automatically deleted via owner
reference
* **Template Updated**: All managed AIScaleTargets are updated to match the new
template
* **Template Deleted**: All managed AIScaleTargets are automatically deleted
### Conflict Resolution
When reconciling a controller (e.g. Deployment, Rollout, etc.), Thoras checks
whether an AIScaleTarget already exists that targets the controller via
`scaleTargetRef` and applies the following `conflictResolution` policies:
* **`skip` (default)**: The existing AIScaleTarget is left unchanged. Changes
made to the template will not be reflected in the conflicting AIScaleTarget.
* **`override`**: The template takes ownership of the conflicting AIScaleTarget.
The template's configuration is applied to the AIScaleTarget.
Note: Thoras checks the `thoras.ai/managed-by-template` label to determine
ownership. Only AIScaleTargets created by that specific template will be updated
or deleted by it.
### Opting a Workload Out
A workload that matches a template's selector can still opt out of template
management by setting an annotation on the workload itself (Deployment,
StatefulSet, or Argo Rollout) — not on its pods:
```yaml theme={null}
metadata:
annotations:
thoras.ai/skip-template: "true"
```
The behavior depends on whether an AIScaleTarget already exists for the
workload:
* **No AIScaleTarget exists yet**: none is created while the annotation is
present.
* **An AIScaleTarget already exists**: it is left in place. The template stops
updating it, but its history, model state, and suggestions are retained.
A retained AIScaleTarget is not paused — it keeps scaling using its last
template-applied configuration. Opting out of a template stops the template
from managing the AIScaleTarget; it does not stop autoscaling. See [Stopping
Autonomous Scaling](/guides/stopping-autoscaling) if your goal is to halt
scaling actions.
Removing the annotation (or setting it to `"false"`) resumes template management
on the next reconcile, updating any retained AIScaleTarget in place.
The annotation value is parsed as a boolean: `"true"`, `"True"`, `"TRUE"`, and
`"1"` opt the workload out. `"false"`, `"0"`, an absent annotation, and empty or
unparseable values are all treated as not opted out.
### Namespace Filtering
Use `policy.includedNamespaces` or `policy.excludedNamespaces` to control which
namespaces the template targets. The two fields are mutually exclusive, set one
or the other, not both.
* **`includedNamespaces`** (allowlist): only the listed namespaces are in scope.
All other namespaces are ignored.
* **`excludedNamespaces`** (denylist): all namespaces are in scope except those
listed.
No namespaces are excluded by default. If you want to protect system namespaces,
add them explicitly:
```yaml theme={null}
policy:
excludedNamespaces:
- kube-node-lease
- kube-public
- kube-system
```
## Advanced Examples
### Using matchExpressions
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
name: microservices-tier
spec:
selector:
matchExpressions:
- key: tier
operator: In
values:
- frontend
- backend
- key: thoras.ai/enabled
operator: Exists
policy:
excludedNamespaces:
- kube-system
- default
- monitoring
template:
model:
forecast_blocks: 4h
forecast_buffer_percentage: 0%
forecast_interval: 1h
mode: balanced
horizontal:
mode: autonomous
```
### Restricting to Specific Namespaces
Use `includedNamespaces` to target only the namespaces you want:
```yaml theme={null}
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
name: production-only
spec:
selector:
matchLabels:
thoras.ai/enabled: "true"
policy:
includedNamespaces:
- production
- staging
template:
model:
forecast_blocks: 4h
forecast_buffer_percentage: 0%
forecast_interval: 1h
mode: balanced
vertical:
mode: auto
containers:
- name: "*"
cpu:
lowerbound: "10m"
memory:
lowerbound: 0
```
### Multiple Templates
You can create multiple templates with different selectors for different
workload categories:
```yaml theme={null}
---
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
name: critical-services
spec:
selector:
matchLabels:
priority: critical
template:
model:
forecast_blocks: 4h
forecast_buffer_percentage: 20%
forecast_interval: 1h
mode: balanced
vertical:
mode: auto
containers:
- name: "*"
cpu:
lowerbound: "500m"
memory:
lowerbound: "1Gi"
---
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
name: standard-services
spec:
selector:
matchExpressions:
- key: priority
operator: NotIn
values:
- critical
- key: thoras.ai/enabled
operator: Exists
template:
model:
forecast_blocks: 4h
forecast_buffer_percentage: 0%
forecast_interval: 1h
mode: balanced
vertical:
mode: auto
containers:
- name: "*"
cpu:
lowerbound: "0"
memory:
lowerbound: "0"
```
## Status and Monitoring
ClusterAIScaleTemplate includes a status subresource with conditions to track
reconciliation state:
```yaml theme={null}
status:
conditions:
- type: Ready
status: "True"
reason: ReconciliationSucceeded
message: "Successfully reconciled 15 namespaces"
lastTransitionTime: "2024-01-15T10:30:00Z"
observedGeneration: 2
```
### Condition Types
| Type | Status | Reason | Description |
| - | - | - | - |
| Ready | True | ReconciliationSucceeded | Template successfully reconciled all namespaces |
| Ready | False | ReconciliationFailed | Failed to reconcile one or more namespaces |
| Ready | False | NamespaceListFailed | Failed to list namespaces |
Check template status:
```bash theme={null}
kubectl get clusteraiscaletemplate production-workloads -o yaml
# or using short name
kubectl get clusterast production-workloads -o yaml
```
View all managed AIScaleTargets:
```bash theme={null}
kubectl get aiscaletarget -A -l thoras.ai/managed-by-template=production-workloads
```
## Best Practices
1. **Use Specific Selectors**: Use precise label selectors to avoid unintended
matches. Consider using a dedicated label like `thoras.ai/enabled: "true"` on
workloads you want to scale.
2. **Test in Non-Production First**: Create a template in a development
environment first to verify the selector matches the intended workloads.
3. **Monitor Template Status**: Regularly check the template's status conditions
to ensure successful reconciliation.
4. **Namespace Filtering**: No namespaces are excluded by default. Use
`policy.excludedNamespaces` to block system namespaces (e.g., `kube-system`,
`kube-public`, `kube-node-lease`) and any namespaces where automated scaling
could cause issues. Use `policy.includedNamespaces` when you want to restrict
the template to a known set of namespaces.
5. **Template Naming**: Use descriptive names that indicate the purpose or scope
(e.g., `production-frontend`, `staging-all-services`).
6. **Label Workloads Explicitly**: Rather than relying on existing labels, add a
specific label to workloads you want managed by templates (e.g.,
`thoras.ai/template: "production-workloads"`).
7. **Avoid Overlapping Templates**: Ensure template selectors don't overlap in
ways that would cause multiple templates to match the same workloads.
## Troubleshooting
### Template Not Creating AIScaleTargets
1. Check the template status:
```bash theme={null}
kubectl get clusterast -o yaml
```
2. Verify workloads match the selector:
```bash theme={null}
kubectl get deployments -A --show-labels
```
3. Check namespace filtering configuration:
```bash theme={null}
kubectl get clusterast -o jsonpath='{.spec.policy.includedNamespaces}'
kubectl get clusterast -o jsonpath='{.spec.policy.excludedNamespaces}'
```
4. View operator logs:
```bash theme={null}
kubectl logs -n thoras -l app=thoras-operator
```
### Workload Matches the Selector But Has No AIScaleTarget
If a workload matches the template's selector and namespace filtering, but no
AIScaleTarget is created (or an existing one stops being updated), check for the
opt-out annotation:
```bash theme={null}
kubectl get deployment -n -o jsonpath='{.metadata.annotations.thoras\.ai/skip-template}'
```
If this returns `true`, the workload has opted out of template management. See
[Opting a Workload Out](#opting-a-workload-out). Remove the annotation to resume
management.
### AIScaleTarget Not Updating When Template Changes
Ensure the AIScaleTarget has the `thoras.ai/managed-by-template` label:
```bash theme={null}
kubectl get ast -n -o jsonpath='{.metadata.labels}'
```
If the label is missing or points to a different template, the AIScaleTarget was
not created by this template and won't be updated by it.
### Deleting Template But AIScaleTargets Remain
Check if the finalizer is stuck:
```bash theme={null}
kubectl get clusterast -o jsonpath='{.metadata.finalizers}'
```
View operator logs for errors during cleanup:
```bash theme={null}
kubectl logs -n thoras -l app=thoras-operator | grep "cleaning up managed AIScaleTargets"
```
## Migration from Manual AIScaleTargets
To migrate existing manually created AIScaleTargets to template management:
1. Create a ClusterAIScaleTemplate with the desired configuration
2. Add matching labels to your workloads
3. The template will skip existing AIScaleTargets (due to default `skip`
conflict resolution)
4. Manually delete existing AIScaleTargets - they will be automatically
recreated by the template
5. Verify new AIScaleTargets have the `thoras.ai/managed-by-template` label
Alternatively, manually add the label to existing AIScaleTargets:
```bash theme={null}
kubectl label ast -n thoras.ai/managed-by-template=
```
**Warning**: Only do this if you're certain the template configuration matches
your existing AIScaleTarget configuration, as the template will overwrite the
spec on the next reconciliation.
## Related Documentation
* [AIScaleTarget Definition](/reference/ast-definition) - Complete reference for
AIScaleTarget specifications
* [Configuring AIScaleTargets](/guides/configuring-asts) - Step-by-step guide
for creating AIScaleTargets
* [Forecasting](/reference/forecasting) - Understanding how Thoras forecasts
resource needs
* [Predictive Horizontal Pod Autoscaling](/guides/hpa) - Guide to horizontal
scaling configuration
* [Predictive Vertical Pod Rightsizing](/guides/vertical-pod-rightsizing) -
Guide to vertical scaling configuration
* [Enabling Autonomous Mode](/guides/enabling-autonomous-mode) - Prerequisites
for autonomous scaling
* [Pod Annotations](/reference/pod-annotations) - The `thoras.ai/managed` family
of annotations the webhook writes to mutated pods, distinct from the
`thoras.ai/managed-by-template` label on this page
* [Stopping Autonomous Scaling](/guides/stopping-autoscaling) - Options for
halting Thoras autoscaling, from cluster-wide to per-workload
# Forecasting
Source: https://docs.thoras.ai/reference/forecasting
Thoras forecasts future resource usage and scaling needs based on historical
patterns. This reference documents how forecasting works and the key concepts
involved.
## How Forecasting Works
Thoras continuously collects workload metrics and uses those timeseries metrics
to generate forecasts. When a forecast is generated, Thoras predicts future
resource usage within the forecast block. When a target is in autonomous mode
and the delta between forecasted and current usage exceeds configured
thresholds, the target is proactively scaled to the predicted maximum within the
forecast block, plus a small buffer.
Thoras generates forecasts for two scaling dimensions:
* **Horizontal** - Predicts pod replica count based on forecasted CPU, memory,
and custom metrics. See [Predictive Horizontal Pod Autoscaling](/guides/hpa)
for details.
* **Vertical** - Predicts container resource requests based on historical
consumption patterns. See
[Vertical Pod Rightsizing](/guides/vertical-pod-rightsizing) for details.
## Forecast Configuration
Forecasting behavior is configured in the `model` section of an `AIScaleTarget`.
See the [AIScaleTarget Definition](/reference/ast-definition#model) for
configuration options including:
* [`forecast_blocks`](/reference/ast-definition#modelforecast_blocks) - the time
horizon for each forecast; Thoras sizes for the predicted peak within it
* [`forecast_buffer_percentage`](/reference/ast-definition#modelforecast_buffer_percentage) -
safety margin on top of forecasted values
* [`forecast_interval`](/reference/ast-definition#modelforecast_interval) - how
frequently forecasts are generated (duration string, e.g. `15m`, `1h`)
## Model Training
Models continuously adapt to a workload's usage patterns, becoming better tuned
for a workload over time.
### Training Period
* **Newly Enrolled AiScaleTarget:** Models begin learning a workload's usage
patterns after collecting initial workload metrics. Initial forecasts are
based on a small sample size of workload usage data, and will improve as the
data sample size increases. AiScaleTargets require 24 hours of metrics by
default before being eligible for autonomous scaling. This is controlled by
the Helm value `thorasForecast.minLookbackToScale` and can be overridden
per-AST via `model.min_lookback_to_scale`. The hard minimum is 3 hours.
* **Minimum:** Allow at least 24 hours in `recommendation` mode before switching
to `autonomous` mode.
* **Optimal:** 2 weeks of data provides the best accuracy, especially for
workloads with weekly patterns.
### Data Requirements
For accurate forecasts, Thoras needs:
* Consistent metric collection from workloads
* Representative traffic patterns (including peak and off-peak periods)
## Container Startup Resource Spikes
Software often experiences temporary spikes in CPU and memory usage during
initialization. These spikes can significantly exceed the container's
steady-state operating resource consumption.
### How Forecasting Handles This
Newly started containers are automatically detected. Metrics from newly started
containers are excluded from forecasts by default, giving the container time to
reach steady-state operation.
The forecaster then trains on adjusted metrics that represent your workload's
actual operational resource usage, resulting in:
* More accurate resource predictions and reduced thrashing
* Right-sized resource allocations
* Reduced infrastructure costs
### Viewing Startup Spikes
In the dashboard, you can toggle "Show restarts" on the Actual vs Predicted
Usage chart to see the difference between:
* **Adjusted usage** (default) - Resource usage with startup spikes filtered
out. This adjusted data is what the forecaster uses to make predictions.
* **Actual usage** - Raw resource usage including all startup spikes.
This helps visualize how forecasting excludes startup spikes.
### Opting Out Of Excluding Startup Metrics From Forecasts
You may opt-out of excluding metrics for newly started containers from
forecasting. Note that opting out may increase the volatility of both usage and
forecasts, especially for low-usage workloads.
To include startup metrics in forecast model training, set `ignoreNewPods` to
`false` in your Helm values:
```yaml theme={null}
forecaster:
# true (default): exclude metrics for newly started containers from model training.
# false: include metrics for newly started containers in model training.
ignoreNewPods: false
```
# Pod Annotations
Source: https://docs.thoras.ai/reference/pod-annotations
Reference for the thoras.ai/ annotations Thoras writes to pods, and what they do and don't guarantee.
## Overview
When the vertical scaling admission webhook mutates a pod's resource requests
(or limits) in `autonomous` mode, it annotates the pod:
```yaml theme={null}
metadata:
annotations:
thoras.ai/managed: "true"
thoras.ai/scaler: "prod/api-server"
thoras.ai/admitted-at: "2026-08-19T14:02:11Z"
```
| Annotation | Description |
| - | - |
| `thoras.ai/managed` | `"true"` if the webhook wrote at least one resource request or limit on this pod. |
| `thoras.ai/scaler` | `/` of the `AIScaleTarget` or `DaemonSetAutoscaler` that drove the mutation. |
| `thoras.ai/admitted-at` | RFC 3339 timestamp of when the webhook mutated the pod, set once at admission. |
These are written once, at pod admission (`CREATE`), by the mutating webhook
only — not by [in-place resize](/guides/vertical-pod-rightsizing#update-policy),
which patches running pods directly without going through admission.
This is scoped to the pod-mutating webhook, which covers both vertical
`AIScaleTarget` rightsizing and initial resource seeding by a
`DaemonSetAutoscaler`. Horizontal scaling (replica count), which mutates the
`*/scale` subresource instead of the pod, is a separate webhook and does not
set these annotations.
## What This Answers
`thoras.ai/managed: "true"` means **"the webhook set this pod's resources at
admission,"** and nothing stronger. Use it to answer: did Thoras's mutating
webhook write to this pod's `resources`, as opposed to the controller's pod
template, another mutating webhook, or a manual `kubectl edit`.
It does **not** mean "Thoras currently manages this pod." Two situations produce
a pod that is annotated `managed: "true"` but is no longer representative of
Thoras's current recommendation:
### Pods Predate Autonomous Mode
An `AIScaleTarget` switched into `autonomous` mode only annotates pods that are
admitted (created) after the switch. Pods that were already running keep being
resized in place by the disruption worker and are never re-admitted, so they
stay unannotated until they're naturally recreated (a rollout, eviction, node
drain, or manual restart). Once a pod turns over, it goes through the webhook
and picks up the annotation on its next admission.
### The Annotation Isn't Cleared When Thoras Stops Managing a Pod
Switching an `AIScaleTarget` out of `autonomous` mode, or unenrolling it (see
[Stopping Autonomous Scaling](/guides/stopping-autoscaling)), reverts running
pods to their controller's template values with an in-place resize wherever
possible — the pods are not recreated, so they are never re-admitted and the
webhook never runs again to clear the annotation. A reverted pod keeps
`thoras.ai/managed: "true"` while sitting at template values. As with the case
above, this self-heals the next time the pod is recreated for any reason: a
fresh admission with nothing to mutate does not set the annotation.
When the in-place revert can't apply (for example, the template declares a
different set of resource keys than the running pod, which would change its QoS
class) and the update policy permits recreation, Thoras falls back to restarting
the workload — those pods are recreated and come back without the annotation.
Both cases are false positives on a pod Thoras did mutate at some earlier
admission, consistent with the annotation's actual meaning — never false
positives claiming Thoras is currently managing a pod it isn't.
## Related Documentation
* [Predictive Vertical Pod Rightsizing](/guides/vertical-pod-rightsizing) -
Configuring `autonomous` mode and update policy
* [Stopping Autonomous Scaling](/guides/stopping-autoscaling) - How pods revert
when autonomous mode is turned off
* [ClusterAIScaleTemplate](/reference/cluster-aiscale-template) - The
`thoras.ai/skip-template` and `thoras.ai/managed-by-template` annotations and
labels, which control template enrollment rather than pod mutation