> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thoras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ClusterAIScaleTemplate

> ClusterAIScaleTemplate is a cluster-scoped custom resource that automatically creates and manages AIScaleTargets for matching workloads across multiple namespaces. This feature eliminates the need to manually create individual AIScaleTargets for each workload, enabling fleet-wide scaling policies.

## Overview

ClusterAIScaleTemplate watches for Deployments, StatefulSets, and Argo Rollouts
that match a label selector, and automatically creates corresponding
[AIScaleTargets](/reference/ast-definition) with a predefined configuration.
When workloads are added, updated, or removed, Thoras automatically manages the
lifecycle of the associated AIScaleTargets.

## Use Cases

* Apply AI-powered scaling to all workloads with a specific label (e.g.,
  `thoras.ai/enabled: "true"`)
* Standardize scaling policies across multiple microservices
* Automatically enable AI scaling for new workloads as they're deployed
* Manage scaling configuration for entire application tiers or environments

## Basic Example

```yaml theme={null}
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
  name: production-workloads
spec:
  # Match workloads with specific labels
  selector:
    matchLabels:
      env: production
      thoras.ai/enabled: "true"

  # Template to apply to matching workloads
  template:
    model:
      forecast_interval: 1h
      forecast_blocks: 4h
      forecast_buffer_percentage: 10%
      mode: balanced

    vertical:
      mode: auto
      update_policy:
        update_mode: in_place_or_recreate
      containers:
        - name: "*"
          cpu:
            lowerbound: "10m"
          memory:
            lowerbound: 0
```

This template will automatically create AIScaleTargets for all Deployments,
StatefulSets, and Argo Rollouts in any namespace that have the labels
`env: production` and `thoras.ai/enabled: "true"`.

## Specification

### ClusterAIScaleTemplateSpec

| Field      | Type                   | Required | Description                                                                                 |
| ---------- | ---------------------- | -------- | ------------------------------------------------------------------------------------------- |
| `selector` | `metav1.LabelSelector` | Yes      | Matches workloads to apply this template to. Supports `matchLabels` and `matchExpressions`. |
| `template` | `AIScaleTemplateSpec`  | Yes      | Defines the AIScaleTarget configuration to apply.                                           |
| `policy`   | `TemplatePolicy`       | No       | Controls template application behavior.                                                     |

### AIScaleTemplateSpec

The template spec contains the same fields as
[AIScaleTarget spec](/reference/ast-definition), minus the `scaleTargetRef`
(which is automatically populated based on the matched workload):

| Field        | Type             | Required | Description                                                                                 |
| ------------ | ---------------- | -------- | ------------------------------------------------------------------------------------------- |
| `model`      | `ModelSpec`      | Yes      | Model configuration (mode, [forecast settings](/reference/forecasting), buffer percentage). |
| `vertical`   | `VerticalSpec`   | No       | [Vertical scaling](/guides/vertical-pod-rightsizing) configuration.                         |
| `horizontal` | `HorizontalSpec` | No       | [Horizontal scaling](/guides/hpa) configuration.                                            |

### TemplatePolicy

| Field                | Type       | Required | Default | Description                                                                        |
| -------------------- | ---------- | -------- | ------- | ---------------------------------------------------------------------------------- |
| `conflictResolution` | `string`   | No       | `skip`  | How to handle conflicts with existing AIScaleTargets. Options: `skip`, `override`. |
| `includedNamespaces` | `[]string` | No       | -       | Allowlist of namespaces to target. Mutually exclusive with `excludedNamespaces`.   |
| `excludedNamespaces` | `[]string` | No       | -       | Denylist of namespaces to skip. Mutually exclusive with `includedNamespaces`.      |

## Behavior

### Automatic AIScaleTarget Creation

When a workload (Deployment, StatefulSet, or Argo Rollout) matches the
template's selector:

1. Thoras creates an AIScaleTarget with the name `<workload-name>-ast`
2. The AIScaleTarget is labeled with
   `thoras.ai/managed-by-template: <template-name>`
3. The AIScaleTarget has an owner reference to the workload (so it's deleted
   when the workload is deleted)
4. The `scaleTargetRef` is automatically populated with the workload's details

### Lifecycle Management

* **Workload Created**: AIScaleTarget is automatically created
* **Workload Updated**: If labels change and no longer match, the AIScaleTarget
  is deleted
* **Workload Deleted**: AIScaleTarget is automatically deleted via owner
  reference
* **Template Updated**: All managed AIScaleTargets are updated to match the new
  template
* **Template Deleted**: All managed AIScaleTargets are automatically deleted

### Conflict Resolution

When reconciling a controller (e.g. Deployment, Rollout, etc.), Thoras checks
whether an AIScaleTarget already exists that targets the controller via
`scaleTargetRef` and applies the following `conflictResolution` policies:

* **`skip` (default)**: The existing AIScaleTarget is left unchanged. Changes
  made to the template will not be reflected in the conflicting AIScaleTarget.
* **`override`**: The template takes ownership of the conflicting AIScaleTarget.
  The template's configuration is applied to the AIScaleTarget.

Note: Thoras checks the `thoras.ai/managed-by-template` label to determine
ownership. Only AIScaleTargets created by that specific template will be updated
or deleted by it.

### Namespace Filtering

Use `policy.includedNamespaces` or `policy.excludedNamespaces` to control which
namespaces the template targets. The two fields are mutually exclusive, set one
or the other, not both.

* **`includedNamespaces`** (allowlist): only the listed namespaces are in scope.
  All other namespaces are ignored.
* **`excludedNamespaces`** (denylist): all namespaces are in scope except those
  listed.

No namespaces are excluded by default. If you want to protect system namespaces,
add them explicitly:

```yaml theme={null}
policy:
  excludedNamespaces:
    - kube-node-lease
    - kube-public
    - kube-system
```

## Advanced Examples

### Using matchExpressions

```yaml theme={null}
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
  name: microservices-tier
spec:
  selector:
    matchExpressions:
      - key: tier
        operator: In
        values:
          - frontend
          - backend
      - key: thoras.ai/enabled
        operator: Exists

  policy:
    excludedNamespaces:
      - kube-system
      - default
      - monitoring

  template:
    model:
      forecast_blocks: 4h
      forecast_buffer_percentage: 0%
      forecast_interval: 1h
      mode: balanced
    horizontal:
      mode: autonomous
```

### Restricting to Specific Namespaces

Use `includedNamespaces` to target only the namespaces you want:

```yaml theme={null}
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
  name: production-only
spec:
  selector:
    matchLabels:
      thoras.ai/enabled: "true"
  policy:
    includedNamespaces:
      - production
      - staging
  template:
    model:
      forecast_blocks: 4h
      forecast_buffer_percentage: 0%
      forecast_interval: 1h
      mode: balanced
    vertical:
      mode: auto
      containers:
        - name: "*"
          cpu:
            lowerbound: "10m"
          memory:
            lowerbound: 0
```

### Multiple Templates

You can create multiple templates with different selectors for different
workload categories:

```yaml theme={null}
---
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
  name: critical-services
spec:
  selector:
    matchLabels:
      priority: critical
  template:
    model:
      forecast_blocks: 4h
      forecast_buffer_percentage: 20%
      forecast_interval: 1h
      mode: balanced
    vertical:
      mode: auto
      containers:
        - name: "*"
          cpu:
            lowerbound: "500m"
          memory:
            lowerbound: "1Gi"

---
apiVersion: thoras.ai/v1
kind: ClusterAIScaleTemplate
metadata:
  name: standard-services
spec:
  selector:
    matchExpressions:
      - key: priority
        operator: NotIn
        values:
          - critical
      - key: thoras.ai/enabled
        operator: Exists
  template:
    model:
      forecast_blocks: 4h
      forecast_buffer_percentage: 0%
      forecast_interval: 1h
      mode: balanced
    vertical:
      mode: auto
      containers:
        - name: "*"
          cpu:
            lowerbound: "0"
          memory:
            lowerbound: "0"
```

## Status and Monitoring

ClusterAIScaleTemplate includes a status subresource with conditions to track
reconciliation state:

```yaml theme={null}
status:
  conditions:
    - type: Ready
      status: "True"
      reason: ReconciliationSucceeded
      message: "Successfully reconciled 15 namespaces"
      lastTransitionTime: "2024-01-15T10:30:00Z"
      observedGeneration: 2
```

### Condition Types

| Type  | Status | Reason                  | Description                                     |
| ----- | ------ | ----------------------- | ----------------------------------------------- |
| Ready | True   | ReconciliationSucceeded | Template successfully reconciled all namespaces |
| Ready | False  | ReconciliationFailed    | Failed to reconcile one or more namespaces      |
| Ready | False  | NamespaceListFailed     | Failed to list namespaces                       |

Check template status:

```bash theme={null}
kubectl get clusteraiscaletemplate production-workloads -o yaml
# or using short name
kubectl get clusterast production-workloads -o yaml
```

View all managed AIScaleTargets:

```bash theme={null}
kubectl get aiscaletarget -A -l thoras.ai/managed-by-template=production-workloads
```

## Best Practices

1. **Use Specific Selectors**: Use precise label selectors to avoid unintended
   matches. Consider using a dedicated label like `thoras.ai/enabled: "true"` on
   workloads you want to scale.

2. **Test in Non-Production First**: Create a template in a development
   environment first to verify the selector matches the intended workloads.

3. **Monitor Template Status**: Regularly check the template's status conditions
   to ensure successful reconciliation.

4. **Namespace Filtering**: No namespaces are excluded by default. Use
   `policy.excludedNamespaces` to block system namespaces (e.g., `kube-system`,
   `kube-public`, `kube-node-lease`) and any namespaces where automated scaling
   could cause issues. Use `policy.includedNamespaces` when you want to restrict
   the template to a known set of namespaces.

5. **Template Naming**: Use descriptive names that indicate the purpose or scope
   (e.g., `production-frontend`, `staging-all-services`).

6. **Label Workloads Explicitly**: Rather than relying on existing labels, add a
   specific label to workloads you want managed by templates (e.g.,
   `thoras.ai/template: "production-workloads"`).

7. **Avoid Overlapping Templates**: Ensure template selectors don't overlap in
   ways that would cause multiple templates to match the same workloads.

## Troubleshooting

### Template Not Creating AIScaleTargets

1. Check the template status:

   ```bash theme={null}
   kubectl get clusterast <template-name> -o yaml
   ```

2. Verify workloads match the selector:

   ```bash theme={null}
   kubectl get deployments -A --show-labels
   ```

3. Check namespace filtering configuration:

   ```bash theme={null}
   kubectl get clusterast <template-name> -o jsonpath='{.spec.policy.includedNamespaces}'
   kubectl get clusterast <template-name> -o jsonpath='{.spec.policy.excludedNamespaces}'
   ```

4. View operator logs:
   ```bash theme={null}
   kubectl logs -n thoras -l app=thoras-operator
   ```

### AIScaleTarget Not Updating When Template Changes

Ensure the AIScaleTarget has the `thoras.ai/managed-by-template` label:

```bash theme={null}
kubectl get ast <ast-name> -n <namespace> -o jsonpath='{.metadata.labels}'
```

If the label is missing or points to a different template, the AIScaleTarget was
not created by this template and won't be updated by it.

### Deleting Template But AIScaleTargets Remain

Check if the finalizer is stuck:

```bash theme={null}
kubectl get clusterast <template-name> -o jsonpath='{.metadata.finalizers}'
```

View operator logs for errors during cleanup:

```bash theme={null}
kubectl logs -n thoras -l app=thoras-operator | grep "cleaning up managed AIScaleTargets"
```

## Migration from Manual AIScaleTargets

To migrate existing manually created AIScaleTargets to template management:

1. Create a ClusterAIScaleTemplate with the desired configuration
2. Add matching labels to your workloads
3. The template will skip existing AIScaleTargets (due to default `skip`
   conflict resolution)
4. Manually delete existing AIScaleTargets - they will be automatically
   recreated by the template
5. Verify new AIScaleTargets have the `thoras.ai/managed-by-template` label

Alternatively, manually add the label to existing AIScaleTargets:

```bash theme={null}
kubectl label ast <ast-name> -n <namespace> thoras.ai/managed-by-template=<template-name>
```

**Warning**: Only do this if you're certain the template configuration matches
your existing AIScaleTarget configuration, as the template will overwrite the
spec on the next reconciliation.

## Related Documentation

* [AIScaleTarget Definition](/reference/ast-definition) - Complete reference for
  AIScaleTarget specifications
* [Configuring AIScaleTargets](/guides/configuring-asts) - Step-by-step guide
  for creating AIScaleTargets
* [Forecasting](/reference/forecasting) - Understanding how Thoras forecasts
  resource needs
* [Predictive Horizontal Pod Autoscaling](/guides/hpa) - Guide to horizontal
  scaling configuration
* [Predictive Vertical Pod Rightsizing](/guides/vertical-pod-rightsizing) -
  Guide to vertical scaling configuration
* [Enabling Autonomous Mode](/guides/enabling-autonomous-mode) - Prerequisites
  for autonomous scaling