Skip to main content
AWS Cost Explorer gives Thoras your cluster’s bill. It can’t say what each workload cost, because Cost Explorer has no pod-level data. The AWS Cost and Usage Report (CUR) does: with split cost allocation data turned on, AWS splits each node’s cost between the pods that ran on it. When you turn on CUR ingestion, the Thoras worker reads that report and stores each workload’s billed cost by hour and by day. The dashboard’s Billed cost tab then shows what AWS billed each AIScaleTarget’s pods, for any namespace or AIScaleTarget you pick. Ingestion is off by default.
Requires chart version 5.12.0 (platform 4.132.0). The cost and usage report is AWS billing data, so it applies to EKS clusters.

Before you start

In the AWS account that pays for the cluster, usually your organization’s management (payer) account:
  1. Turn on split cost allocation data for Amazon EKS in the Billing and Cost Management console, under Cost Management preferences -> Split cost allocation data.
  2. In Data Exports, create a standard data export of type CUR 2.0, with these settings:
    • Include resource IDs: selected
    • Split cost allocation data: selected
    • Time granularity: Hourly
    • Compression type and file format: Parquet
    • File versioning: Overwrite existing data export file
Note the export’s S3 bucket, S3 path prefix, and Export name. Thoras reads the export from s3://<bucket>/<prefix>/<export name>/.
AWS doesn’t backfill. Split cost data starts with the month you turn it on, and the export starts when you create it, so billed cost per AIScaleTarget covers only the time since then. AWS refreshes the export about once a day, and it can take up to 24 hours after you turn split cost allocation data on before pod costs appear.

Grant the worker read access

The Thoras worker reads the export with the AWS credentials its pod picks up. On EKS, bind an IAM role to the worker’s service account with IRSA, as in Grant access to Cost Explorer. If you already collect Cost Explorer data, add these permissions to that role. The worker needs s3:ListBucket on the bucket and s3:GetObject on the export. Without s3:ListBucket, S3 reports a missing file as access denied, so Thoras can’t tell the two apart. If the export is in the cluster’s own account, the worker’s role needs only this policy:
If the export is in another account, such as your payer account, give access in one of two ways.

Through a bucket policy

Give the worker’s role the policy above, and add a statement to the bucket’s policy in the payer account that grants the same access to that role:
Cross-account access needs both the bucket policy and the permission on the worker’s own role.

Through a role in the payer account

Create a role in the payer account with the S3 policy above, and let the worker’s role assume it:
Then allow the worker’s role to assume it:
Set its ARN as curRoleArn below. If its trust policy requires an external ID, set curRoleExternalId too.
If the bucket is encrypted with a KMS key (SSE-KMS), the role that reads it also needs kms:Decrypt on that key.

Enable ingestion

Turn ingestion on, point the worker at the export, and show the Billed cost tab in the dashboard:
Then helm upgrade your release. The worker reads the report when it starts and every 6 hours after that. An organization’s export covers every account and cluster it pays for. Thoras keeps only the rows for this cluster and for the account of the worker’s own IAM role, even when it reads through curRoleArn. Keep that role in the cluster’s account.

With a network policy

If you turn on networkPolicy.enabled, the worker needs HTTPS egress to S3 and STS. Its policy already allows HTTPS when either is true:
  • The worker also collects Cost Explorer data or syncs with a console. It can then reach thorasWorker.httpsEgressCidrs, any address by default.
  • The policy uses the kubernetes flavor and networkPolicy.apiServerPorts includes 443, as it does by default. That rule allows its ports to any address.
Otherwise, add a rule. With the cilium flavor:
With the kubernetes flavor:

What you see

Open Cost analysis -> Billed cost and pick a namespace or AIScaleTarget with Edit scope:
  • The graph shows Billed (CUR): what AWS billed the selected AIScaleTargets’ pods each day or hour. Each pod’s cost is its share of its node’s cost, as AWS splits it in the report. The hourly view is available for a scope even if you don’t collect hourly Cost Explorer data.
  • Days or hours with no cost in the report stay on the graph and are marked as such, for example before the report’s history starts or while a workload is scaled to zero.
  • Billed cost by AIScaleTarget lists each AIScaleTarget in the scope, with its workloads and its billed cost for the selected days, and a total. Sort it by billed cost or by name. An AIScaleTarget with nothing billed shows ”—”.
With no scope, the graph shows the cluster’s bill from Cost Explorer, and the table lists every AIScaleTarget. The AIScaleTargets add up to less than the cluster’s bill, which also covers idle capacity, workloads without an AIScaleTarget, storage, and networking. Until the report has been read, a scope’s cost is estimated from its share of the cluster’s bill by requested CPU and memory, and the tab says so. Billed cost per AIScaleTarget is shown in the cluster’s dashboard. It isn’t sent to the console.

How costs are matched to AIScaleTargets

Thoras stores the report’s cost per workload and matches it to your AIScaleTargets when you view it:
  • An AIScaleTarget for a Deployment or StatefulSet matches that workload by name, in the AIScaleTarget’s namespace.
  • An AIScaleTarget for an Argo Rollout matches the Rollout’s ReplicaSets.
  • An AIScaleTarget with a label selector matches the controllers it selects, in its namespace.
The match uses the workload’s whole history in the report, including from before the AIScaleTarget existed. A workload covered by several AIScaleTargets counts once in the total.

What Thoras stores

  • Each workload’s billed cost by hour for 14 days, and by day for 13 months.
  • Only allocated cost: what AWS charged to pods. Unused node capacity isn’t attributed.
  • The worker reads up to 14 months back. Each run reads at most 3 months that AWS has changed since the last read, newest first, so a long history fills in over the first few runs. AWS keeps updating a month for a while after it ends, and Thoras reads it again whenever it does.

Verify it’s working

Check the worker logs:
After each month it reads, the worker logs Ingested a cost and usage report period, with the month and the number of rows it stored:
Then open the Billed cost tab with a scope. Once the report has been read, the graph shows Billed (CUR) and the Billed cost by AIScaleTarget table lists your AIScaleTargets.

Troubleshooting

The Billed cost by AIScaleTarget section explains what is wrong when it can’t show billed cost. “Billed cost per AIScaleTarget needs the AWS cost and usage report…” — ingestion isn’t turned on, or hasn’t run yet. Check the values in Enable ingestion. “Reading the cost and usage report for … failed on …” — the banner ends with the reason:
  • no cost and usage report manifest found for the last 14 billing periods — the worker found no export at s3://<curBucket>/<curPrefix>/<curExportName>/. Check the three values against the export, and that its File versioning is Overwrite existing data export file. If the message ends with access denied, check the permissions in Grant the worker read access, including s3:ListBucket and the bucket policy or the role’s trust policy.
  • cost and usage report access denied — the worker found the export but can’t read its files. Check s3:GetObject on the export, and kms:Decrypt if the bucket uses a KMS key.
  • cost and usage report export is not usable: … — the export’s settings don’t match Before you start. The message names what is missing, such as split cost allocation data, resource IDs, the Parquet format, or hourly granularity. Change the export’s settings; AWS applies them from the next delivery.
“The cost and usage report has not been read yet…” — the worker has tried but hasn’t read a month yet. Check the worker logs for the reason. “The cost and usage report for … has no pods for this cluster…” — the report was read, but it has no pod costs for this account and cluster. Check that split cost allocation data is on for EKS, that the worker’s own IAM role is in the cluster’s account, and that the worker resolves the right cluster name. Set thorasWorker.curClusterName to your EKS cluster’s name if it doesn’t. While this shows, a scope’s cost is the estimate. “The cost and usage report was last read on …, so recent days may be missing.” — the worker hasn’t read the report for more than two days. Check that the worker is running and look at its logs. “The cost and usage report’s status could not be loaded…” — the dashboard couldn’t reach the API. Reload the page. Cost and usage report ingestion cannot run until its configuration or the export changes in the worker logs — the worker stopped trying until something changes. The log’s error says what, such as a cluster name it can’t resolve. Set thorasWorker.curClusterName to your EKS cluster’s name.