Requires chart version
5.12.0 (platform 4.132.0). The cost and usage
report is AWS billing data, so it applies to EKS clusters.Before you start
In the AWS account that pays for the cluster, usually your organization’s management (payer) account:- Turn on split cost allocation data for Amazon EKS in the Billing and Cost Management console, under Cost Management preferences -> Split cost allocation data.
- In Data Exports, create a
standard data export
of type CUR 2.0, with these settings:
- Include resource IDs: selected
- Split cost allocation data: selected
- Time granularity: Hourly
- Compression type and file format: Parquet
- File versioning: Overwrite existing data export file
s3://<bucket>/<prefix>/<export name>/.
Grant the worker read access
The Thoras worker reads the export with the AWS credentials its pod picks up. On EKS, bind an IAM role to the worker’s service account with IRSA, as in Grant access to Cost Explorer. If you already collect Cost Explorer data, add these permissions to that role. The worker needss3:ListBucket on the bucket and s3:GetObject on the export.
Without s3:ListBucket, S3 reports a missing file as access denied, so Thoras
can’t tell the two apart.
If the export is in the cluster’s own account, the worker’s role needs only this
policy:
Through a bucket policy
Give the worker’s role the policy above, and add a statement to the bucket’s policy in the payer account that grants the same access to that role:Through a role in the payer account
Create a role in the payer account with the S3 policy above, and let the worker’s role assume it:curRoleArn below. If its trust policy requires an external ID,
set curRoleExternalId too.
If the bucket is encrypted with a KMS key (SSE-KMS), the role that reads it
also needs
kms:Decrypt on that key.Enable ingestion
Turn ingestion on, point the worker at the export, and show the Billed cost tab in the dashboard:helm upgrade your release. The worker reads the report when it starts and
every 6 hours after that.
An organization’s export covers every account and cluster it pays for. Thoras
keeps only the rows for this cluster and for the account of the worker’s own IAM
role, even when it reads through
curRoleArn. Keep that role in the cluster’s
account.
With a network policy
If you turn onnetworkPolicy.enabled, the worker needs HTTPS egress to S3 and
STS. Its policy already allows HTTPS when either is true:
- The worker also collects Cost Explorer data or syncs with a console. It can
then reach
thorasWorker.httpsEgressCidrs, any address by default. - The policy uses the
kubernetesflavor andnetworkPolicy.apiServerPortsincludes443, as it does by default. That rule allows its ports to any address.
cilium flavor:
kubernetes flavor:
What you see
Open Cost analysis -> Billed cost and pick a namespace or AIScaleTarget with Edit scope:- The graph shows Billed (CUR): what AWS billed the selected AIScaleTargets’ pods each day or hour. Each pod’s cost is its share of its node’s cost, as AWS splits it in the report. The hourly view is available for a scope even if you don’t collect hourly Cost Explorer data.
- Days or hours with no cost in the report stay on the graph and are marked as such, for example before the report’s history starts or while a workload is scaled to zero.
- Billed cost by AIScaleTarget lists each AIScaleTarget in the scope, with its workloads and its billed cost for the selected days, and a total. Sort it by billed cost or by name. An AIScaleTarget with nothing billed shows ”—”.
How costs are matched to AIScaleTargets
Thoras stores the report’s cost per workload and matches it to your AIScaleTargets when you view it:- An AIScaleTarget for a Deployment or StatefulSet matches that workload by name, in the AIScaleTarget’s namespace.
- An AIScaleTarget for an Argo Rollout matches the Rollout’s ReplicaSets.
- An AIScaleTarget with a label selector matches the controllers it selects, in its namespace.
What Thoras stores
- Each workload’s billed cost by hour for 14 days, and by day for 13 months.
- Only allocated cost: what AWS charged to pods. Unused node capacity isn’t attributed.
- The worker reads up to 14 months back. Each run reads at most 3 months that AWS has changed since the last read, newest first, so a long history fills in over the first few runs. AWS keeps updating a month for a while after it ends, and Thoras reads it again whenever it does.
Verify it’s working
Check the worker logs:Ingested a cost and usage report period, with the month and the number of rows
it stored:
Troubleshooting
The Billed cost by AIScaleTarget section explains what is wrong when it can’t show billed cost. “Billed cost per AIScaleTarget needs the AWS cost and usage report…” — ingestion isn’t turned on, or hasn’t run yet. Check the values in Enable ingestion. “Reading the cost and usage report for … failed on …” — the banner ends with the reason:no cost and usage report manifest found for the last 14 billing periods— the worker found no export ats3://<curBucket>/<curPrefix>/<curExportName>/. Check the three values against the export, and that its File versioning is Overwrite existing data export file. If the message ends withaccess denied, check the permissions in Grant the worker read access, includings3:ListBucketand the bucket policy or the role’s trust policy.cost and usage report access denied— the worker found the export but can’t read its files. Checks3:GetObjecton the export, andkms:Decryptif the bucket uses a KMS key.cost and usage report export is not usable: …— the export’s settings don’t match Before you start. The message names what is missing, such as split cost allocation data, resource IDs, the Parquet format, or hourly granularity. Change the export’s settings; AWS applies them from the next delivery.
thorasWorker.curClusterName to your EKS cluster’s name if it doesn’t. While
this shows, a scope’s cost is the estimate.
“The cost and usage report was last read on …, so recent days may be
missing.” — the worker hasn’t read the report for more than two days. Check
that the worker is running and look at its logs.
“The cost and usage report’s status could not be loaded…” — the dashboard
couldn’t reach the API. Reload the page.
Cost and usage report ingestion cannot run until its configuration or the export changes
in the worker logs — the worker stopped trying until something changes. The
log’s error says what, such as a cluster name it can’t resolve. Set
thorasWorker.curClusterName to your EKS cluster’s name.
