FinOps: Cloud Cost Optimization for AI
AI workloads are among the most expensive computing tasks ever created. A single large model training run can cost hundreds of thousands of dollars. Without deliberate cost management, cloud bills for AI can grow faster than revenue. FinOps is the discipline of making every cloud dollar work as hard as possible.
What is FinOps?
FinOps (Financial Operations) is the practice of bringing financial accountability to cloud spending — combining engineering, finance, and business perspectives to help organizations understand cloud costs, make deliberate tradeoffs, and optimize spend without sacrificing the speed of innovation.
The Major Cost Drivers for AI in the Cloud
Training Compute
GPU hours for training runs. An H100 node costs $30–60/hour on major clouds. A week-long training run on 32 H100 nodes = $30 × 32 × 168 = ~$161,280. Training costs scale with model size, dataset size, and number of training tokens. The good news: training is one-time (or periodic), not continuous.
Inference Serving
Inference is continuous — every user query triggers compute. For large models, a single inference endpoint with a loaded 70B model requires 4+ H100 GPUs running 24/7 = $120–240/hour = $87K–175K/month just to keep one replica running. Autoscaling to zero and efficient batching are critical cost controls for inference.
Data Storage & Transfer
Large training datasets (10–100TB) cost $230–2,300/month on S3 Standard. Egress (downloading data to GPU clusters, especially cross-region) can add significantly. A 100TB dataset moved from S3 to a GPU cluster in a different region costs ~$9,000 in egress alone.
Developer Infrastructure
Notebooks, experiment tracking databases, model registries, CI/CD systems, staging environments — these add up. Teams often under-count the "infrastructure supporting the infrastructure" in their AI cost models.
Cost Optimization Strategies
Spot/Preemptible Instances for Training
Spot instances (AWS) and preemptible VMs (GCP) offer 60–90% discounts on GPU compute, in exchange for the risk of being interrupted with 2 minutes' warning when the provider needs capacity back. With checkpoint-resume training (save checkpoints every 100–200 steps, restart from last checkpoint on interruption), spot instances are practical for most training runs. This single change often cuts training costs by 70–80%.
Reserved Instances for Stable Workloads
If you're running inference endpoints 24/7, commit to 1-year or 3-year reserved instances for 30–60% discount vs. on-demand. Only commit to capacity you're confident you'll use — reserved instances don't save money if you're not using them.
Scale-to-Zero for Inference
For lower-traffic AI APIs, deploy to serverless containers (Google Cloud Run, AWS Lambda with containers, Modal) that scale to zero when idle. Pay only for actual inference time, not for idle GPU capacity. A demo or internal tool that gets 100 requests/day costs almost nothing vs. $30–60/hour for a dedicated GPU endpoint.
Right-Sizing Models
Using a 70B model when a 7B model achieves 95% of the quality at 10% of the cost is a major hidden waste in AI systems. Benchmark smaller models before defaulting to the largest available. For many real-world use cases, quantized smaller models outperform large cloud API calls in both cost and latency.
Caching Responses
AI APIs are expensive per query. For workloads with repeated or similar queries (FAQ bots, document summarization of the same documents), semantic caching stores previous responses and serves them for similar queries — without calling the model. Tools like GPTCache and Redis with vector search implement this. Depending on query distribution, caching can reduce API costs by 30–70%.
Cost Attribution & Tagging
You can't optimize what you can't measure. Cloud cost attribution means tagging every resource with metadata that tells you which team, product, model, or experiment it belongs to — so you can break down the bill meaningfully.
The Tagging Strategy
Every cloud resource (VM, storage bucket, GPU instance, endpoint) should be tagged with: team, project, environment (production/staging/dev), model-name, and experiment-id. Cost explorer tools (AWS Cost Explorer, GCP Billing Reports) then let you slice the bill by any of these dimensions. Without tags, your entire AI bill is a black box. With tags, you know exactly that the NLP team's staging environment is $8,000/month and probably should be shut down at night.
Frequently Asked Questions
How much should AI infrastructure cost as a percentage of revenue?
There's no universal benchmark — it depends heavily on the business model. AI-native companies (where the AI IS the product) often spend 15–35% of revenue on compute. For companies where AI is a supporting feature, 3–10% is more typical. The key metric isn't the absolute percentage but the unit economics: cost per API call, cost per inference, cost per training run amortized over the model's lifetime. Track these and set targets for improvement.
What is Savings Plans vs. Reserved Instances on AWS?
Reserved Instances commit to a specific instance type in a specific region (e.g., 2x p4d.24xlarge in us-east-1 for 1 year). Savings Plans commit to a dollar amount of compute per hour ($X/hour) and apply automatically to any eligible compute usage — more flexible but slightly less discount. For AI GPU workloads with predictable steady-state inference, reserved instances for specific GPU instance types typically give slightly better discounts. For mixed workloads, Compute Savings Plans are easier to manage.
Are there cheaper alternatives to AWS, GCP, and Azure for GPU compute?
Yes — significantly cheaper. CoreWeave specializes in GPU cloud and offers H100 clusters at competitive rates. Lambda Labs and Vast.ai offer much cheaper GPU rentals (sometimes 70% less than AWS). Modal and Replicate offer serverless GPU compute with pay-per-second billing. The tradeoff: less ecosystem integration, fewer managed services, and potentially less reliability. For pure training compute, these alternatives often make economic sense. For tightly integrated production inference tied to your cloud provider's ecosystem, the major clouds' managed services may justify the premium.
Frequently Asked Questions
What will I learn here?
This page covers the core concepts and techniques you need to understand the topic and progress confidently to the next lesson.
How should I use this page?
Start with the overview, then follow the section links to deepen your understanding. Use the table of contents on the right to jump to specific sections.
What should I read next?
Use the navigation below to continue to the next lesson or explore related topics.