# AI Cost Management and Optimization\n\n**Status**: ✅ Production-Ready (cost tracking, budgets, caching benefits)\n\nComprehensive guide to managing LLM API costs, optimizing usage through caching and rate limiting, and tracking spending. The provisioning platform\nincludes built-in cost controls to prevent runaway spending while maximizing value.\n\n## Cost Overview\n\n### API Provider Pricing\n\n|  | Provider | Model | Input | Output | Per MTok |  |\n|  | ---------- | ------- | ------- | -------- | ---------- |  |\n|  | **Anthropic** | Claude Sonnet 4 | $3 | $15 | $0.003 input / $0.015 output |  |\n|  |  | Claude Opus 4 | $15 | $45 | Higher accuracy, longer context |  |\n|  |  | Claude Haiku 4 | $0.80 | $4 | Fast, for simple queries |  |\n|  | **OpenAI** | GPT-4 Turbo | $0.01 | $0.03 | Per 1K tokens |  |\n|  |  | GPT-4 | $0.03 | $0.06 | Legacy, avoid |  |\n|  |  | GPT-4o | $5 | $15 | Per MTok |  |\n|  | **Local** | Llama 2, Mistral | Free | Free | Hardware cost only |  |\n\n### Cost Examples\n\n```\nScenario 1: Generate simple database configuration\n  - Input: 500 tokens (description + schema)\n  - Output: 200 tokens (generated config)\n  - Cost: (500 × $3 + 200 × $15) / 1,000,000 = $0.0045\n  - With caching (hit rate 50%): $0.0023\n\nScenario 2: Deep troubleshooting analysis\n  - Input: 5000 tokens (logs + context)\n  - Output: 2000 tokens (analysis + recommendations)\n  - Cost: (5000 × $3 + 2000 × $15) / 1,000,000 = $0.045\n  - With caching (hit rate 70%): $0.0135\n\nScenario 3: Monthly usage (typical organization)\n  - ~1000 config generations @ $0.005 = $5\n  - ~500 troubleshooting calls @ $0.045 = $22.50\n  - ~2000 form assists @ $0.002 = $4\n  - ~200 agent executions @ $0.10 = $20\n  - **Total: ~$50-100/month for small org**\n  - **Total: ~$500-1000/month for large org**\n```\n\n## Cost Control Mechanisms\n\n### Request Caching\n\nCaching is the primary cost reduction strategy, cutting costs by 50-80%:\n\n```\nWithout Caching:\n  User 1: "Generate PostgreSQL config" → API call → $0.005\n  User 2: "Generate PostgreSQL config" → API call → $0.005\n  Total: $0.010 (2 identical requests)\n\nWith LRU Cache:\n  User 1: "Generate PostgreSQL config" → API call → $0.005\n  User 2: "Generate PostgreSQL config" → Cache hit → $0.00001\n  Total: $0.00501 (500x cost reduction for identical)\n\nWith Semantic Cache:\n  User 1: "Generate PostgreSQL database config" → API call → $0.005\n  User 2: "Create a PostgreSQL database" → Semantic hit → $0.00001\n  (Slightly different wording, but same intent)\n  Total: $0.00501 (near 500x reduction for similar)\n```\n\n### Cache Configuration\n\n```\n[ai.cache]\nenabled = true\ncache_type = "redis"  # Distributed cache across instances\nttl_seconds = 3600    # 1-hour cache lifetime\n\n# Cache size limits\nmax_size_mb = 500\neviction_policy = "lru"  # Least Recently Used\n\n# Semantic caching - cache similar queries\n[ai.cache.semantic]\nenabled = true\nsimilarity_threshold = 0.95  # Cache if 95%+ similar to previous query\ncache_embeddings = true      # Cache embedding vectors themselves\n\n# Cache metrics\n[ai.cache.metrics]\ntrack_hit_rate = true\ntrack_space_usage = true\nalert_on_low_hit_rate = true\n```\n\n### Rate Limiting\n\nPrevent usage spikes from unexpected costs:\n\n```\n[ai.limits]\n# Per-request limits\nmax_tokens = 4096\nmax_input_tokens = 8192\nmax_output_tokens = 4096\n\n# Throughput limits\nrpm_limit = 60                    # 60 requests per minute\nrpm_burst = 100                   # Allow burst to 100\ndaily_request_limit = 5000        # Max 5000 requests/day\n\n# Cost limits\ndaily_cost_limit_usd = 100        # Stop at $100/day\nmonthly_cost_limit_usd = 2000     # Stop at $2000/month\n\n# Budget alerts\nwarn_at_percent = 80              # Warn when at 80% of daily budget\nstop_at_percent = 95              # Stop when at 95% of budget\n```\n\n### Workspace-Level Budgets\n\n```\n[ai.workspace_budgets]\n# Per-workspace cost limits\ndev.daily_limit_usd = 10\nstaging.daily_limit_usd = 50\nprod.daily_limit_usd = 100\n\n# Can override globally for specific workspaces\nteams.team-a.monthly_limit = 500\nteams.team-b.monthly_limit = 300\n```\n\n## Cost Tracking\n\n### Track Spending\n\n```\n# View current month spending\nprovisioning admin costs show ai\n\n# Forecast monthly spend\nprovisioning admin costs forecast ai --days-remaining 15\n\n# Analyze by feature\nprovisioning admin costs analyze ai --by feature\n\n# Analyze by user\nprovisioning admin costs analyze ai --by user\n\n# Export for billing\nprovisioning admin costs export ai --format csv --output costs.csv\n```\n\n### Cost Breakdown\n\n```\nMonth: January 2025\n\nTotal Spending: $285.42\n\nBy Feature:\n  Config Generation:    $150.00 (52%) [300 requests × avg $0.50]\n  Troubleshooting:      $95.00  (33%) [80 requests × avg $1.19]\n  Form Assistance:      $30.00  (11%) [5000 requests × avg $0.006]\n  Agents:               $10.42  (4%)  [20 runs × avg $0.52]\n\nBy Provider:\n  Anthropic (Claude):   $200.00 (70%)\n  OpenAI (GPT-4):       $85.42  (30%)\n  Local:                $0      (0%)\n\nBy User:\n  alice@company.com:    $50.00  (18%)\n  bob@company.com:      $45.00  (16%)\n  ...\n  other (20 users):     $190.42 (67%)\n\nBy Workspace:\n  production:           $150.00 (53%)\n  staging:              $85.00  (30%)\n  development:          $50.42  (18%)\n\nCache Performance:\n  Requests: 50,000\n  Cache hits: 35,000 (70%)\n  Cache misses: 15,000 (30%)\n  Cost savings from cache: ~$175 (38% reduction)\n```\n\n## Optimization Strategies\n\n### Strategy 1: Increase Cache Hit Rate\n\n```\n# Longer TTL = more cache hits\n[ai.cache]\nttl_seconds = 7200  # 2 hours instead of 1 hour\n\n# Semantic caching helps with slight variations\n[ai.cache.semantic]\nenabled = true\nsimilarity_threshold = 0.90  # Lower threshold = more hits\n\n# Result: Increase hit rate from 65% → 80%\n# Cost reduction: 15% → 23%\n```\n\n### Strategy 2: Use Local Models\n\n```\n[ai]\nprovider = "local"\nmodel = "mistral-7b"  # Free, runs on GPU\n\n# Cost: Hardware ($5-20/month) instead of API calls\n# Savings: 50-100 config generations/month × $0.005 = $0.25-0.50\n# Hardware amortized cost: <$0.50/month on existing GPU\n\n# Tradeoff: Slightly lower quality, 2x slower\n```\n\n### Strategy 3: Use Haiku for Simple Tasks\n\n```\nTask Complexity vs Model:\n\nSimple (form assist): Claude Haiku 4 ($0.80/$4)\nMedium (config gen): Claude Sonnet 4 ($3/$15)\nComplex (agents): Claude Opus 4 ($15/$45)\n\nExample optimization:\n  Before: All tasks use Sonnet 4\n  - 5000 form assists/month: 5000 × $0.006 = $30\n  \n  After: Route by complexity\n  - 5000 form assists → Haiku: 5000 × $0.001 = $5 (83% savings)\n  - 200 config gen → Sonnet: 200 × $0.005 = $1\n  - 10 agent runs → Opus: 10 × $0.10 = $1\n```\n\n### Strategy 4: Batch Operations\n\n```\n# Instead of individual requests, batch similar operations:\n\n# Before: 100 configs, 100 separate API calls\nprovisioning ai generate "PostgreSQL config" --output db1.ncl\nprovisioning ai generate "PostgreSQL config" --output db2.ncl\n# ... 100 calls = $0.50\n\n# After: Batch similar requests\nprovisioning ai batch --input configs-list.yaml\n# Groups similar requests, reuses cache\n# ... 3-5 API calls = $0.02 (90% savings)\n```\n\n### Strategy 5: Smart Feature Enablement\n\n```\n[ai.features]\n# Enable high-ROI features\nconfig_generation = true    # High value, moderate cost\ntroubleshooting = true      # High value, higher cost\nrag_search = true           # Low cost, high value\n\n# Disable low-ROI features if cost-constrained\nform_assistance = false     # Low value, non-zero cost (if budget tight)\nagents = false              # Complex, requires multiple calls\n```\n\n## Budget Management Workflow\n\n### 1. Set Budget\n\n```\n# Set monthly budget\nprovisioning config set ai.budget.monthly_limit_usd 500\n\n# Set daily limit\nprovisioning config set ai.limits.daily_cost_limit_usd 50\n\n# Set workspace limits\nprovisioning config set ai.workspace_budgets.prod.monthly_limit 300\nprovisioning config set ai.workspace_budgets.dev.monthly_limit 100\n```\n\n### 2. Monitor Spending\n\n```\n# Daily check\nprovisioning admin costs show ai\n\n# Weekly analysis\nprovisioning admin costs analyze ai --period week\n\n# Monthly review\nprovisioning admin costs analyze ai --period month\n```\n\n### 3. Adjust If Needed\n\n```\n# If overspending:\n# - Increase cache TTL\n# - Enable local models for simple tasks\n# - Reduce form assistance (high volume, low cost but adds up)\n# - Route complex tasks to Haiku instead of Opus\n\n# If underspending:\n# - Enable new features (agents, form assistance)\n# - Increase rate limits\n# - Lower cache hit requirements (broader semantic matching)\n```\n\n### 4. Forecast and Plan\n\n```\n# Current monthly run rate\nprovisioning admin costs forecast ai\n\n# If trending over budget, recommend actions:\n# - Reduce daily limit\n# - Switch to local model for 50% of tasks\n# - Increase batch processing\n\n# If trending under budget:\n# - Enable agents for automation workflows\n# - Enable form assistance across all workspaces\n```\n\n## Cost Allocation\n\n### Chargeback Models\n\n**Per-Workspace Model**:\n```\nDevelopment workspace: $50/month\nStaging workspace:     $100/month\nProduction workspace:  $300/month\n------\nTotal:                 $450/month\n```\n\n**Per-User Model**:\n```\nEach user charged based on their usage\nEncourages efficiency\nDifficult to track/allocate\n```\n\n**Shared Pool Model**:\n```\nAll teams share $1000/month budget\nBudget splits by consumption rate\nEncourages optimization\nMost flexible\n```\n\n## Cost Reporting\n\n### Generate Reports\n\n```\n# Monthly cost report\nprovisioning admin costs report ai \\n  --format pdf \\n  --period month \\n  --output cost-report-2025-01.pdf\n\n# Detailed analysis for finance\nprovisioning admin costs report ai \\n  --format xlsx \\n  --include-forecasts \\n  --include-optimization-suggestions\n\n# Executive summary\nprovisioning admin costs report ai \\n  --format markdown \\n  --summary-only\n```\n\n## Cost-Benefit Analysis\n\n### ROI Examples\n\n```\nScenario 1: Developer Time Savings\n  Problem: Manual config creation takes 2 hours\n  Solution: AI config generation, 10 minutes (12x faster)\n  Time saved: 1.83 hours/config\n  Hourly rate: $100\n  Value: $183/config\n  \n  AI cost: $0.005/config\n  ROI: 36,600x (far exceeds cost)\n\nScenario 2: Troubleshooting Efficiency\n  Problem: Manual debugging takes 4 hours\n  Solution: AI troubleshooting analysis, 2 minutes\n  Time saved: 3.97 hours\n  Value: $397/incident\n  \n  AI cost: $0.045/incident\n  ROI: 8,822x\n\nScenario 3: Reduction in Failed Deployments\n  Before: 5% of 1000 deployments fail (50 failures)\n  Failure cost: $500 each (lost time, data cleanup)\n  Total: $25,000/month\n  \n  After: With AI analysis, 2% fail (20 failures)\n  Total: $10,000/month\n  Savings: $15,000/month\n  \n  AI cost: $200/month\n  Net savings: $14,800/month\n  ROI: 74:1\n```\n\n## Advanced Cost Optimization\n\n### Hybrid Strategy (Recommended)\n\n```\n✓ Local models for:\n  - Form assistance (high volume, low complexity)\n  - Simple validation checks\n  - Document retrieval (RAG)\n  Cost: Hardware only (~$500 setup)\n\n✓ Cloud API for:\n  - Complex generation (requires latest model capability)\n  - Troubleshooting (needs high accuracy)\n  - Agents (complex reasoning)\n  Cost: $50-200/month per organization\n\nResult:\n  - 70% of requests → Local (free after hardware amortization)\n  - 30% of requests → Cloud ($50/month)\n  - 80% overall cost reduction vs cloud-only\n```\n\n## Monitoring and Alerts\n\n### Cost Anomaly Detection\n\n```\n# Enable anomaly detection\nprovisioning config set ai.monitoring.anomaly_detection true\n\n# Set thresholds\nprovisioning config set ai.monitoring.cost_spike_percent 150\n# Alert if daily cost is 150% of average\n\n# System alerts:\n# - Daily cost exceeded by 10x normal\n# - New expensive operation (agent run)\n# - Cache hit rate dropped below 40%\n# - Rate limit nearly exhausted\n```\n\n### Alert Configuration\n\n```\n[ai.monitoring.alerts]\nenabled = true\nspike_threshold_percent = 150\ncheck_interval_minutes = 5\n\n[ai.monitoring.alerts.channels]\nemail = "ops@company.com"\nslack = "[https://hooks.slack.com/..."](https://hooks.slack.com/...")\npagerduty = "integration-key"\n\n# Alert thresholds\n[ai.monitoring.alerts.thresholds]\ndaily_budget_warning_percent = 80\ndaily_budget_critical_percent = 95\nmonthly_budget_warning_percent = 70\n```\n\n## Related Documentation\n\n- [Architecture](architecture.md) - AI system overview\n- [Configuration](configuration.md) - Cost control settings\n- [Security Policies](security-policies.md) - Cost-aware policies\n- [RAG System](rag-system.md) - Caching details\n- [ADR-015](../architecture/adr/adr-015-ai-integration-architecture.md) - Design decisions\n\n---\n\n**Last Updated**: 2025-01-13\n**Status**: ✅ Production-Ready\n**Average Savings**: 50-80% through caching\n**Typical Cost**: $50-500/month per organization\n**ROI**: 100:1 to 10,000:1 depending on use case
