# AI-Assisted Troubleshooting and Debugging\n\n**Status**: ✅ Production-Ready (AI troubleshooting analysis, log parsing)\n\nThe AI troubleshooting system provides intelligent debugging assistance for infrastructure failures. The system analyzes deployment logs, identifies\nroot causes, suggests fixes, and generates corrected configurations based on failure patterns.\n\n## Feature Overview\n\n### What It Does\n\nTransform deployment failures into actionable insights:\n\n```\nDeployment Fails with Error\n        ↓\nAI analyzes logs:\n  - Identifies failure phase (networking, database, k8s, etc.)\n  - Detects root cause (resource limits, configuration, timeout)\n  - Correlates with similar past failures\n  - Reviews deployment configuration\n        ↓\nAI generates report:\n  - Root cause explanation in plain English\n  - Configuration issues identified\n  - Suggested fixes with rationale\n  - Alternative solutions\n  - Links to relevant documentation\n        ↓\nDeveloper reviews and accepts:\n  - Understands what went wrong\n  - Knows how to fix it\n  - Can implement fix with confidence\n```\n\n## Troubleshooting Workflow\n\n### Automatic Detection and Analysis\n\n```\n┌──────────────────────────────────────────┐\n│ Deployment Monitoring                    │\n│ - Watches deployment for failures        │\n│ - Captures logs in real-time             │\n│ - Detects failure events                 │\n└──────────────┬───────────────────────────┘\n               ↓\n┌──────────────────────────────────────────┐\n│ Log Collection                           │\n│ - Gather all relevant logs               │\n│ - Include stack traces                   │\n│ - Capture metrics at failure time        │\n│ - Get resource usage data                │\n└──────────────┬───────────────────────────┘\n               ↓\n┌──────────────────────────────────────────┐\n│ Context Retrieval (RAG)                  │\n│ - Find similar past failures             │\n│ - Retrieve troubleshooting guides        │\n│ - Get schema constraints                 │\n│ - Find best practices                    │\n└──────────────┬───────────────────────────┘\n               ↓\n┌──────────────────────────────────────────┐\n│ AI Analysis                              │\n│ - Identify failure pattern               │\n│ - Determine root cause                   │\n│ - Generate hypotheses                    │\n│ - Score likely causes                    │\n└──────────────┬───────────────────────────┘\n               ↓\n┌──────────────────────────────────────────┐\n│ Solution Generation                      │\n│ - Create fixed configuration             │\n│ - Generate step-by-step fix guide        │\n│ - Suggest preventative measures          │\n│ - Provide alternative approaches         │\n└──────────────┬───────────────────────────┘\n               ↓\n┌──────────────────────────────────────────┐\n│ Report and Recommendations               │\n│ - Explain what went wrong                │\n│ - Show how to fix it                     │\n│ - Provide corrected configuration        │\n│ - Link to prevention strategies          │\n└──────────────────────────────────────────┘\n```\n\n## Usage Examples\n\n### Example 1: Database Connection Timeout\n\n**Failure**:\n```\nDeployment: deploy-2025-01-13-001\nStatus: FAILED at phase database_migration\nError: connection timeout after 30s connecting to postgres://...\n```\n\n**Run Troubleshooting**:\n```\n$ provisioning ai troubleshoot deploy-2025-01-13-001\n\nAnalyzing deployment failure...\n\n╔════════════════════════════════════════════════════════════════╗\n║ Root Cause Analysis: Database Connection Timeout              ║\n╠════════════════════════════════════════════════════════════════╣\n║                                                                ║\n║ Phase: database_migration (occurred during migration job)     ║\n║ Error: Timeout after 30 seconds connecting to database        ║\n║                                                                ║\n║ Most Likely Causes (confidence):                              ║\n║   1. Database security group blocks migration job (85%)       ║\n║   2. Database instance not fully initialized yet (60%)        ║\n║   3. Network connectivity issue (40%)                         ║\n║                                                                ║\n║ Analysis:                                                     ║\n║   - Database was created only 2 seconds before connection    ║\n║   - Migration job started immediately (no wait time)         ║\n║   - Security group: allows 5432 only from default SG         ║\n║   - Migration pod uses different security group              ║\n║                                                                ║\n╠════════════════════════════════════════════════════════════════╣\n║ Recommended Fix                                                ║\n╠════════════════════════════════════════════════════════════════╣\n║                                                                ║\n║ Issue: Migration security group not in database's inbound    ║\n║                                                                ║\n║ Solution: Add migration pod security group to DB inbound     ║\n║                                                                ║\n║   database.security_group.ingress = [                         ║\n║     {                                                          ║\n║       from_port = 5432,                                       ║\n║       to_port = 5432,                                         ║\n║       source_security_group = "migration-pods-sg"             ║\n║     }                                                          ║\n║   ]                                                            ║\n║                                                                ║\n║ Alternative: Add 30-second wait after database creation      ║\n║                                                                ║\n║   deployment.phases.database.post_actions = [                 ║\n║     {action = "wait_for_database", timeout_seconds = 30}     ║\n║   ]                                                            ║\n║                                                                ║\n╠════════════════════════════════════════════════════════════════╣\n║ Prevention                                                     ║\n╠════════════════════════════════════════════════════════════════╣\n║                                                                ║\n║ To prevent this in future deployments:                        ║\n║                                                                ║\n║ 1. Always verify security group rules before migration       ║\n║ 2. Add health check: `SELECT 1` before starting migration    ║\n║ 3. Increase initial timeout: database can be slow to start   ║\n║ 4. Use RDS wait condition instead of time-based wait         ║\n║                                                                ║\n║ See: docs/troubleshooting/database-connectivity.md            ║\n║      docs/guides/database-migrations.md                       ║\n║                                                                ║\n╚════════════════════════════════════════════════════════════════╝\n\nGenerate corrected configuration? [yes/no]: yes\n\nConfiguration generated and saved to:\n  workspaces/prod/database.ncl.fixed\n\nChanges made:\n  ✓ Added migration security group to database inbound\n  ✓ Added health check before migration\n  ✓ Increased connection timeout to 60s\n\nReady to redeploy with corrected configuration? [yes/no]: yes\n```\n\n### Example 2: Kubernetes Deployment Error\n\n**Failure**:\n```\nDeployment: deploy-2025-01-13-002\nStatus: FAILED at phase kubernetes_workload\nError: failed to create deployment app: Pod exceeded capacity\n```\n\n**Troubleshooting**:\n```\n$ provisioning ai troubleshoot deploy-2025-01-13-002 --detailed\n\n╔════════════════════════════════════════════════════════════════╗\n║ Root Cause: Pod Exceeded Node Capacity                        ║\n╠════════════════════════════════════════════════════════════════╣\n║                                                                ║\n║ Failure Analysis:                                             ║\n║                                                                ║\n║ Error: Pod requests 4CPU/8GB, but largest node has 2CPU/4GB  ║\n║ Cluster: 3 nodes, each t3.medium (2CPU/4GB)                  ║\n║ Pod requirements:                                             ║\n║   - CPU: 4 (requested) + 2 (reserved system) = 6 needed      ║\n║   - Memory: 8Gi (requested) + 1Gi (system) = 9Gi needed      ║\n║                                                                ║\n║ Why this happened:                                            ║\n║   Pod spec updated to 4CPU/8GB but node group wasn't        ║\n║   Node group still has t3.medium (too small)                 ║\n║   No autoscaling configured (won't scale up automatically)   ║\n║                                                                ║\n║ Solution Options:                                             ║\n║   1. Reduce pod resource requests to 2CPU/4GB (simpler)      ║\n║   2. Scale up node group to t3.large (2x cost, safer)        ║\n║   3. Use both: t3.large nodes + reduce pod requests          ║\n║                                                                ║\n╠════════════════════════════════════════════════════════════════╣\n║ Recommended: Option 2 (Scale up nodes)                        ║\n╠════════════════════════════════════════════════════════════════╣\n║                                                                ║\n║ Reason: Pod requests are reasonable for production app       ║\n║         Better to scale infrastructure than reduce resources  ║\n║                                                                ║\n║ Changes needed:                                               ║\n║                                                                ║\n║   kubernetes.node_group = {                                   ║\n║     instance_type = "t3.large"  # was t3.medium              ║\n║     min_size = 3                                              ║\n║     max_size = 10                                             ║\n║                                                                ║\n║     auto_scaling = {                                          ║\n║       enabled = true                                          ║\n║       target_cpu_percent = 70                                 ║\n║     }                                                          ║\n║   }                                                            ║\n║                                                                ║\n║ Cost Impact:                                                  ║\n║   Current: 3 × t3.medium = ~$90/month                        ║\n║   Proposed: 3 × t3.large = ~$180/month                       ║\n║   With autoscaling, average: ~$150/month (some scale-down)   ║\n║                                                                ║\n╚════════════════════════════════════════════════════════════════╝\n```\n\n## CLI Commands\n\n### Basic Troubleshooting\n\n```\n# Troubleshoot recent deployment\nprovisioning ai troubleshoot deploy-2025-01-13-001\n\n# Get detailed analysis\nprovisioning ai troubleshoot deploy-2025-01-13-001 --detailed\n\n# Analyze with specific focus\nprovisioning ai troubleshoot deploy-2025-01-13-001 --focus networking\n\n# Get alternative solutions\nprovisioning ai troubleshoot deploy-2025-01-13-001 --alternatives\n```\n\n### Working with Logs\n\n```\n# Troubleshoot from custom logs\nprovisioning ai troubleshoot \\n| --logs "$(journalctl -u provisioning --no-pager | tail -100)" |\n\n# Troubleshoot from file\nprovisioning ai troubleshoot --log-file /var/log/deployment.log\n\n# Troubleshoot from cloud provider\nprovisioning ai troubleshoot \\n  --cloud-logs aws-deployment-123 \\n  --region us-east-1\n```\n\n### Generate Reports\n\n```\n# Generate detailed troubleshooting report\nprovisioning ai troubleshoot deploy-123 \\n  --report \\n  --output troubleshooting-report.md\n\n# Generate with suggestions\nprovisioning ai troubleshoot deploy-123 \\n  --report \\n  --include-suggestions \\n  --output report-with-fixes.md\n\n# Generate compliance report (PCI-DSS, HIPAA)\nprovisioning ai troubleshoot deploy-123 \\n  --report \\n  --compliance pci-dss \\n  --output compliance-report.pdf\n```\n\n## Analysis Depth\n\n### Shallow Analysis (Fast)\n\n```\nprovisioning ai troubleshoot deploy-123 --depth shallow\n\nAnalyzes:\n- First error message\n- Last few log lines\n- Basic pattern matching\n- Returns in 30-60 seconds\n```\n\n### Deep Analysis (Thorough)\n\n```\nprovisioning ai troubleshoot deploy-123 --depth deep\n\nAnalyzes:\n- Full log context\n- Correlates multiple errors\n- Checks resource metrics\n- Compares to past failures\n- Generates alternative hypotheses\n- Returns in 5-10 seconds\n```\n\n## Integration with Monitoring\n\n### Automatic Troubleshooting\n\n```\n# Enable auto-troubleshoot on failures\nprovisioning config set ai.troubleshooting.auto_analyze true\n\n# Deployments that fail automatically get analyzed\n# Reports available in provisioning dashboard\n# Alerts sent to on-call engineer with analysis\n```\n\n### WebUI Integration\n\n```\nDeployment Dashboard\n  ├─ deployment-123 [FAILED]\n  │   └─ AI Analysis\n  │       ├─ Root Cause: Database timeout\n  │       ├─ Suggested Fix: ✓ View\n  │       ├─ Corrected Config: ✓ Download\n  │       └─ Alternative Solutions: 3 options\n```\n\n## Learning from Failures\n\n### Pattern Recognition\n\nThe system learns common failure patterns:\n\n```\nCollected Patterns:\n├─ Database Timeouts (25% of failures)\n│  └─ Usually: Security group, connection pool, slow startup\n├─ Kubernetes Pod Failures (20%)\n│  └─ Usually: Insufficient resources, bad config\n├─ Network Connectivity (15%)\n│  └─ Usually: Security groups, routing, DNS\n└─ Other (40%)\n   └─ Various causes, each analyzed individually\n```\n\n### Improvement Tracking\n\n```\n# See patterns in your deployments\nprovisioning ai analytics failures --period month\n\nMonth Summary:\n  Total deployments: 50\n  Failed: 5 (10% failure rate)\n  \n  Common causes:\n  1. Security group rules (3 failures, 60%)\n  2. Resource limits (1 failure, 20%)\n  3. Configuration error (1 failure, 20%)\n  \n  Improvement opportunities:\n  - Pre-check security groups before deployment\n  - Add health checks for resource sizing\n  - Add configuration validation\n```\n\n## Configuration\n\n### Troubleshooting Settings\n\n```\n[ai.troubleshooting]\nenabled = true\n\n# Analysis depth\ndefault_depth = "deep"  # or "shallow" for speed\nmax_analysis_time_seconds = 30\n\n# Features\nauto_analyze_failed_deployments = true\ngenerate_corrected_config = true\nsuggest_prevention = true\n\n# Learning\ntrack_failure_patterns = true\nlearn_from_similar_failures = true\nimprove_suggestions_over_time = true\n\n# Reporting\nauto_send_report = false  # Email report to user\nreport_format = "markdown"  # or "json", "pdf"\ninclude_alternatives = true\n\n# Cost impact analysis\nestimate_fix_cost = true\nestimate_alternative_costs = true\n```\n\n### Failure Detection\n\n```\n[ai.troubleshooting.detection]\n# Monitor logs for these patterns\nwatch_patterns = [\n  "error",\n  "timeout",\n  "failed",\n  "unable to",\n  "refused",\n  "denied",\n  "exceeded",\n  "quota",\n]\n\n# Minimum log lines before analyzing\nmin_log_lines = 10\n\n# Time window for log collection\nlog_window_seconds = 300\n```\n\n## Best Practices\n\n### For Effective Troubleshooting\n\n1. **Keep Detailed Logs**: Enable verbose logging in deployments\n2. **Include Context**: Share full logs, not just error snippet\n3. **Check Suggestions**: Review AI suggestions even if obvious\n4. **Learn Patterns**: Track recurring failures and address root cause\n5. **Update Configs**: Use corrected configs from AI, validate them\n\n### For Prevention\n\n1. **Use Health Checks**: Add database/service health checks\n2. **Test Before Deploy**: Use dry-run to catch issues early\n3. **Monitor Metrics**: Watch CPU/memory before failures occur\n4. **Review Policies**: Ensure security groups are correct\n5. **Document Changes**: When updating configs, note the change\n\n## Limitations\n\n### What AI Can Troubleshoot\n\n✅ Configuration errors\n✅ Resource limit problems\n✅ Networking/security group issues\n✅ Database connectivity problems\n✅ Deployment ordering issues\n✅ Common application errors\n✅ Performance problems\n\n### What Requires Human Review\n\n⚠️ Data corruption scenarios\n⚠️ Multi-failure cascades\n⚠️ Unclear error messages\n⚠️ Custom application code failures\n⚠️ Third-party service issues\n⚠️ Physical infrastructure failures\n\n## Examples and Guides\n\n### Common Issues - Quick Links\n\n- [Database Connectivity](../troubleshooting/database-connectivity.md)\n- [Kubernetes Pod Failures](../troubleshooting/kubernetes-pods.md)\n- [Network Configuration](../troubleshooting/networking.md)\n- [Performance Issues](../troubleshooting/performance.md)\n- [Resource Limits](../troubleshooting/resource-limits.md)\n\n## Related Documentation\n\n- [Architecture](architecture.md) - AI system overview\n- [RAG System](rag-system.md) - Context retrieval for troubleshooting\n- [Configuration](configuration.md) - Setup guide\n- [Security Policies](security-policies.md) - Safe log handling\n- [ADR-015](../architecture/adr/adr-015-ai-integration-architecture.md) - Design decisions\n\n---\n\n**Last Updated**: 2025-01-13\n**Status**: ✅ Production-Ready\n**Success Rate**: 85-95% accuracy in root cause identification\n**Supported**: All deployment types (infrastructure, Kubernetes, database)
