# Detector System

Automatic infrastructure analysis and intelligent problem detection service.

## Overview

The `detector` microservice continuously analyzes infrastructure state and identifies:

- **Cost Inefficiencies**: Overprovisioned resources, idle instances, unused storage
- **Compliance Violations**: Security policy breaches, encryption gaps, audit failures
- **Performance Issues**: Bottlenecks, resource constraints, latency anomalies
- **Security Risks**: Exposed resources, misconfigured access, vulnerable versions
- **Reliability Problems**: Single points of failure, insufficient redundancy, missing backups

## Architecture

```text
┌──────────────────────────┐
│   Infrastructure State   │
│  (AWS, UpCloud, etc)     │
└────────────┬─────────────┘
             │
             ▼
     ┌───────────────┐
     │   Detector    │
     │   Service     │
     └───────────────┘
             │
    ┌────────┼────────┐
    │        │        │
    ▼        ▼        ▼
┌─────────┐ ┌──────────┐ ┌─────────────┐
│  Cost   │ │Compliance│ │  Performance│
│Analyzer │ │ Checker  │ │  Analyzer   │
└────┬────┘ └────┬─────┘ └──────┬──────┘
     │           │              │
     └───────────┬──────────────┘
                 │
                 ▼
         ┌──────────────┐
         │  Reporting   │
         │   Engine     │
         └──────────────┘
                 │
    ┌────────────┼────────────┐
    │            │            │
    ▼            ▼            ▼
┌────────┐  ┌────────┐  ┌─────────┐
│ Console│  │  Email │  │Dashboard│
│ Report │  │ Report │  │ Metrics │
└────────┘  └────────┘  └─────────┘
```

## Detection Categories

### 1. Cost Optimization Detector

```nickel
{
  detector = {
    cost_analysis = {
      enabled = true,

      checks = [
        {
          name = "overprovisioned_instances",
          metric = "cpu_utilization",
          threshold = 20,
          recommendation = "Downsize instance"
        },
        {
          name = "idle_resources",
          metric = "network_in",
          threshold = "0 bytes/day",
          recommendation = "Delete unused resource"
        },
        {
          name = "unused_storage",
          metric = "last_accessed",
          threshold = "30 days",
          recommendation = "Archive or delete"
        },
        {
          name = "non_reserved_instances",
          recommendation = "Purchase reserved instances",
          potential_savings = "70% on compute costs"
        }
      ]
    }
  }
}
```

**Findings Example**:

```json
{
  "finding_id": "cost-001",
  "category": "cost_optimization",
  "severity": "medium",
  "resource": {
    "type": "ec2_instance",
    "id": "i-0123456789abcdef0",
    "name": "unused-app-server",
    "instance_type": "m5.2xlarge"
  },
  "issue": "Instance utilization < 5%",
  "metrics": {
    "cpu_utilization_avg": 2.3,
    "memory_utilization_avg": 1.8,
    "network_in_bytes": 0,
    "monthly_cost": 500
  },
  "recommendation": "Terminate instance or downsize to t3.micro",
  "potential_savings": {
    "monthly": 450,
    "annual": 5400
  }
}
```

### 2. Compliance Checker

```nickel
{
  detector = {
    compliance = {
      enabled = true,
      frameworks = ["soc2", "gdpr", "hipaa", "pci-dss"],

      checks = [
        {
          framework = "soc2",
          control = "CC6.1",
          check = "encryption_at_rest",
          resources = ["database", "storage"]
        },
        {
          framework = "gdpr",
          control = "32",
          check = "data_residency",
          requirement = "EU data must stay in EU"
        },
        {
          framework = "hipaa",
          control = "164.308(a)(4)",
          check = "encryption_in_transit",
          requirement = "TLS 1.3 required"
        }
      ]
    }
  }
}
```

**Findings Example**:

```json
{
  "finding_id": "compliance-023",
  "category": "compliance",
  "severity": "critical",
  "framework": "soc2",
  "control": "CC6.1",
  "issue": "Database encryption at rest not enabled",
  "resources": [
    {
      "type": "database",
      "id": "prod-db",
      "status": "encryption_disabled"
    }
  ],
  "remediation": "Enable encryption at rest with KMS key",
  "deadline": "immediate"
}
```

### 3. Performance Analyzer

```nickel
{
  detector = {
    performance = {
      enabled = true,

      checks = [
        {
          name = "high_latency",
          metric = "http_response_time_p99",
          threshold = "500ms",
          comparison = "greater_than"
        },
        {
          name = "high_error_rate",
          metric = "error_rate",
          threshold = "1%",
          comparison = "greater_than"
        },
        {
          name = "database_connection_pool",
          metric = "connections_waiting",
          threshold = "100",
          comparison = "greater_than"
        },
        {
          name = "cache_hit_rate",
          metric = "cache_hit_ratio",
          threshold = "80%",
          comparison = "less_than"
        }
      ]
    }
  }
}
```

**Findings Example**:

```json
{
  "finding_id": "perf-045",
  "category": "performance",
  "severity": "high",
  "service": "api-backend",
  "issue": "P99 latency degradation detected",
  "metrics": {
    "p99_latency_ms": 1240,
    "baseline_ms": 200,
    "degradation_percent": 520,
    "affected_requests": 15000
  },
  "root_cause_hypothesis": "Database connection pool exhaustion",
  "investigation_steps": [
    "Check database slow query logs",
    "Monitor connection pool size",
    "Analyze recent code changes"
  ],
  "recommendation": "Scale database connection pool"
}
```

### 4. Security Risk Detector

```nickel
{
  detector = {
    security = {
      enabled = true,

      checks = [
        {
          name = "exposed_resources",
          scan = "security_group_rules",
          pattern = "0.0.0.0/0"
        },
        {
          name = "outdated_packages",
          scan = "vulnerability_database",
          framework = "snyk"
        },
        {
          name = "unencrypted_communication",
          scan = "tls_configuration",
          minimum_version = "1.3"
        },
        {
          name = "missing_mfa",
          scan = "iam_policies"
        }
      ]
    }
  }
}
```

**Findings Example**:

```json
{
  "finding_id": "sec-089",
  "category": "security",
  "severity": "critical",
  "issue": "RDS database publicly accessible",
  "resource": {
    "type": "rds_database",
    "id": "prod-mysql",
    "publicly_accessible": true
  },
  "risk": "Unauthorized database access",
  "remediation": "Set publicly_accessible = false",
  "cves_affected": ["CVE-2021-1234", "CVE-2021-5678"]
}
```

## Running Detectors

### Command Line

```bash
# Run all detectors
provisioning detector analyze --workspace production

# Run specific detector
provisioning detector analyze \
    --workspace production \
    --detector cost \
    --output json > findings.json

# Run with filters
provisioning detector analyze \
    --workspace production \
    --severity critical \
    --category security \
    --min-potential-savings 1000
```

### Nushell Script

```nushell
def run-comprehensive-analysis [] {
    print "Running comprehensive infrastructure analysis..."

    # Run cost detector
    let cost_findings = (
        provisioning detector analyze \
            --detector cost \
            --format json
    )

    # Run compliance checker
    let compliance_findings = (
        provisioning detector analyze \
            --detector compliance \
            --frameworks ["soc2", "gdpr"]
    )

    # Run performance analyzer
    let perf_findings = (
        provisioning detector analyze \
            --detector performance
    )

    # Run security detector
    let sec_findings = (
        provisioning detector analyze \
            --detector security
    )

    # Aggregate findings
    let all_findings = (
        [$cost_findings, $compliance_findings, $perf_findings, $sec_findings]
        | flatten
        | sort-by severity
    )

    # Generate report
    provisioning detector report \
        --findings $all_findings \
        --output "analysis-report.html"

    print $"✓ Analysis complete: ($all_findings | length) findings"
}
```

### Scheduled Analysis

```nushell
# Schedule daily analysis at 2 AM
provisioning schedule create \
    --name "Daily infrastructure analysis" \
    --cron "0 2 * * *" \
    --command "provisioning detector analyze --workspace production" \
    --notifications {
        email: "[ops@example.com](mailto:ops@example.com)",
        slack: "#infrastructure"
    }
```

## Finding Severity Levels

| Severity | Definition | Action | Timeline |
| ---------| - ----------| - -------| - ---------- |
| **Critical** | Immediate threat to data/availability | Fix now | < 1 hour |
| **High** | Significant risk or waste | Plan remediation | < 1 day |
| **Medium** | Notable issue | Schedule fix | < 1 week |
| **Low** | Minor concern | Backlog | < 1 month |
| **Info** | FYI observation | Monitor | Ongoing |

## Report Generation

### Console Report

```bash
provisioning detector report --format console

# Output:
# ═══════════════════════════════════════════════════════════════════
# INFRASTRUCTURE ANALYSIS REPORT
# ═══════════════════════════════════════════════════════════════════
# Workspace: production
# Analysis Date: 2025-01-16 14:30 UTC
# Duration: 4m 23s
#
# ───────────────────────────────────────────────────────────────────
# SUMMARY
# ───────────────────────────────────────────────────────────────────
# Total Findings: 47
#   Critical: 3
#   High:     12
#   Medium:   18
#   Low:      14
#
# Potential Cost Savings: $45,230/month ($542,760/year)
# Compliance Violations: 5 (SOC2, GDPR, PCI-DSS)
# Performance Issues: 8
# Security Risks: 6
#
# ───────────────────────────────────────────────────────────────────
# TOP FINDINGS
# ───────────────────────────────────────────────────────────────────
# 1. [CRITICAL] Database publicly accessible
#    Recommendation: Set publicly_accessible = false
#    Risk: Data breach
#
# 2. [HIGH] Overprovisioned EC2 instances
#    Potential Savings: $15,000/month
#    Count: 23 instances
```

### HTML Report

```bash
provisioning detector report \
    --format html \
    --output "analysis-report.html" \
    --include-metrics \
    --include-trends

# Generates interactive dashboard with:
# - Finding summaries and details
# - Severity distribution charts
# - Cost savings calculator
# - Compliance status by framework
# - Trend analysis over time
```

### Email Report

```bash
provisioning detector report \
    --format email \
    --recipients "[ops@example.com](mailto:ops@example.com)" \
    --frequency daily \
    --summary-only \
    --highlight-critical
```

## Detector Configuration

```yaml
# ~/.config/provisioning/detector.yaml
detector:
  enabled: true

  scan_interval_hours: 24

  cost_analysis:
    enabled: true
    minimum_savings_threshold: 100

  compliance:
    enabled: true
    frameworks:
      - soc2
      - gdpr
      - hipaa
      - pci-dss

  performance:
    enabled: true
    baseline_window_days: 7

  security:
    enabled: true
    scan_vulnerabilities: true
    cve_database: snyk

  reporting:
    formats: [console, json, html, email]
    recipients:
      - [ops@example.com](mailto:ops@example.com)
      - [compliance@example.com](mailto:compliance@example.com)
    schedule: daily
```

## Performance Characteristics

```text
Analysis Time per Workspace:
├─ Small (< 100 resources): 2-5 minutes
├─ Medium (100-1000 resources): 5-15 minutes
├─ Large (1000-10000 resources): 15-45 minutes
└─ XL (> 10000 resources): 45+ minutes (parallelized)

Memory Usage:
├─ Small workspace: 128 MB
├─ Medium workspace: 256 MB
└─ Large workspace: 512 MB - 2 GB

Database Impact:
├─ Read operations: ~1000 queries
├─ Write operations: ~50 (findings storage)
└─ Estimated query time: < 30 seconds
```

## Integration with Orchestrator

```nushell
# Trigger remediation based on detector findings
def auto-remediate-findings [] {
    let findings = (
        provisioning detector analyze \
            --severity critical \
            --category security
    )

    $findings | each { | finding |
        match $finding.category {
            "security" => {
                # Auto-remediate security issues
                provision security remediate --finding-id $finding.id
            },
            "cost" => {
                # Schedule cost optimization
                provisioning schedule optimize --finding-id $finding.id
            },
            _ => {
                # Create ticket for manual review
                provisioning ticket create \
                    --title $finding.issue \
                    --description $finding.remediation
            }
        }
    }
}
```

## Best Practices

1. **Regular Analysis**: Run daily (cost, compliance) and hourly (performance)
2. **Act on Findings**: Critical findings should be addressed within 1 hour
3. **Track Trends**: Monitor improvement over time
4. **Automate Remediation**: Auto-fix low-risk findings when possible
5. **Review Reports**: Weekly review of findings with team
6. **Adjust Thresholds**: Customize detector thresholds for your infrastructure
7. **Multi-Workspace**: Run analysis across all workspaces regularly

## See Also

- [Cost Optimization Examples](../examples/cost-optimization-examples.md)
- [Security System](./security-system.md)
- [Compliance and Audit Example](../examples/compliance-and-audit-example.md)
- [Automated Incident Response](../architecture/adr-010-incident-response-automation.md)
