# Service Monitoring & Alerting Setup

**Complete guide for monitoring the 9-service platform with Prometheus, Grafana, and AlertManager**

**Version**: 1.0.0
**Last Updated**: 2026-01-05
**Target Audience**: DevOps Engineers, Platform Operators
**Status**: Production Ready

---

## Overview

This guide provides complete setup instructions for monitoring and alerting on the provisioning platform using industry-standard tools:

- **Prometheus**: Metrics collection and time-series database
- **Grafana**: Visualization and dashboarding
- **AlertManager**: Alert routing and notification

---

## Architecture

```bash
Services (metrics endpoints)
    ↓
Prometheus (scrapes every 30s)
    ↓
AlertManager (evaluates rules)
    ↓
Notification Channels (email, slack, pagerduty)

Prometheus Data
    ↓
Grafana (queries)
    ↓
Dashboards & Visualization
```

---

## Prerequisites

### Software Requirements

```bash
# Prometheus (for metrics)
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-2.48.0.linux-amd64.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus

# Grafana (for dashboards)
sudo apt-get install -y grafana-server

# AlertManager (for alerting)
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager
```

### System Requirements

- **CPU**: 2+ cores
- **Memory**: 4 GB minimum, 8 GB recommended
- **Disk**: 100 GB for metrics retention (30 days)
- **Network**: Access to all service endpoints

### Ports

| Component | Port | Purpose |
|-----------|------|---------|
| Prometheus | 9090 | Web UI & API |
| Grafana | 3000 | Web UI |
| AlertManager | 9093 | Web UI & API |
| Node Exporter | 9100 | System metrics |

---

## Service Metrics Endpoints

All platform services expose metrics on the `/metrics` endpoint:

```bash
# Health and metrics endpoints for each service
curl http://localhost:8200/health    # Vault health
curl http://localhost:8200/metrics   # Vault metrics (Prometheus format)

curl http://localhost:8081/health    # Registry health
curl http://localhost:8081/metrics   # Registry metrics

curl http://localhost:8083/health    # RAG health
curl http://localhost:8083/metrics   # RAG metrics

curl http://localhost:8082/health    # AI Service health
curl http://localhost:8082/metrics   # AI Service metrics

curl http://localhost:9090/health    # Orchestrator health
curl http://localhost:9090/metrics   # Orchestrator metrics

curl http://localhost:8080/health    # Control Center health
curl http://localhost:8080/metrics   # Control Center metrics

curl http://localhost:8084/health    # MCP Server health
curl http://localhost:8084/metrics   # MCP Server metrics
```

---

## Prometheus Configuration

### 1. Create Prometheus Config

```yaml
# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
  external_labels:
    monitor: 'provisioning-platform'
    environment: 'production'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

rule_files:
  - '/etc/prometheus/rules/*.yml'

scrape_configs:
  # Core Platform Services
  - job_name: 'vault-service'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8200']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'vault-service'

  - job_name: 'extension-registry'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8081']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'registry'

  - job_name: 'rag-service'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8083']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'rag'

  - job_name: 'ai-service'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8082']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'ai-service'

  - job_name: 'orchestrator'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'orchestrator'

  - job_name: 'control-center'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8080']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'control-center'

  - job_name: 'mcp-server'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8084']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: 'mcp-server'

  # System Metrics (Node Exporter)
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          instance: 'system'

  # SurrealDB (if multiuser/enterprise)
  - job_name: 'surrealdb'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['surrealdb:8000']

  # Etcd (if enterprise)
  - job_name: 'etcd'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['etcd:2379']
```

### 2. Start Prometheus

```bash
# Create necessary directories
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mkdir -p /etc/prometheus/rules

# Start Prometheus
cd /opt/prometheus
sudo ./prometheus --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=consoles \
  --web.console.libraries=console_libraries

# Or as systemd service
sudo tee /etc/systemd/system/prometheus.service > /dev/null << EOF
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus

Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
```

### 3. Verify Prometheus

```bash
# Check Prometheus is running
curl -s http://localhost:9090/-/healthy

# List scraped targets
curl -s http://localhost:9090/api/v1/targets | jq .

# Query test metric
curl -s 'http://localhost:9090/api/v1/query?query=up' | jq .
```

---

## Alert Rules Configuration

### 1. Create Alert Rules

```yaml
# /etc/prometheus/rules/platform-alerts.yml
groups:
  - name: platform_availability
    interval: 30s
    rules:
      - alert: ServiceDown
        expr: up{job=~"vault-service|registry|rag|ai-service|orchestrator"} == 0
        for: 5m
        labels:
          severity: critical
          service: '{{ $labels.job }}'
        annotations:
          summary: "{{ $labels.job }} is DOWN"
          description: "{{ $labels.job }} has been down for 5+ minutes"

      - alert: ServiceSlowResponse
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
          service: '{{ $labels.job }}'
        annotations:
          summary: "{{ $labels.job }} slow response times"
          description: "95th percentile latency above 1 second"

  - name: platform_errors
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
          service: '{{ $labels.job }}'
        annotations:
          summary: "{{ $labels.job }} high error rate"
          description: "Error rate above 5% for 5 minutes"

      - alert: DatabaseConnectionError
        expr: increase(database_connection_errors_total[5m]) > 10
        for: 2m
        labels:
          severity: critical
          component: database
        annotations:
          summary: "Database connection failures detected"
          description: "{{ $value }} connection errors in last 5 minutes"

      - alert: QueueBacklog
        expr: orchestrator_queue_depth > 1000
        for: 5m
        labels:
          severity: warning
          component: orchestrator
        annotations:
          summary: "Orchestrator queue backlog growing"
          description: "Queue depth: {{ $value }} tasks"

  - name: platform_resources
    interval: 30s
    rules:
      - alert: HighMemoryUsage
        expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
        for: 5m
        labels:
          severity: warning
          resource: memory
        annotations:
          summary: "{{ $labels.container_name }} memory usage critical"
          description: "Memory usage: {{ $value | humanizePercentage }}"

      - alert: HighDiskUsage
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes < 0.1
        for: 5m
        labels:
          severity: warning
          resource: disk
        annotations:
          summary: "Disk space critically low"
          description: "Available disk space: {{ $value | humanizePercentage }}"

      - alert: HighCPUUsage
        expr: (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)) > 0.9
        for: 10m
        labels:
          severity: warning
          resource: cpu
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage: {{ $value | humanizePercentage }}"

      - alert: DiskIOLatency
        expr: node_disk_io_time_seconds_total > 100
        for: 5m
        labels:
          severity: warning
          resource: disk
        annotations:
          summary: "High disk I/O latency"
          description: "I/O latency: {{ $value }}ms"

  - name: platform_network
    interval: 30s
    rules:
      - alert: HighNetworkLatency
        expr: probe_duration_seconds > 0.5
        for: 5m
        labels:
          severity: warning
          component: network
        annotations:
          summary: "High network latency detected"
          description: "Latency: {{ $value }}ms"

      - alert: PacketLoss
        expr: node_network_transmit_errors_total > 100
        for: 5m
        labels:
          severity: warning
          component: network
        annotations:
          summary: "Packet loss detected"
          description: "Transmission errors: {{ $value }}"

  - name: platform_services
    interval: 30s
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
          service: vault
        annotations:
          summary: "Vault is sealed"
          description: "Vault instance is sealed and requires unseal operation"

      - alert: RegistryAuthError
        expr: increase(registry_auth_failures_total[5m]) > 5
        for: 2m
        labels:
          severity: warning
          service: registry
        annotations:
          summary: "Registry authentication failures"
          description: "{{ $value }} auth failures in last 5 minutes"

      - alert: RAGVectorDBDown
        expr: rag_vectordb_connection_status == 0
        for: 2m
        labels:
          severity: critical
          service: rag
        annotations:
          summary: "RAG Vector Database disconnected"
          description: "Vector DB connection lost"

      - alert: AIServiceMCPError
        expr: increase(ai_service_mcp_errors_total[5m]) > 10
        for: 2m
        labels:
          severity: warning
          service: ai_service
        annotations:
          summary: "AI Service MCP integration errors"
          description: "{{ $value }} errors in last 5 minutes"

      - alert: OrchestratorLeaderElectionIssue
        expr: orchestrator_leader_elected == 0
        for: 5m
        labels:
          severity: critical
          service: orchestrator
        annotations:
          summary: "Orchestrator leader election failed"
          description: "No leader elected in cluster"
```

### 2. Validate Alert Rules

```bash
# Check rule syntax
/opt/prometheus/promtool check rules /etc/prometheus/rules/platform-alerts.yml

# Reload Prometheus with new rules (without restart)
curl -X POST http://localhost:9090/-/reload
```

---

## AlertManager Configuration

### 1. Create AlertManager Config

```yaml
# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'YOUR_SLACK_WEBHOOK_URL'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

route:
  receiver: 'platform-notifications'
  group_by: ['alertname', 'service', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h

  routes:
    # Critical alerts go to PagerDuty
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 0s
      repeat_interval: 5m

    # Warnings go to Slack
    - match:
        severity: warning
      receiver: 'slack-warnings'
      repeat_interval: 1h

    # Service-specific routing
    - match:
        service: vault
      receiver: 'vault-team'
      group_by: ['service', 'severity']

    - match:
        service: orchestrator
      receiver: 'orchestrator-team'
      group_by: ['service', 'severity']

receivers:
  - name: 'platform-notifications'
    slack_configs:
      - channel: '#platform-alerts'
        title: 'Platform Alert'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#platform-warnings'
        title: 'Warning: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        description: '{{ .GroupLabels.alertname }}'
        details:
          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'

  - name: 'vault-team'
    email_configs:
      - to: 'vault-team@company.com'
        from: 'alertmanager@company.com'
        smarthost: 'smtp.company.com:587'
        auth_username: 'alerts@company.com'
        auth_password: 'PASSWORD'
        headers:
          Subject: 'Vault Alert: {{ .GroupLabels.alertname }}'

  - name: 'orchestrator-team'
    email_configs:
      - to: 'orchestrator-team@company.com'
        from: 'alertmanager@company.com'
        smarthost: 'smtp.company.com:587'

inhibit_rules:
  # Don't alert on errors if service is already down
  - source_match:
      severity: 'critical'
      alertname: 'ServiceDown'
    target_match_re:
      severity: 'warning|info'
    equal: ['service', 'instance']

  # Don't alert on resource exhaustion if service is down
  - source_match:
      alertname: 'ServiceDown'
    target_match_re:
      alertname: 'HighMemoryUsage|HighCPUUsage'
    equal: ['instance']
```

### 2. Start AlertManager

```bash
cd /opt/alertmanager
sudo ./alertmanager --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager

# Or as systemd service
sudo tee /etc/systemd/system/alertmanager.service > /dev/null << EOF
[Unit]
Description=AlertManager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Type=simple
ExecStart=/opt/alertmanager/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager

Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
```

### 3. Verify AlertManager

```bash
# Check AlertManager is running
curl -s http://localhost:9093/-/healthy

# List active alerts
curl -s http://localhost:9093/api/v1/alerts | jq .

# Check configuration
curl -s http://localhost:9093/api/v1/status | jq .
```

---

## Grafana Dashboards

### 1. Install Grafana

```bash
# Install Grafana
sudo apt-get install -y grafana-server

# Start Grafana
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

# Access at http://localhost:3000
# Default: admin/admin
```

### 2. Add Prometheus Data Source

```bash
# Via API
curl -X POST http://localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://localhost:9090",
    "access": "proxy",
    "isDefault": true
  }'
```

### 3. Create Platform Overview Dashboard

```json
{
  "dashboard": {
    "title": "Platform Overview",
    "description": "9-service provisioning platform metrics",
    "tags": ["platform", "overview"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Service Status",
        "type": "stat",
        "targets": [
          {
            "expr": "up{job=~\"vault-service|registry|rag|ai-service|orchestrator|control-center|mcp-server\"}"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "mappings": [
              {
                "type": "value",
                "value": "1",
                "text": "UP"
              },
              {
                "type": "value",
                "value": "0",
                "text": "DOWN"
              }
            ]
          }
        }
      },
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])"
          }
        ]
      },
      {
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"5..\"}[5m])"
          }
        ]
      },
      {
        "title": "Latency (p95)",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))"
          }
        ]
      },
      {
        "title": "Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "container_memory_usage_bytes / 1024 / 1024"
          }
        ]
      },
      {
        "title": "Disk Usage",
        "type": "gauge",
        "targets": [
          {
            "expr": "(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100"
          }
        ]
      }
    ]
  }
}
```

### 4. Import Dashboard via API

```bash
# Save dashboard JSON to file
cat > platform-overview.json << 'EOF'
{
  "dashboard": { ... }
}
EOF

# Import dashboard
curl -X POST http://localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -d @platform-overview.json
```

---

## Health Check Monitoring

### 1. Service Health Check Script

```bash
#!/bin/bash
# scripts/check-service-health.sh

SERVICES=(
  "vault:8200"
  "registry:8081"
  "rag:8083"
  "ai-service:8082"
  "orchestrator:9090"
  "control-center:8080"
  "mcp-server:8084"
)

UNHEALTHY=0

for service in "${SERVICES[@]}"; do
  IFS=':' read -r name port <<< "$service"

  response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:$port/health)

  if [ "$response" = "200" ]; then
    echo "✓ $name is healthy"
  else
    echo "✗ $name is UNHEALTHY (HTTP $response)"
    ((UNHEALTHY++))
  fi
done

if [ $UNHEALTHY -gt 0 ]; then
  echo ""
  echo "WARNING: $UNHEALTHY service(s) unhealthy"
  exit 1
fi

exit 0
```

### 2. Liveness Probe Configuration

```yaml
# For Kubernetes deployments
apiVersion: v1
kind: Pod
metadata:
  name: vault-service
spec:
  containers:
  - name: vault-service
    image: vault-service:latest
    livenessProbe:
      httpGet:
        path: /health
        port: 8200
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 3

    readinessProbe:
      httpGet:
        path: /health
        port: 8200
      initialDelaySeconds: 10
      periodSeconds: 5
      failureThreshold: 2
```

---

## Log Aggregation (ELK Stack)

### 1. Elasticsearch Setup

```bash
# Install Elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.0-linux-x86_64.tar.gz
tar xvfz elasticsearch-8.11.0-linux-x86_64.tar.gz
cd elasticsearch-8.11.0/bin
./elasticsearch
```

### 2. Filebeat Configuration

```yaml
# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/provisioning/*.log
    fields:
      service: provisioning-platform
      environment: production

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "changeme"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
```

### 3. Kibana Dashboard

```bash
# Access at http://localhost:5601
# Create index pattern: provisioning-*
# Create visualizations for:
# - Error rate over time
# - Service availability
# - Performance metrics
# - Request volume
```

---

## Monitoring Dashboard Queries

### Common Prometheus Queries

```promql
# Service availability (last hour)
avg(increase(up[1h])) by (job)

# Request rate per service
sum(rate(http_requests_total[5m])) by (job)

# Error rate per service
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)

# Latency percentiles
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# Memory usage per service
container_memory_usage_bytes / 1024 / 1024 / 1024

# CPU usage per service
rate(container_cpu_usage_seconds_total[5m]) * 100

# Disk I/O operations
rate(node_disk_io_time_seconds_total[5m])

# Network throughput
rate(node_network_transmit_bytes_total[5m])

# Queue depth (Orchestrator)
orchestrator_queue_depth

# Task processing rate
rate(orchestrator_tasks_total[5m])

# Task failure rate
rate(orchestrator_tasks_failed_total[5m])

# Cache hit ratio
rate(service_cache_hits_total[5m]) / (rate(service_cache_hits_total[5m]) + rate(service_cache_misses_total[5m]))

# Database connection pool status
database_connection_pool_usage{job="orchestrator"}

# TLS certificate expiration
(ssl_certificate_expiry - time()) / 86400
```

---

## Alert Testing

### 1. Test Alert Firing

```bash
# Manually fire test alert
curl -X POST http://localhost:9093/api/v1/alerts \
  -H 'Content-Type: application/json' \
  -d '[
    {
      "status": "firing",
      "labels": {
        "alertname": "TestAlert",
        "severity": "critical"
      },
      "annotations": {
        "summary": "This is a test alert",
        "description": "Test alert to verify notification routing"
      }
    }
  ]'
```

### 2. Stop Service to Trigger Alert

```bash
# Stop a service to trigger ServiceDown alert
pkill -9 vault-service

# Within 5 minutes, alert should fire
# Check AlertManager UI: http://localhost:9093

# Restart service
cargo run --release -p vault-service &

# Alert should resolve after service is back up
```

### 3. Generate Load to Test Error Alerts

```bash
# Generate request load
ab -n 10000 -c 100 http://localhost:9090/api/v1/health

# Monitor error rate in Prometheus
curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .
```

---

## Backup & Retention Policies

### 1. Prometheus Data Backup

```bash
#!/bin/bash
# scripts/backup-prometheus-data.sh

BACKUP_DIR="/backups/prometheus"
RETENTION_DAYS=30

# Create snapshot
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot

# Backup snapshot
SNAPSHOT=$(ls -t /var/lib/prometheus/snapshots | head -1)
tar -czf "$BACKUP_DIR/prometheus-$SNAPSHOT.tar.gz" \
  "/var/lib/prometheus/snapshots/$SNAPSHOT"

# Upload to S3
aws s3 cp "$BACKUP_DIR/prometheus-$SNAPSHOT.tar.gz" \
  s3://backups/prometheus/

# Clean old backups
find "$BACKUP_DIR" -mtime +$RETENTION_DAYS -delete
```

### 2. Prometheus Retention Configuration

```bash
# Keep metrics for 15 days
/opt/prometheus/prometheus \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.retention.size=50 GB
```

---

## Maintenance & Troubleshooting

### Common Issues

#### Prometheus Won't Scrape Service

```bash
# Check configuration
/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml

# Verify service is accessible
curl http://localhost:8200/metrics

# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="vault-service")'

# Check scrape error
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | .lastError'
```

#### AlertManager Not Sending Notifications

```bash
# Verify AlertManager config
/opt/alertmanager/amtool config routes

# Test webhook
curl -X POST http://localhost:3012/ -d '{"test": "alert"}'

# Check AlertManager logs
journalctl -u alertmanager -n 100 -f

# Verify notification channels configured
curl -s http://localhost:9093/api/v1/receivers
```

#### High Memory Usage

```bash
# Reduce Prometheus retention
prometheus --storage.tsdb.retention.time=7d --storage.tsdb.max-block-duration=2h

# Disable unused scrape jobs
# Edit prometheus.yml and remove unused jobs

# Monitor memory
ps aux | grep prometheus | grep -v grep
```

---

## Production Deployment Checklist

- [ ] Prometheus installed and running
- [ ] AlertManager installed and running
- [ ] Grafana installed and configured
- [ ] Prometheus scraping all 8 services
- [ ] Alert rules deployed and validated
- [ ] Notification channels configured (Slack, email, PagerDuty)
- [ ] AlertManager webhooks tested
- [ ] Grafana dashboards created
- [ ] Log aggregation stack deployed (optional)
- [ ] Backup scripts configured
- [ ] Retention policies set
- [ ] Health checks configured
- [ ] Team notified of alerting setup
- [ ] Runbooks created for common alerts
- [ ] Alert testing procedure documented

---

## Quick Commands Reference

```bash
# Prometheus
curl http://localhost:9090/api/v1/targets           # List scrape targets
curl 'http://localhost:9090/api/v1/query?query=up' # Query metric
curl -X POST http://localhost:9090/-/reload         # Reload config

# AlertManager
curl http://localhost:9093/api/v1/alerts            # List active alerts
curl http://localhost:9093/api/v1/receivers         # List receivers
curl http://localhost:9093/api/v2/status            # Check status

# Grafana
curl -u admin:admin http://localhost:3000/api/datasources  # List data sources
curl -u admin:admin http://localhost:3000/api/dashboards   # List dashboards

# Validation
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/platform-alerts.yml
amtool config routes
```

---

## Documentation & Runbooks

### Sample Runbook: Service Down

```markdown
# Service Down Alert

## Detection
Alert fires when service is unreachable for 5+ minutes

## Immediate Actions
1. Check service is running: pgrep -f service-name
2. Check service port: ss -tlnp | grep 8200
3. Check service logs: tail -100 /var/log/provisioning/service.log

## Diagnosis
1. Service crashed: look for panic/error in logs
2. Port conflict: lsof -i :8200
3. Configuration issue: validate config file
4. Dependency down: check database/cache connectivity

## Remediation
1. Restart service: pkill service && cargo run --release -p service &
2. Check health: curl http://localhost:8200/health
3. Verify dependencies: curl http://localhost:5432/health

## Escalation
If service doesn't recover after restart, escalate to on-call engineer
```

---

## Resources

- [Prometheus Documentation](https://prometheus.io/docs/)
- [AlertManager Documentation](https://prometheus.io/docs/alerting/latest/overview/)
- [Grafana Documentation](https://grafana.com/docs/)
- [Platform Deployment Guide](deployment-guide.md)
- [Service Management Guide](service-management-guide.md)

---

**Last Updated**: 2026-01-05
**Version**: 1.0.0
**Status**: Production Ready ✅
