# Service Monitoring & Alerting Setup\n\n**Complete guide for monitoring the 9-service platform with Prometheus, Grafana, and AlertManager**\n\n**Version**: 1.0.0\n**Last Updated**: 2026-01-05\n**Target Audience**: DevOps Engineers, Platform Operators\n**Status**: Production Ready\n\n---\n\n## Overview\n\nThis guide provides complete setup instructions for monitoring and alerting on the provisioning platform using industry-standard tools:\n\n- **Prometheus**: Metrics collection and time-series database\n- **Grafana**: Visualization and dashboarding\n- **AlertManager**: Alert routing and notification\n\n---\n\n## Architecture\n\n```\nServices (metrics endpoints)\n    ↓\nPrometheus (scrapes every 30s)\n    ↓\nAlertManager (evaluates rules)\n    ↓\nNotification Channels (email, slack, pagerduty)\n\nPrometheus Data\n    ↓\nGrafana (queries)\n    ↓\nDashboards & Visualization\n```\n\n---\n\n## Prerequisites\n\n### Software Requirements\n\n```\n# Prometheus (for metrics)\nwget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz\ntar xvfz prometheus-2.48.0.linux-amd64.tar.gz\nsudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus\n\n# Grafana (for dashboards)\nsudo apt-get install -y grafana-server\n\n# AlertManager (for alerting)\nwget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz\ntar xvfz alertmanager-0.26.0.linux-amd64.tar.gz\nsudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager\n```\n\n### System Requirements\n\n- **CPU**: 2+ cores\n- **Memory**: 4 GB minimum, 8 GB recommended\n- **Disk**: 100 GB for metrics retention (30 days)\n- **Network**: Access to all service endpoints\n\n### Ports\n\n| Component | Port | Purpose |\n| ----------- | ------ | --------- |\n| Prometheus | 9090 | Web UI & API |\n| Grafana | 3000 | Web UI |\n| AlertManager | 9093 | Web UI & API |\n| Node Exporter | 9100 | System metrics |\n\n---\n\n## Service Metrics Endpoints\n\nAll platform services expose metrics on the `/metrics` endpoint:\n\n```\n# Health and metrics endpoints for each service\ncurl http://localhost:8200/health    # Vault health\ncurl http://localhost:8200/metrics   # Vault metrics (Prometheus format)\n\ncurl http://localhost:8081/health    # Registry health\ncurl http://localhost:8081/metrics   # Registry metrics\n\ncurl http://localhost:8083/health    # RAG health\ncurl http://localhost:8083/metrics   # RAG metrics\n\ncurl http://localhost:8082/health    # AI Service health\ncurl http://localhost:8082/metrics   # AI Service metrics\n\ncurl http://localhost:9090/health    # Orchestrator health\ncurl http://localhost:9090/metrics   # Orchestrator metrics\n\ncurl http://localhost:8080/health    # Control Center health\ncurl http://localhost:8080/metrics   # Control Center metrics\n\ncurl http://localhost:8084/health    # MCP Server health\ncurl http://localhost:8084/metrics   # MCP Server metrics\n```\n\n---\n\n## Prometheus Configuration\n\n### 1. Create Prometheus Config\n\n```\n# /etc/prometheus/prometheus.yml\nglobal:\n  scrape_interval: 30s\n  evaluation_interval: 30s\n  external_labels:\n    monitor: 'provisioning-platform'\n    environment: 'production'\n\nalerting:\n  alertmanagers:\n    - static_configs:\n        - targets:\n            - localhost:9093\n\nrule_files:\n  - '/etc/prometheus/rules/*.yml'\n\nscrape_configs:\n  # Core Platform Services\n  - job_name: 'vault-service'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:8200']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'vault-service'\n\n  - job_name: 'extension-registry'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:8081']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'registry'\n\n  - job_name: 'rag-service'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:8083']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'rag'\n\n  - job_name: 'ai-service'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:8082']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'ai-service'\n\n  - job_name: 'orchestrator'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:9090']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'orchestrator'\n\n  - job_name: 'control-center'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:8080']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'control-center'\n\n  - job_name: 'mcp-server'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['localhost:8084']\n    relabel_configs:\n      - source_labels: [__address__]\n        target_label: instance\n        replacement: 'mcp-server'\n\n  # System Metrics (Node Exporter)\n  - job_name: 'node'\n    static_configs:\n      - targets: ['localhost:9100']\n        labels:\n          instance: 'system'\n\n  # SurrealDB (if multiuser/enterprise)\n  - job_name: 'surrealdb'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['surrealdb:8000']\n\n  # Etcd (if enterprise)\n  - job_name: 'etcd'\n    metrics_path: '/metrics'\n    static_configs:\n      - targets: ['etcd:2379']\n```\n\n### 2. Start Prometheus\n\n```\n# Create necessary directories\nsudo mkdir -p /etc/prometheus /var/lib/prometheus\nsudo mkdir -p /etc/prometheus/rules\n\n# Start Prometheus\ncd /opt/prometheus\nsudo ./prometheus --config.file=/etc/prometheus/prometheus.yml \\n  --storage.tsdb.path=/var/lib/prometheus \\n  --web.console.templates=consoles \\n  --web.console.libraries=console_libraries\n\n# Or as systemd service\nsudo tee /etc/systemd/system/prometheus.service > /dev/null << EOF\n[Unit]\nDescription=Prometheus\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nUser=prometheus\nType=simple\nExecStart=/opt/prometheus/prometheus \\n  --config.file=/etc/prometheus/prometheus.yml \\n  --storage.tsdb.path=/var/lib/prometheus\n\nRestart=on-failure\nRestartSec=10\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\nsudo systemctl daemon-reload\nsudo systemctl enable prometheus\nsudo systemctl start prometheus\n```\n\n### 3. Verify Prometheus\n\n```\n# Check Prometheus is running\ncurl -s http://localhost:9090/-/healthy\n\n# List scraped targets\ncurl -s http://localhost:9090/api/v1/targets | jq .\n\n# Query test metric\ncurl -s 'http://localhost:9090/api/v1/query?query=up' | jq .\n```\n\n---\n\n## Alert Rules Configuration\n\n### 1. Create Alert Rules\n\n```\n# /etc/prometheus/rules/platform-alerts.yml\ngroups:\n  - name: platform_availability\n    interval: 30s\n    rules:\n      - alert: ServiceDown\n        expr: up{job=~"vault-service|registry|rag|ai-service|orchestrator"} == 0\n        for: 5m\n        labels:\n          severity: critical\n          service: '{{ $labels.job }}'\n        annotations:\n          summary: "{{ $labels.job }} is DOWN"\n          description: "{{ $labels.job }} has been down for 5+ minutes"\n\n      - alert: ServiceSlowResponse\n        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1\n        for: 5m\n        labels:\n          severity: warning\n          service: '{{ $labels.job }}'\n        annotations:\n          summary: "{{ $labels.job }} slow response times"\n          description: "95th percentile latency above 1 second"\n\n  - name: platform_errors\n    interval: 30s\n    rules:\n      - alert: HighErrorRate\n        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05\n        for: 5m\n        labels:\n          severity: warning\n          service: '{{ $labels.job }}'\n        annotations:\n          summary: "{{ $labels.job }} high error rate"\n          description: "Error rate above 5% for 5 minutes"\n\n      - alert: DatabaseConnectionError\n        expr: increase(database_connection_errors_total[5m]) > 10\n        for: 2m\n        labels:\n          severity: critical\n          component: database\n        annotations:\n          summary: "Database connection failures detected"\n          description: "{{ $value }} connection errors in last 5 minutes"\n\n      - alert: QueueBacklog\n        expr: orchestrator_queue_depth > 1000\n        for: 5m\n        labels:\n          severity: warning\n          component: orchestrator\n        annotations:\n          summary: "Orchestrator queue backlog growing"\n          description: "Queue depth: {{ $value }} tasks"\n\n  - name: platform_resources\n    interval: 30s\n    rules:\n      - alert: HighMemoryUsage\n        expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9\n        for: 5m\n        labels:\n          severity: warning\n          resource: memory\n        annotations:\n          summary: "{{ $labels.container_name }} memory usage critical"\n          description: "Memory usage: {{ $value | humanizePercentage }}"\n\n      - alert: HighDiskUsage\n        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes < 0.1\n        for: 5m\n        labels:\n          severity: warning\n          resource: disk\n        annotations:\n          summary: "Disk space critically low"\n          description: "Available disk space: {{ $value | humanizePercentage }}"\n\n      - alert: HighCPUUsage\n        expr: (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)) > 0.9\n        for: 10m\n        labels:\n          severity: warning\n          resource: cpu\n        annotations:\n          summary: "High CPU usage detected"\n          description: "CPU usage: {{ $value | humanizePercentage }}"\n\n      - alert: DiskIOLatency\n        expr: node_disk_io_time_seconds_total > 100\n        for: 5m\n        labels:\n          severity: warning\n          resource: disk\n        annotations:\n          summary: "High disk I/O latency"\n          description: "I/O latency: {{ $value }}ms"\n\n  - name: platform_network\n    interval: 30s\n    rules:\n      - alert: HighNetworkLatency\n        expr: probe_duration_seconds > 0.5\n        for: 5m\n        labels:\n          severity: warning\n          component: network\n        annotations:\n          summary: "High network latency detected"\n          description: "Latency: {{ $value }}ms"\n\n      - alert: PacketLoss\n        expr: node_network_transmit_errors_total > 100\n        for: 5m\n        labels:\n          severity: warning\n          component: network\n        annotations:\n          summary: "Packet loss detected"\n          description: "Transmission errors: {{ $value }}"\n\n  - name: platform_services\n    interval: 30s\n    rules:\n      - alert: VaultSealed\n        expr: vault_core_unsealed == 0\n        for: 1m\n        labels:\n          severity: critical\n          service: vault\n        annotations:\n          summary: "Vault is sealed"\n          description: "Vault instance is sealed and requires unseal operation"\n\n      - alert: RegistryAuthError\n        expr: increase(registry_auth_failures_total[5m]) > 5\n        for: 2m\n        labels:\n          severity: warning\n          service: registry\n        annotations:\n          summary: "Registry authentication failures"\n          description: "{{ $value }} auth failures in last 5 minutes"\n\n      - alert: RAGVectorDBDown\n        expr: rag_vectordb_connection_status == 0\n        for: 2m\n        labels:\n          severity: critical\n          service: rag\n        annotations:\n          summary: "RAG Vector Database disconnected"\n          description: "Vector DB connection lost"\n\n      - alert: AIServiceMCPError\n        expr: increase(ai_service_mcp_errors_total[5m]) > 10\n        for: 2m\n        labels:\n          severity: warning\n          service: ai_service\n        annotations:\n          summary: "AI Service MCP integration errors"\n          description: "{{ $value }} errors in last 5 minutes"\n\n      - alert: OrchestratorLeaderElectionIssue\n        expr: orchestrator_leader_elected == 0\n        for: 5m\n        labels:\n          severity: critical\n          service: orchestrator\n        annotations:\n          summary: "Orchestrator leader election failed"\n          description: "No leader elected in cluster"\n```\n\n### 2. Validate Alert Rules\n\n```\n# Check rule syntax\n/opt/prometheus/promtool check rules /etc/prometheus/rules/platform-alerts.yml\n\n# Reload Prometheus with new rules (without restart)\ncurl -X POST http://localhost:9090/-/reload\n```\n\n---\n\n## AlertManager Configuration\n\n### 1. Create AlertManager Config\n\n```\n# /etc/alertmanager/alertmanager.yml\nglobal:\n  resolve_timeout: 5m\n  slack_api_url: 'YOUR_SLACK_WEBHOOK_URL'\n  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'\n\nroute:\n  receiver: 'platform-notifications'\n  group_by: ['alertname', 'service', 'severity']\n  group_wait: 10s\n  group_interval: 10s\n  repeat_interval: 12h\n\n  routes:\n    # Critical alerts go to PagerDuty\n    - match:\n        severity: critical\n      receiver: 'pagerduty-critical'\n      group_wait: 0s\n      repeat_interval: 5m\n\n    # Warnings go to Slack\n    - match:\n        severity: warning\n      receiver: 'slack-warnings'\n      repeat_interval: 1h\n\n    # Service-specific routing\n    - match:\n        service: vault\n      receiver: 'vault-team'\n      group_by: ['service', 'severity']\n\n    - match:\n        service: orchestrator\n      receiver: 'orchestrator-team'\n      group_by: ['service', 'severity']\n\nreceivers:\n  - name: 'platform-notifications'\n    slack_configs:\n      - channel: '#platform-alerts'\n        title: 'Platform Alert'\n        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'\n        send_resolved: true\n\n  - name: 'slack-warnings'\n    slack_configs:\n      - channel: '#platform-warnings'\n        title: 'Warning: {{ .GroupLabels.alertname }}'\n        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'\n\n  - name: 'pagerduty-critical'\n    pagerduty_configs:\n      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'\n        description: '{{ .GroupLabels.alertname }}'\n        details:\n          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'\n\n  - name: 'vault-team'\n    email_configs:\n      - to: 'vault-team@company.com'\n        from: 'alertmanager@company.com'\n        smarthost: 'smtp.company.com:587'\n        auth_username: 'alerts@company.com'\n        auth_password: 'PASSWORD'\n        headers:\n          Subject: 'Vault Alert: {{ .GroupLabels.alertname }}'\n\n  - name: 'orchestrator-team'\n    email_configs:\n      - to: 'orchestrator-team@company.com'\n        from: 'alertmanager@company.com'\n        smarthost: 'smtp.company.com:587'\n\ninhibit_rules:\n  # Don't alert on errors if service is already down\n  - source_match:\n      severity: 'critical'\n      alertname: 'ServiceDown'\n    target_match_re:\n      severity: 'warning|info'\n    equal: ['service', 'instance']\n\n  # Don't alert on resource exhaustion if service is down\n  - source_match:\n      alertname: 'ServiceDown'\n    target_match_re:\n      alertname: 'HighMemoryUsage|HighCPUUsage'\n    equal: ['instance']\n```\n\n### 2. Start AlertManager\n\n```\ncd /opt/alertmanager\nsudo ./alertmanager --config.file=/etc/alertmanager/alertmanager.yml \\n  --storage.path=/var/lib/alertmanager\n\n# Or as systemd service\nsudo tee /etc/systemd/system/alertmanager.service > /dev/null << EOF\n[Unit]\nDescription=AlertManager\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nUser=alertmanager\nType=simple\nExecStart=/opt/alertmanager/alertmanager \\n  --config.file=/etc/alertmanager/alertmanager.yml \\n  --storage.path=/var/lib/alertmanager\n\nRestart=on-failure\nRestartSec=10\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\nsudo systemctl daemon-reload\nsudo systemctl enable alertmanager\nsudo systemctl start alertmanager\n```\n\n### 3. Verify AlertManager\n\n```\n# Check AlertManager is running\ncurl -s http://localhost:9093/-/healthy\n\n# List active alerts\ncurl -s http://localhost:9093/api/v1/alerts | jq .\n\n# Check configuration\ncurl -s http://localhost:9093/api/v1/status | jq .\n```\n\n---\n\n## Grafana Dashboards\n\n### 1. Install Grafana\n\n```\n# Install Grafana\nsudo apt-get install -y grafana-server\n\n# Start Grafana\nsudo systemctl enable grafana-server\nsudo systemctl start grafana-server\n\n# Access at http://localhost:3000\n# Default: admin/admin\n```\n\n### 2. Add Prometheus Data Source\n\n```\n# Via API\ncurl -X POST http://localhost:3000/api/datasources \\n  -H "Content-Type: application/json" \\n  -u admin:admin \\n  -d '{\n    "name": "Prometheus",\n    "type": "prometheus",\n    "url": "http://localhost:9090",\n    "access": "proxy",\n    "isDefault": true\n  }'\n```\n\n### 3. Create Platform Overview Dashboard\n\n```\n{\n  "dashboard": {\n    "title": "Platform Overview",\n    "description": "9-service provisioning platform metrics",\n    "tags": ["platform", "overview"],\n    "timezone": "browser",\n    "panels": [\n      {\n        "title": "Service Status",\n        "type": "stat",\n        "targets": [\n          {\n            "expr": "up{job=~\"vault-service|registry|rag|ai-service|orchestrator|control-center|mcp-server\"}"\n          }\n        ],\n        "fieldConfig": {\n          "defaults": {\n            "mappings": [\n              {\n                "type": "value",\n                "value": "1",\n                "text": "UP"\n              },\n              {\n                "type": "value",\n                "value": "0",\n                "text": "DOWN"\n              }\n            ]\n          }\n        }\n      },\n      {\n        "title": "Request Rate",\n        "type": "graph",\n        "targets": [\n          {\n            "expr": "rate(http_requests_total[5m])"\n          }\n        ]\n      },\n      {\n        "title": "Error Rate",\n        "type": "graph",\n        "targets": [\n          {\n            "expr": "rate(http_requests_total{status=~\"5..\"}[5m])"\n          }\n        ]\n      },\n      {\n        "title": "Latency (p95)",\n        "type": "graph",\n        "targets": [\n          {\n            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))"\n          }\n        ]\n      },\n      {\n        "title": "Memory Usage",\n        "type": "graph",\n        "targets": [\n          {\n            "expr": "container_memory_usage_bytes / 1024 / 1024"\n          }\n        ]\n      },\n      {\n        "title": "Disk Usage",\n        "type": "gauge",\n        "targets": [\n          {\n            "expr": "(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100"\n          }\n        ]\n      }\n    ]\n  }\n}\n```\n\n### 4. Import Dashboard via API\n\n```\n# Save dashboard JSON to file\ncat > platform-overview.json << 'EOF'\n{\n  "dashboard": { ... }\n}\nEOF\n\n# Import dashboard\ncurl -X POST http://localhost:3000/api/dashboards/db \\n  -H "Content-Type: application/json" \\n  -u admin:admin \\n  -d @platform-overview.json\n```\n\n---\n\n## Health Check Monitoring\n\n### 1. Service Health Check Script\n\n```\n#!/bin/bash\n# scripts/check-service-health.sh\n\nSERVICES=(\n  "vault:8200"\n  "registry:8081"\n  "rag:8083"\n  "ai-service:8082"\n  "orchestrator:9090"\n  "control-center:8080"\n  "mcp-server:8084"\n)\n\nUNHEALTHY=0\n\nfor service in "${SERVICES[@]}"; do\n  IFS=':' read -r name port <<< "$service"\n\n  response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:$port/health)\n\n  if [ "$response" = "200" ]; then\n    echo "✓ $name is healthy"\n  else\n    echo "✗ $name is UNHEALTHY (HTTP $response)"\n    ((UNHEALTHY++))\n  fi\ndone\n\nif [ $UNHEALTHY -gt 0 ]; then\n  echo ""\n  echo "WARNING: $UNHEALTHY service(s) unhealthy"\n  exit 1\nfi\n\nexit 0\n```\n\n### 2. Liveness Probe Configuration\n\n```\n# For Kubernetes deployments\napiVersion: v1\nkind: Pod\nmetadata:\n  name: vault-service\nspec:\n  containers:\n  - name: vault-service\n    image: vault-service:latest\n    livenessProbe:\n      httpGet:\n        path: /health\n        port: 8200\n      initialDelaySeconds: 30\n      periodSeconds: 10\n      failureThreshold: 3\n\n    readinessProbe:\n      httpGet:\n        path: /health\n        port: 8200\n      initialDelaySeconds: 10\n      periodSeconds: 5\n      failureThreshold: 2\n```\n\n---\n\n## Log Aggregation (ELK Stack)\n\n### 1. Elasticsearch Setup\n\n```\n# Install Elasticsearch\nwget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.0-linux-x86_64.tar.gz\ntar xvfz elasticsearch-8.11.0-linux-x86_64.tar.gz\ncd elasticsearch-8.11.0/bin\n./elasticsearch\n```\n\n### 2. Filebeat Configuration\n\n```\n# /etc/filebeat/filebeat.yml\nfilebeat.inputs:\n  - type: log\n    enabled: true\n    paths:\n      - /var/log/provisioning/*.log\n    fields:\n      service: provisioning-platform\n      environment: production\n\noutput.elasticsearch:\n  hosts: ["localhost:9200"]\n  username: "elastic"\n  password: "changeme"\n\nlogging.level: info\nlogging.to_files: true\nlogging.files:\n  path: /var/log/filebeat\n```\n\n### 3. Kibana Dashboard\n\n```\n# Access at http://localhost:5601\n# Create index pattern: provisioning-*\n# Create visualizations for:\n# - Error rate over time\n# - Service availability\n# - Performance metrics\n# - Request volume\n```\n\n---\n\n## Monitoring Dashboard Queries\n\n### Common Prometheus Queries\n\n```\n# Service availability (last hour)\navg(increase(up[1h])) by (job)\n\n# Request rate per service\nsum(rate(http_requests_total[5m])) by (job)\n\n# Error rate per service\nsum(rate(http_requests_total{status=~"5.."}[5m])) by (job)\n\n# Latency percentiles\nhistogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))\nhistogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))\n\n# Memory usage per service\ncontainer_memory_usage_bytes / 1024 / 1024 / 1024\n\n# CPU usage per service\nrate(container_cpu_usage_seconds_total[5m]) * 100\n\n# Disk I/O operations\nrate(node_disk_io_time_seconds_total[5m])\n\n# Network throughput\nrate(node_network_transmit_bytes_total[5m])\n\n# Queue depth (Orchestrator)\norchestrator_queue_depth\n\n# Task processing rate\nrate(orchestrator_tasks_total[5m])\n\n# Task failure rate\nrate(orchestrator_tasks_failed_total[5m])\n\n# Cache hit ratio\nrate(service_cache_hits_total[5m]) / (rate(service_cache_hits_total[5m]) + rate(service_cache_misses_total[5m]))\n\n# Database connection pool status\ndatabase_connection_pool_usage{job="orchestrator"}\n\n# TLS certificate expiration\n(ssl_certificate_expiry - time()) / 86400\n```\n\n---\n\n## Alert Testing\n\n### 1. Test Alert Firing\n\n```\n# Manually fire test alert\ncurl -X POST http://localhost:9093/api/v1/alerts \\n  -H 'Content-Type: application/json' \\n  -d '[\n    {\n      "status": "firing",\n      "labels": {\n        "alertname": "TestAlert",\n        "severity": "critical"\n      },\n      "annotations": {\n        "summary": "This is a test alert",\n        "description": "Test alert to verify notification routing"\n      }\n    }\n  ]'\n```\n\n### 2. Stop Service to Trigger Alert\n\n```\n# Stop a service to trigger ServiceDown alert\npkill -9 vault-service\n\n# Within 5 minutes, alert should fire\n# Check AlertManager UI: http://localhost:9093\n\n# Restart service\ncargo run --release -p vault-service &\n\n# Alert should resolve after service is back up\n```\n\n### 3. Generate Load to Test Error Alerts\n\n```\n# Generate request load\nab -n 10000 -c 100 http://localhost:9090/api/v1/health\n\n# Monitor error rate in Prometheus\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .\n```\n\n---\n\n## Backup & Retention Policies\n\n### 1. Prometheus Data Backup\n\n```\n#!/bin/bash\n# scripts/backup-prometheus-data.sh\n\nBACKUP_DIR="/backups/prometheus"\nRETENTION_DAYS=30\n\n# Create snapshot\ncurl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot\n\n# Backup snapshot\nSNAPSHOT=$(ls -t /var/lib/prometheus/snapshots | head -1)\ntar -czf "$BACKUP_DIR/prometheus-$SNAPSHOT.tar.gz" \\n  "/var/lib/prometheus/snapshots/$SNAPSHOT"\n\n# Upload to S3\naws s3 cp "$BACKUP_DIR/prometheus-$SNAPSHOT.tar.gz" \\n  s3://backups/prometheus/\n\n# Clean old backups\nfind "$BACKUP_DIR" -mtime +$RETENTION_DAYS -delete\n```\n\n### 2. Prometheus Retention Configuration\n\n```\n# Keep metrics for 15 days\n/opt/prometheus/prometheus \\n  --storage.tsdb.retention.time=15d \\n  --storage.tsdb.retention.size=50 GB\n```\n\n---\n\n## Maintenance & Troubleshooting\n\n### Common Issues\n\n#### Prometheus Won't Scrape Service\n\n```\n# Check configuration\n/opt/prometheus/promtool check config /etc/prometheus/prometheus.yml\n\n# Verify service is accessible\ncurl http://localhost:8200/metrics\n\n# Check Prometheus targets\ncurl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="vault-service")'\n\n# Check scrape error\ncurl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | .lastError'\n```\n\n#### AlertManager Not Sending Notifications\n\n```\n# Verify AlertManager config\n/opt/alertmanager/amtool config routes\n\n# Test webhook\ncurl -X POST http://localhost:3012/ -d '{"test": "alert"}'\n\n# Check AlertManager logs\njournalctl -u alertmanager -n 100 -f\n\n# Verify notification channels configured\ncurl -s http://localhost:9093/api/v1/receivers\n```\n\n#### High Memory Usage\n\n```\n# Reduce Prometheus retention\nprometheus --storage.tsdb.retention.time=7d --storage.tsdb.max-block-duration=2h\n\n# Disable unused scrape jobs\n# Edit prometheus.yml and remove unused jobs\n\n# Monitor memory\nps aux | grep prometheus | grep -v grep\n```\n\n---\n\n## Production Deployment Checklist\n\n- [ ] Prometheus installed and running\n- [ ] AlertManager installed and running\n- [ ] Grafana installed and configured\n- [ ] Prometheus scraping all 8 services\n- [ ] Alert rules deployed and validated\n- [ ] Notification channels configured (Slack, email, PagerDuty)\n- [ ] AlertManager webhooks tested\n- [ ] Grafana dashboards created\n- [ ] Log aggregation stack deployed (optional)\n- [ ] Backup scripts configured\n- [ ] Retention policies set\n- [ ] Health checks configured\n- [ ] Team notified of alerting setup\n- [ ] Runbooks created for common alerts\n- [ ] Alert testing procedure documented\n\n---\n\n## Quick Commands Reference\n\n```\n# Prometheus\ncurl http://localhost:9090/api/v1/targets           # List scrape targets\ncurl 'http://localhost:9090/api/v1/query?query=up' # Query metric\ncurl -X POST http://localhost:9090/-/reload         # Reload config\n\n# AlertManager\ncurl http://localhost:9093/api/v1/alerts            # List active alerts\ncurl http://localhost:9093/api/v1/receivers         # List receivers\ncurl http://localhost:9093/api/v2/status            # Check status\n\n# Grafana\ncurl -u admin:admin http://localhost:3000/api/datasources  # List data sources\ncurl -u admin:admin http://localhost:3000/api/dashboards   # List dashboards\n\n# Validation\npromtool check config /etc/prometheus/prometheus.yml\npromtool check rules /etc/prometheus/rules/platform-alerts.yml\namtool config routes\n```\n\n---\n\n## Documentation & Runbooks\n\n### Sample Runbook: Service Down\n\n```\n# Service Down Alert\n\n## Detection\nAlert fires when service is unreachable for 5+ minutes\n\n## Immediate Actions\n1. Check service is running: pgrep -f service-name\n2. Check service port: ss -tlnp | grep 8200\n3. Check service logs: tail -100 /var/log/provisioning/service.log\n\n## Diagnosis\n1. Service crashed: look for panic/error in logs\n2. Port conflict: lsof -i :8200\n3. Configuration issue: validate config file\n4. Dependency down: check database/cache connectivity\n\n## Remediation\n1. Restart service: pkill service && cargo run --release -p service &\n2. Check health: curl http://localhost:8200/health\n3. Verify dependencies: curl http://localhost:5432/health\n\n## Escalation\nIf service doesn't recover after restart, escalate to on-call engineer\n```\n\n---\n\n## Resources\n\n- [Prometheus Documentation](https://prometheus.io/docs/)\n- [AlertManager Documentation](https://prometheus.io/docs/alerting/latest/overview/)\n- [Grafana Documentation](https://grafana.com/docs/)\n- [Platform Deployment Guide](deployment-guide.md)\n- [Service Management Guide](service-management-guide.md)\n\n---\n\n**Last Updated**: 2026-01-05\n**Version**: 1.0.0\n**Status**: Production Ready ✅
