<p align="center">
    <img src="../resources/provisioning_logo.svg" alt="Provisioning Logo" width="300"/>
</p>

<p align="center">
    <img src="../resources/logo-text.svg" alt="Provisioning" width="500"/>
</p>

# Operations

Production deployment, monitoring, maintenance, and operational best practices for running
Provisioning infrastructure at scale.

## Overview

This section covers everything needed to operate Provisioning in production:

- **Deployment strategies** - Single-cloud, multi-cloud, hybrid with zero-downtime updates
- **Service management** - Microservice lifecycle, scaling, health checks, failover
- **Observability** - Metrics (Prometheus), logs (ELK), traces (Jaeger), dashboards
- **Incident response** - Detection, triage, remediation, postmortem automation
- **Backup & recovery** - Strategies, testing, disaster recovery, point-in-time restore
- **Performance optimization** - Profiling, caching, scaling, resource optimization
- **Troubleshooting** - Debugging, log analysis, diagnostic tools, support

## Operational Guides

### Deployment and Management

- **[Deployment Modes](./deployment-modes.md)** - Single-cloud, multi-cloud, hybrid, canary, blue-green, rolling updates with zero downtime.

- **[Service Management](./service-management.md)** - Microservice lifecycle, scaling policies, health checks, graceful shutdown, rolling restarts.

- **[Platform Installer](./installer-system.md)** - TUI and unattended installation, provider setup, workspace creation, post-install configuration.

### Monitoring and Observability

<p align="center">
    <img src="../resources/diagrams/operations/monitoring-stack.svg"
         alt="Monitoring Stack Prometheus Grafana Fluentd ElasticSearch Alertmanager" width="800"/>
</p>

- **[Monitoring Setup](./monitoring.md)** - Prometheus metrics, Grafana
  dashboards, alerting rules, SLO monitoring, 12 microservices

- **[Logging and Analysis](./logging.md)** - Centralized logging with ELK Stack, log aggregation, filtering, searching, performance analysis.

- **[Distributed Tracing](./tracing.md)** - Jaeger integration, span collection, trace visualization, latency analysis across microservices.

### Resilience and Recovery

- **[Incident Response](./incident-response-procedures.md)** - Severity
  levels, triage, investigation, mitigation, escalation, postmortem

- **[Backup Strategies](./backup-strategies.md)** - Full, incremental, PITR backups with RTO/RPO targets, testing procedures, recovery workflows.

- **[Disaster Recovery](./disaster-recovery.md)** - DR planning, failover procedures, failback strategies, RTO/RPO targets, testing schedules.

<p align="center">
    <img src="../resources/diagrams/operations/disaster-recovery-topology.svg"
         alt="Disaster Recovery Topology Multi-Region Failover Primary Backup" width="850"/>
</p>

- **[Performance Optimization](./performance-optimization.md)** - Profiling,
  bottlenecks, caching strategies, connection pooling, right-sizing

### Troubleshooting

- **[Troubleshooting Guide](./troubleshooting.md)** - Common issues, debugging techniques, log analysis, diagnostic tools, support resources.

- **[Platform Health](./platform-health.md)** - Health check procedures, system status, component status, SLO metrics, error budgets.

## Operational Workflows

### I'm deploying to production
Follow: [Deployment Modes](./deployment-modes.md) → [Service Management](./service-management.md) → [Monitoring Setup](./monitoring.md)

### I need to monitor infrastructure
Setup: [Monitoring Setup](./monitoring.md) for metrics, [Logging and Analysis](./logging.md) for logs, [Distributed Tracing](./tracing.md) for traces

### I'm experiencing an incident
Execute: [Incident Response](./incident-response-procedures.md) with triage, investigation, mitigation, escalation

### I need to backup and recover
Implement: [Backup Strategies](./backup-strategies.md) with testing, [Disaster Recovery](./disaster-recovery.md) for major outages

### I need to optimize performance
Follow: [Performance Optimization](./performance-optimization.md) for profiling and tuning

### I need help troubleshooting
Consult: [Troubleshooting Guide](./troubleshooting.md) for common issues and solutions

## Deployment Architecture

```text
Development
  ↓
Staging (test all)
  ↓
Canary (1% traffic)
  ↓
Rolling (increase % gradually)
  ↓
Production (100%)
```

## SLO Targets

| Service | Availability | P99 Latency | Error Budget |
| --------| - ------------| - ------------| - -------------- |
| API Gateway | 99.99% | <100ms | 4m 26s/month |
| Orchestrator | 99.9% | <500ms | 43m 46s/month |
| Control-Center | 99.95% | <300ms | 21m 56s/month |
| Detector | 99.5% | <2s | 3h 36s/month |
| All Others | 99.9% | <1s | 43m 46s/month |

## Monitoring Stack

- **Metrics** - Prometheus (15s scrape interval, 15d retention)
- **Logs** - ELK Stack (Elasticsearch, Logstash, Kibana) with 30d retention
- **Traces** - Jaeger (sampling 10%, 24h retention)
- **Dashboards** - Grafana with pre-built dashboards per microservice
- **Alerting** - AlertManager with escalation rules and notification channels

## Operational Commands

```bash
# Check system health
provisioning status health

# View metrics
provisioning metrics view --service orchestrator

# Check SLO status
provisioning slo status

# Run diagnostics
provisioning diagnose system

# Backup infrastructure
provisioning backup create --name daily-$(date +%Y%m%d)

# Restore from backup
provisioning backup restore --backup-id backup-id
```

## Related Documentation

- **Architecture** → See `provisioning/docs/src/architecture/`
- **Features** → See `provisioning/docs/src/features/`
- **Development** → See `provisioning/docs/src/development/`
- **Security** → See `provisioning/docs/src/security/`
- **Examples** → See `provisioning/docs/src/examples/`
