# Orchestrator Authentication & Authorization Integration\n\n**Version**: 1.0.0\n**Date**: 2025-10-08\n**Status**: Implemented\n\n## Overview\n\nComplete authentication and authorization flow integration for the Provisioning Orchestrator, connecting all security components (JWT validation, MFA\nverification, Cedar authorization, rate limiting, and audit logging) into a cohesive security middleware chain.\n\n## Architecture\n\n### Security Middleware Chain\n\nThe middleware chain is applied in this specific order to ensure proper security:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                    Incoming HTTP Request                        │\n└────────────────────────┬────────────────────────────────────────┘\n                         │\n                         ▼\n        ┌────────────────────────────────┐\n        │  1. Rate Limiting Middleware   │\n        │  - Per-IP request limits       │\n        │  - Sliding window              │\n        │  - Exempt IPs                  │\n        └────────────┬───────────────────┘\n                     │ (429 if exceeded)\n                     ▼\n        ┌────────────────────────────────┐\n        │  2. Authentication Middleware  │\n        │  - Extract Bearer token        │\n        │  - Validate JWT signature      │\n        │  - Check expiry, issuer, aud   │\n        │  - Check revocation            │\n        └────────────┬───────────────────┘\n                     │ (401 if invalid)\n                     ▼\n        ┌────────────────────────────────┐\n        │  3. MFA Verification           │\n        │  - Check MFA status in token   │\n        │  - Enforce for sensitive ops   │\n        │  - Production deployments      │\n        │  - All DELETE operations       │\n        └────────────┬───────────────────┘\n                     │ (403 if required but missing)\n                     ▼\n        ┌────────────────────────────────┐\n        │  4. Authorization Middleware   │\n        │  - Build Cedar request         │\n        │  - Evaluate policies           │\n        │  - Check permissions           │\n        │  - Log decision                │\n        └────────────┬───────────────────┘\n                     │ (403 if denied)\n                     ▼\n        ┌────────────────────────────────┐\n        │  5. Audit Logging Middleware   │\n        │  - Log complete request        │\n        │  - User, action, resource      │\n        │  - Authorization decision      │\n        │  - Response status             │\n        └────────────┬───────────────────┘\n                     │\n                     ▼\n        ┌────────────────────────────────┐\n        │      Protected Handler         │\n        │  - Access security context     │\n        │  - Execute business logic      │\n        └────────────────────────────────┘\n```\n\n## Implementation Details\n\n### 1. Security Context Builder (`middleware/security_context.rs`)\n\n**Purpose**: Build complete security context from authenticated requests.\n\n**Key Features**:\n\n- Extracts JWT token claims\n- Determines MFA verification status\n- Extracts IP address (X-Forwarded-For, X-Real-IP)\n- Extracts user agent and session info\n- Provides permission checking methods\n\n**Lines of Code**: 275\n\n**Example**:\n\n```\npub struct SecurityContext {\n    pub user_id: String,\n    pub token: ValidatedToken,\n    pub mfa_verified: bool,\n    pub ip_address: IpAddr,\n    pub user_agent: Option<String>,\n    pub permissions: Vec<String>,\n    pub workspace: String,\n    pub request_id: String,\n    pub session_id: Option<String>,\n}\n\nimpl SecurityContext {\n    pub fn has_permission(&self, permission: &str) -> bool { ... }\n    pub fn has_any_permission(&self, permissions: &[&str]) -> bool { ... }\n    pub fn has_all_permissions(&self, permissions: &[&str]) -> bool { ... }\n}\n```\n\n### 2. Enhanced Authentication Middleware (`middleware/auth.rs`)\n\n**Purpose**: JWT token validation with revocation checking.\n\n**Key Features**:\n\n- Bearer token extraction\n- JWT signature validation (RS256)\n- Expiry, issuer, audience checks\n- Token revocation status\n- Security context injection\n\n**Lines of Code**: 245\n\n**Flow**:\n\n1. Extract `Authorization: Bearer <token>` header\n2. Validate JWT with TokenValidator\n3. Build SecurityContext\n4. Inject into request extensions\n5. Continue to next middleware or return 401\n\n**Error Responses**:\n\n- `401 Unauthorized`: Missing/invalid token, expired, revoked\n- `403 Forbidden`: Insufficient permissions\n\n### 3. MFA Verification Middleware (`middleware/mfa.rs`)\n\n**Purpose**: Enforce MFA for sensitive operations.\n\n**Key Features**:\n\n- Path-based MFA requirements\n- Method-based enforcement (all DELETEs)\n- Production environment protection\n- Clear error messages\n\n**Lines of Code**: 290\n\n**MFA Required For**:\n\n- Production deployments (`/production/`, `/prod/`)\n- All DELETE operations\n- Server operations (POST, PUT, DELETE)\n- Cluster operations (POST, PUT, DELETE)\n- Batch submissions\n- Rollback operations\n- Configuration changes (POST, PUT, DELETE)\n- Secret management\n- User/role management\n\n**Example**:\n\n```\nfn requires_mfa(method: &str, path: &str) -> bool {\n    if path.contains("/production/") { return true; }\n    if method == "DELETE" { return true; }\n    if path.contains("/deploy") { return true; }\n    // ...\n}\n```\n\n### 4. Enhanced Authorization Middleware (`middleware/authz.rs`)\n\n**Purpose**: Cedar policy evaluation with audit logging.\n\n**Key Features**:\n\n- Builds Cedar authorization request from HTTP request\n- Maps HTTP methods to Cedar actions (GET→Read, POST→Create, etc.)\n- Extracts resource types from paths\n- Evaluates Cedar policies with context (MFA, IP, time, workspace)\n- Logs all authorization decisions to audit log\n- Non-blocking audit logging (tokio::spawn)\n\n**Lines of Code**: 380\n\n**Resource Mapping**:\n\n```\n/api/v1/servers/srv-123    → Resource::Server("srv-123")\n/api/v1/taskserv/kubernetes → Resource::TaskService("kubernetes")\n/api/v1/cluster/prod        → Resource::Cluster("prod")\n/api/v1/config/settings     → Resource::Config("settings")\n```\n\n**Action Mapping**:\n\n```\nGET    → Action::Read\nPOST   → Action::Create\nPUT    → Action::Update\nDELETE → Action::Delete\n```\n\n### 5. Rate Limiting Middleware (`middleware/rate_limit.rs`)\n\n**Purpose**: Prevent API abuse with per-IP rate limiting.\n\n**Key Features**:\n\n- Sliding window rate limiting\n- Per-IP request tracking\n- Configurable limits and windows\n- Exempt IP support\n- Automatic cleanup of old entries\n- Statistics tracking\n\n**Lines of Code**: 420\n\n**Configuration**:\n\n```\npub struct RateLimitConfig {\n    pub max_requests: u32,          // for example, 100\n    pub window_duration: Duration,  // for example, 60 seconds\n    pub exempt_ips: Vec<IpAddr>,    // for example, internal services\n    pub enabled: bool,\n}\n\n// Default: 100 requests per minute\n```\n\n**Statistics**:\n\n```\npub struct RateLimitStats {\n    pub total_ips: usize,      // Number of tracked IPs\n    pub total_requests: u32,   // Total requests made\n    pub limited_ips: usize,    // IPs that hit the limit\n    pub config: RateLimitConfig,\n}\n```\n\n### 6. Security Integration Module (`security_integration.rs`)\n\n**Purpose**: Helper module to integrate all security components.\n\n**Key Features**:\n\n- `SecurityComponents` struct grouping all middleware\n- `SecurityConfig` for configuration\n- `initialize()` method to set up all components\n- `disabled()` method for development mode\n- `apply_security_middleware()` helper for router setup\n\n**Lines of Code**: 265\n\n**Usage Example**:\n\n```\nuse provisioning_orchestrator::security_integration::{\n    SecurityComponents, SecurityConfig\n};\n\n// Initialize security\nlet config = SecurityConfig {\n    public_key_path: PathBuf::from("keys/public.pem"),\n    jwt_issuer: "control-center".to_string(),\n    jwt_audience: "orchestrator".to_string(),\n    cedar_policies_path: PathBuf::from("policies"),\n    auth_enabled: true,\n    authz_enabled: true,\n    mfa_enabled: true,\n    rate_limit_config: RateLimitConfig::new(100, 60),\n};\n\nlet security = SecurityComponents::initialize(config, audit_logger).await?;\n\n// Apply to router\nlet app = Router::new()\n    .route("/api/v1/servers", post(create_server))\n    .route("/api/v1/servers/:id", delete(delete_server));\n\nlet secured_app = apply_security_middleware(app, &security);\n```\n\n## Integration with AppState\n\n### Updated AppState Structure\n\n```\npub struct AppState {\n    // Existing fields\n    pub task_storage: Arc<dyn TaskStorage>,\n    pub batch_coordinator: BatchCoordinator,\n    pub dependency_resolver: DependencyResolver,\n    pub state_manager: Arc<WorkflowStateManager>,\n    pub monitoring_system: Arc<MonitoringSystem>,\n    pub progress_tracker: Arc<ProgressTracker>,\n    pub rollback_system: Arc<RollbackSystem>,\n    pub test_orchestrator: Arc<TestOrchestrator>,\n    pub dns_manager: Arc<DnsManager>,\n    pub extension_manager: Arc<ExtensionManager>,\n    pub oci_manager: Arc<OciManager>,\n    pub service_orchestrator: Arc<ServiceOrchestrator>,\n    pub audit_logger: Arc<AuditLogger>,\n    pub args: Args,\n\n    // NEW: Security components\n    pub security: SecurityComponents,\n}\n```\n\n### Initialization in main.rs\n\n```\n#[tokio::main]\nasync fn main() -> Result<()> {\n    let args = Args::parse();\n\n    // Initialize AppState (creates audit_logger)\n    let state = Arc::new(AppState::new(args).await?);\n\n    // Initialize security components\n    let security_config = SecurityConfig {\n        public_key_path: PathBuf::from("keys/public.pem"),\n        jwt_issuer: env::var("JWT_ISSUER").unwrap_or("control-center".to_string()),\n        jwt_audience: "orchestrator".to_string(),\n        cedar_policies_path: PathBuf::from("policies"),\n        auth_enabled: env::var("AUTH_ENABLED").unwrap_or("true".to_string()) == "true",\n        authz_enabled: env::var("AUTHZ_ENABLED").unwrap_or("true".to_string()) == "true",\n        mfa_enabled: env::var("MFA_ENABLED").unwrap_or("true".to_string()) == "true",\n        rate_limit_config: RateLimitConfig::new(\n            env::var("RATE_LIMIT_MAX").unwrap_or("100".to_string()).parse().unwrap(),\n            env::var("RATE_LIMIT_WINDOW").unwrap_or("60".to_string()).parse().unwrap(),\n        ),\n    };\n\n    let security = SecurityComponents::initialize(\n        security_config,\n        state.audit_logger.clone()\n    ).await?;\n\n    // Public routes (no auth)\n    let public_routes = Router::new()\n        .route("/health", get(health_check));\n\n    // Protected routes (full security chain)\n    let protected_routes = Router::new()\n        .route("/api/v1/servers", post(create_server))\n        .route("/api/v1/servers/:id", delete(delete_server))\n        .route("/api/v1/taskserv", post(create_taskserv))\n        .route("/api/v1/cluster", post(create_cluster))\n        // ... more routes\n        ;\n\n    // Apply security middleware to protected routes\n    let secured_routes = apply_security_middleware(protected_routes, &security)\n        .with_state(state.clone());\n\n    // Combine routes\n    let app = Router::new()\n        .merge(public_routes)\n        .merge(secured_routes)\n        .layer(CorsLayer::permissive());\n\n    // Start server\n    let listener = tokio::net::TcpListener::bind("0.0.0.0:9090").await?;\n    axum::serve(listener, app).await?;\n\n    Ok(())\n}\n```\n\n## Protected Endpoints\n\n### Endpoint Categories\n\n| Category | Example Endpoints | Auth Required | MFA Required | Cedar Policy |\n| ---------- | ------------------- | --------------- | -------------- | -------------- |\n| **Health** | `/health` | ❌ | ❌ | ❌ |\n| **Read-Only** | `GET /api/v1/servers` | ✅ | ❌ | ✅ |\n| **Server Mgmt** | `POST /api/v1/servers` | ✅ | ❌ | ✅ |\n| **Server Delete** | `DELETE /api/v1/servers/:id` | ✅ | ✅ | ✅ |\n| **Taskserv Mgmt** | `POST /api/v1/taskserv` | ✅ | ❌ | ✅ |\n| **Cluster Mgmt** | `POST /api/v1/cluster` | ✅ | ✅ | ✅ |\n| **Production** | `POST /api/v1/production/*` | ✅ | ✅ | ✅ |\n| **Batch Ops** | `POST /api/v1/batch/submit` | ✅ | ✅ | ✅ |\n| **Rollback** | `POST /api/v1/rollback` | ✅ | ✅ | ✅ |\n| **Config Write** | `POST /api/v1/config` | ✅ | ✅ | ✅ |\n| **Secrets** | `GET /api/v1/secret/*` | ✅ | ✅ | ✅ |\n\n## Complete Authentication Flow\n\n### Step-by-Step Flow\n\n```\n1. CLIENT REQUEST\n   ├─ Headers:\n   │  ├─ Authorization: Bearer <jwt_token>\n   │  ├─ X-Forwarded-For: 192.168.1.100\n   │  ├─ User-Agent: MyClient/1.0\n   │  └─ X-MFA-Verified: true\n   └─ Path: DELETE /api/v1/servers/prod-srv-01\n\n2. RATE LIMITING MIDDLEWARE\n   ├─ Extract IP: 192.168.1.100\n   ├─ Check limit: 45/100 requests in window\n   ├─ Decision: ALLOW (under limit)\n   └─ Continue →\n\n3. AUTHENTICATION MIDDLEWARE\n   ├─ Extract Bearer token\n   ├─ Validate JWT:\n   │  ├─ Signature: ✅ Valid (RS256)\n   │  ├─ Expiry: ✅ Valid until 2025-10-09 10:00:00\n   │  ├─ Issuer: ✅ control-center\n   │  ├─ Audience: ✅ orchestrator\n   │  └─ Revoked: ✅ Not revoked\n   ├─ Build SecurityContext:\n   │  ├─ user_id: "user-456"\n   │  ├─ workspace: "production"\n   │  ├─ permissions: ["read", "write", "delete"]\n   │  ├─ mfa_verified: true\n   │  └─ ip_address: 192.168.1.100\n   ├─ Decision: ALLOW (valid token)\n   └─ Continue →\n\n4. MFA VERIFICATION MIDDLEWARE\n   ├─ Check endpoint: DELETE /api/v1/servers/prod-srv-01\n   ├─ Requires MFA: ✅ YES (DELETE operation)\n   ├─ MFA status: ✅ Verified\n   ├─ Decision: ALLOW (MFA verified)\n   └─ Continue →\n\n5. AUTHORIZATION MIDDLEWARE\n   ├─ Build Cedar request:\n   │  ├─ Principal: User("user-456")\n   │  ├─ Action: Delete\n   │  ├─ Resource: Server("prod-srv-01")\n   │  └─ Context:\n   │     ├─ mfa_verified: true\n   │     ├─ ip_address: "192.168.1.100"\n   │     ├─ time: 2025-10-08T14:30:00Z\n   │     └─ workspace: "production"\n   ├─ Evaluate Cedar policies:\n   │  ├─ Policy 1: Allow if user.role == "admin" ✅\n   │  ├─ Policy 2: Allow if mfa_verified == true ✅\n   │  └─ Policy 3: Deny if not business_hours ❌\n   ├─ Decision: ALLOW (2 allow, 1 deny = allow)\n   ├─ Log to audit: Authorization GRANTED\n   └─ Continue →\n\n6. AUDIT LOGGING MIDDLEWARE\n   ├─ Record:\n   │  ├─ User: user-456 (IP: 192.168.1.100)\n   │  ├─ Action: ServerDelete\n   │  ├─ Resource: prod-srv-01\n   │  ├─ Authorization: GRANTED\n   │  ├─ MFA: Verified\n   │  └─ Timestamp: 2025-10-08T14:30:00Z\n   └─ Continue →\n\n7. PROTECTED HANDLER\n   ├─ Execute business logic\n   ├─ Delete server prod-srv-01\n   └─ Return: 200 OK\n\n8. AUDIT LOGGING (Response)\n   ├─ Update event:\n   │  ├─ Status: 200 OK\n   │  ├─ Duration: 1.234s\n   │  └─ Result: SUCCESS\n   └─ Write to audit log\n\n9. CLIENT RESPONSE\n   └─ 200 OK: Server deleted successfully\n```\n\n## Configuration\n\n### Environment Variables\n\n```\n# JWT Configuration\nJWT_ISSUER=control-center\nJWT_AUDIENCE=orchestrator\nPUBLIC_KEY_PATH=/path/to/keys/public.pem\n\n# Cedar Policies\nCEDAR_POLICIES_PATH=/path/to/policies\n\n# Security Toggles\nAUTH_ENABLED=true\nAUTHZ_ENABLED=true\nMFA_ENABLED=true\n\n# Rate Limiting\nRATE_LIMIT_MAX=100\nRATE_LIMIT_WINDOW=60\nRATE_LIMIT_EXEMPT_IPS=10.0.0.1,10.0.0.2\n\n# Audit Logging\nAUDIT_ENABLED=true\nAUDIT_RETENTION_DAYS=365\n```\n\n### Development Mode\n\nFor development/testing, all security can be disabled:\n\n```\n// In main.rs\nlet security = if env::var("DEVELOPMENT_MODE").unwrap_or("false".to_string()) == "true" {\n    SecurityComponents::disabled(audit_logger.clone())\n} else {\n    SecurityComponents::initialize(security_config, audit_logger.clone()).await?\n};\n```\n\n## Testing\n\n### Integration Tests\n\nLocation: `provisioning/platform/orchestrator/tests/security_integration_tests.rs`\n\n**Test Coverage**:\n\n- ✅ Rate limiting enforcement\n- ✅ Rate limit statistics\n- ✅ Exempt IP handling\n- ✅ Authentication missing token\n- ✅ MFA verification for sensitive operations\n- ✅ Cedar policy evaluation\n- ✅ Complete security flow\n- ✅ Security components initialization\n- ✅ Configuration defaults\n\n**Lines of Code**: 340\n\n**Run Tests**:\n\n```\ncd provisioning/platform/orchestrator\ncargo test security_integration_tests\n```\n\n## File Summary\n\n| File | Purpose | Lines | Tests |\n| ------ | --------- | ------- | ------- |\n| `middleware/security_context.rs` | Security context builder | 275 | 8 |\n| `middleware/auth.rs` | JWT authentication | 245 | 5 |\n| `middleware/mfa.rs` | MFA verification | 290 | 15 |\n| `middleware/authz.rs` | Cedar authorization | 380 | 4 |\n| `middleware/rate_limit.rs` | Rate limiting | 420 | 8 |\n| `middleware/mod.rs` | Module exports | 25 | 0 |\n| `security_integration.rs` | Integration helpers | 265 | 2 |\n| `tests/security_integration_tests.rs` | Integration tests | 340 | 11 |\n| **Total** | | **2,240** | **53** |\n\n## Benefits\n\n### Security\n\n- ✅ Complete authentication flow with JWT validation\n- ✅ MFA enforcement for sensitive operations\n- ✅ Fine-grained authorization with Cedar policies\n- ✅ Rate limiting prevents API abuse\n- ✅ Complete audit trail for compliance\n\n### Architecture\n\n- ✅ Modular middleware design\n- ✅ Clear separation of concerns\n- ✅ Reusable security components\n- ✅ Easy to test and maintain\n- ✅ Configuration-driven behavior\n\n### Operations\n\n- ✅ Can enable/disable features independently\n- ✅ Development mode for testing\n- ✅ Comprehensive error messages\n- ✅ Real-time statistics and monitoring\n- ✅ Non-blocking audit logging\n\n## Future Enhancements\n\n1. **Token Refresh**: Automatic token refresh before expiry\n2. **IP Whitelisting**: Additional IP-based access control\n3. **Geolocation**: Block requests from specific countries\n4. **Advanced Rate Limiting**: Per-user, per-endpoint limits\n5. **Session Management**: Track active sessions, force logout\n6. **2FA Integration**: Direct integration with TOTP/SMS providers\n7. **Policy Hot Reload**: Update Cedar policies without restart\n8. **Metrics Dashboard**: Real-time security metrics visualization\n\n## Related Documentation\n\n- Cedar Policy Language\n- JWT Token Management\n- MFA Setup Guide\n- Audit Log Format\n- Rate Limiting Best Practices\n\n## Version History\n\n| Version | Date | Changes |\n| --------- | ------ | --------- |\n| 1.0.0 | 2025-10-08 | Initial implementation |\n\n---\n\n**Maintained By**: Security Team\n**Review Cycle**: Quarterly\n**Last Reviewed**: 2025-10-08
