Orchestrator Authentication & Authorization Integration

Version: 1.0.0 Date: 2025-10-08 Status: Implemented

Overview

Complete authentication and authorization flow integration for the Provisioning Orchestrator, connecting all security components (JWT validation, MFA verification, Cedar authorization, rate limiting, and audit logging) into a cohesive security middleware chain.

Architecture

Security Middleware Chain

The middleware chain is applied in this specific order to ensure proper security:

┌─────────────────────────────────────────────────────────────────┐
│                    Incoming HTTP Request                        │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────┐
        │  1. Rate Limiting Middleware   │
        │  - Per-IP request limits       │
        │  - Sliding window              │
        │  - Exempt IPs                  │
        └────────────┬───────────────────┘
                     │ (429 if exceeded)
                     ▼
        ┌────────────────────────────────┐
        │  2. Authentication Middleware  │
        │  - Extract Bearer token        │
        │  - Validate JWT signature      │
        │  - Check expiry, issuer, aud   │
        │  - Check revocation            │
        └────────────┬───────────────────┘
                     │ (401 if invalid)
                     ▼
        ┌────────────────────────────────┐
        │  3. MFA Verification           │
        │  - Check MFA status in token   │
        │  - Enforce for sensitive ops   │
        │  - Production deployments      │
        │  - All DELETE operations       │
        └────────────┬───────────────────┘
                     │ (403 if required but missing)
                     ▼
        ┌────────────────────────────────┐
        │  4. Authorization Middleware   │
        │  - Build Cedar request         │
        │  - Evaluate policies           │
        │  - Check permissions           │
        │  - Log decision                │
        └────────────┬───────────────────┘
                     │ (403 if denied)
                     ▼
        ┌────────────────────────────────┐
        │  5. Audit Logging Middleware   │
        │  - Log complete request        │
        │  - User, action, resource      │
        │  - Authorization decision      │
        │  - Response status             │
        └────────────┬───────────────────┘
                     │
                     ▼
        ┌────────────────────────────────┐
        │      Protected Handler         │
        │  - Access security context     │
        │  - Execute business logic      │
        └────────────────────────────────┘

Implementation Details

1. Security Context Builder (middleware/security_context.rs)

Purpose: Build complete security context from authenticated requests.

Key Features:

• Extracts JWT token claims
• Determines MFA verification status
• Extracts IP address (X-Forwarded-For, X-Real-IP)
• Extracts user agent and session info
• Provides permission checking methods

Lines of Code: 275

Example:

pub struct SecurityContext {
    pub user_id: String,
    pub token: ValidatedToken,
    pub mfa_verified: bool,
    pub ip_address: IpAddr,
    pub user_agent: Option<String>,
    pub permissions: Vec<String>,
    pub workspace: String,
    pub request_id: String,
    pub session_id: Option<String>,
}

impl SecurityContext {
    pub fn has_permission(&self, permission: &str) -> bool { ... }
    pub fn has_any_permission(&self, permissions: &[&str]) -> bool { ... }
    pub fn has_all_permissions(&self, permissions: &[&str]) -> bool { ... }
}

2. Enhanced Authentication Middleware (middleware/auth.rs)

Purpose: JWT token validation with revocation checking.

Key Features:

• Bearer token extraction
• JWT signature validation (RS256)
• Expiry, issuer, audience checks
• Token revocation status
• Security context injection

Lines of Code: 245

Flow:

1. Extract Authorization: Bearer <token> header
2. Validate JWT with TokenValidator
3. Build SecurityContext
4. Inject into request extensions
5. Continue to next middleware or return 401

Error Responses:

• 401 Unauthorized: Missing/invalid token, expired, revoked
• 403 Forbidden: Insufficient permissions

3. MFA Verification Middleware (middleware/mfa.rs)

Purpose: Enforce MFA for sensitive operations.

Key Features:

• Path-based MFA requirements
• Method-based enforcement (all DELETEs)
• Production environment protection
• Clear error messages

Lines of Code: 290

MFA Required For:

• Production deployments (/production/, /prod/)
• All DELETE operations
• Server operations (POST, PUT, DELETE)
• Cluster operations (POST, PUT, DELETE)
• Batch submissions
• Rollback operations
• Configuration changes (POST, PUT, DELETE)
• Secret management
• User/role management

Example:

fn requires_mfa(method: &str, path: &str) -> bool {
    if path.contains("/production/") { return true; }
    if method == "DELETE" { return true; }
    if path.contains("/deploy") { return true; }
    // ...
}

4. Enhanced Authorization Middleware (middleware/authz.rs)

Purpose: Cedar policy evaluation with audit logging.

Key Features:

• Builds Cedar authorization request from HTTP request
• Maps HTTP methods to Cedar actions (GET→Read, POST→Create, etc.)
• Extracts resource types from paths
• Evaluates Cedar policies with context (MFA, IP, time, workspace)
• Logs all authorization decisions to audit log
• Non-blocking audit logging (tokio::spawn)

Lines of Code: 380

Resource Mapping:

/api/v1/servers/srv-123    → Resource::Server("srv-123")
/api/v1/taskserv/kubernetes → Resource::TaskService("kubernetes")
/api/v1/cluster/prod        → Resource::Cluster("prod")
/api/v1/config/settings     → Resource::Config("settings")

Action Mapping:

GET    → Action::Read
POST   → Action::Create
PUT    → Action::Update
DELETE → Action::Delete

5. Rate Limiting Middleware (middleware/rate_limit.rs)

Purpose: Prevent API abuse with per-IP rate limiting.

Key Features:

• Sliding window rate limiting
• Per-IP request tracking
• Configurable limits and windows
• Exempt IP support
• Automatic cleanup of old entries
• Statistics tracking

Lines of Code: 420

Configuration:

pub struct RateLimitConfig {
    pub max_requests: u32,          // e.g., 100
    pub window_duration: Duration,  // e.g., 60 seconds
    pub exempt_ips: Vec<IpAddr>,    // e.g., internal services
    pub enabled: bool,
}

// Default: 100 requests per minute

Statistics:

pub struct RateLimitStats {
    pub total_ips: usize,      // Number of tracked IPs
    pub total_requests: u32,   // Total requests made
    pub limited_ips: usize,    // IPs that hit the limit
    pub config: RateLimitConfig,
}

6. Security Integration Module (security_integration.rs)

Purpose: Helper module to integrate all security components.

Key Features:

• SecurityComponents struct grouping all middleware
• SecurityConfig for configuration
• initialize() method to set up all components
• disabled() method for development mode
• apply_security_middleware() helper for router setup

Lines of Code: 265

Usage Example:

use provisioning_orchestrator::security_integration::{
    SecurityComponents, SecurityConfig
};

// Initialize security
let config = SecurityConfig {
    public_key_path: PathBuf::from("keys/public.pem"),
    jwt_issuer: "control-center".to_string(),
    jwt_audience: "orchestrator".to_string(),
    cedar_policies_path: PathBuf::from("policies"),
    auth_enabled: true,
    authz_enabled: true,
    mfa_enabled: true,
    rate_limit_config: RateLimitConfig::new(100, 60),
};

let security = SecurityComponents::initialize(config, audit_logger).await?;

// Apply to router
let app = Router::new()
    .route("/api/v1/servers", post(create_server))
    .route("/api/v1/servers/:id", delete(delete_server));

let secured_app = apply_security_middleware(app, &security);

Integration with AppState

Updated AppState Structure

pub struct AppState {
    // Existing fields
    pub task_storage: Arc<dyn TaskStorage>,
    pub batch_coordinator: BatchCoordinator,
    pub dependency_resolver: DependencyResolver,
    pub state_manager: Arc<WorkflowStateManager>,
    pub monitoring_system: Arc<MonitoringSystem>,
    pub progress_tracker: Arc<ProgressTracker>,
    pub rollback_system: Arc<RollbackSystem>,
    pub test_orchestrator: Arc<TestOrchestrator>,
    pub dns_manager: Arc<DnsManager>,
    pub extension_manager: Arc<ExtensionManager>,
    pub oci_manager: Arc<OciManager>,
    pub service_orchestrator: Arc<ServiceOrchestrator>,
    pub audit_logger: Arc<AuditLogger>,
    pub args: Args,

    // NEW: Security components
    pub security: SecurityComponents,
}

Initialization in main.rs

#[tokio::main]
async fn main() -> Result<()> {
    let args = Args::parse();

    // Initialize AppState (creates audit_logger)
    let state = Arc::new(AppState::new(args).await?);

    // Initialize security components
    let security_config = SecurityConfig {
        public_key_path: PathBuf::from("keys/public.pem"),
        jwt_issuer: env::var("JWT_ISSUER").unwrap_or("control-center".to_string()),
        jwt_audience: "orchestrator".to_string(),
        cedar_policies_path: PathBuf::from("policies"),
        auth_enabled: env::var("AUTH_ENABLED").unwrap_or("true".to_string()) == "true",
        authz_enabled: env::var("AUTHZ_ENABLED").unwrap_or("true".to_string()) == "true",
        mfa_enabled: env::var("MFA_ENABLED").unwrap_or("true".to_string()) == "true",
        rate_limit_config: RateLimitConfig::new(
            env::var("RATE_LIMIT_MAX").unwrap_or("100".to_string()).parse().unwrap(),
            env::var("RATE_LIMIT_WINDOW").unwrap_or("60".to_string()).parse().unwrap(),
        ),
    };

    let security = SecurityComponents::initialize(
        security_config,
        state.audit_logger.clone()
    ).await?;

    // Public routes (no auth)
    let public_routes = Router::new()
        .route("/health", get(health_check));

    // Protected routes (full security chain)
    let protected_routes = Router::new()
        .route("/api/v1/servers", post(create_server))
        .route("/api/v1/servers/:id", delete(delete_server))
        .route("/api/v1/taskserv", post(create_taskserv))
        .route("/api/v1/cluster", post(create_cluster))
        // ... more routes
        ;

    // Apply security middleware to protected routes
    let secured_routes = apply_security_middleware(protected_routes, &security)
        .with_state(state.clone());

    // Combine routes
    let app = Router::new()
        .merge(public_routes)
        .merge(secured_routes)
        .layer(CorsLayer::permissive());

    // Start server
    let listener = tokio::net::TcpListener::bind("0.0.0.0:9090").await?;
    axum::serve(listener, app).await?;

    Ok(())
}

Protected Endpoints

Endpoint Categories

Complete Authentication Flow

Step-by-Step Flow

1. CLIENT REQUEST
   ├─ Headers:
   │  ├─ Authorization: Bearer <jwt_token>
   │  ├─ X-Forwarded-For: 192.168.1.100
   │  ├─ User-Agent: MyClient/1.0
   │  └─ X-MFA-Verified: true
   └─ Path: DELETE /api/v1/servers/prod-srv-01

2. RATE LIMITING MIDDLEWARE
   ├─ Extract IP: 192.168.1.100
   ├─ Check limit: 45/100 requests in window
   ├─ Decision: ALLOW (under limit)
   └─ Continue →

3. AUTHENTICATION MIDDLEWARE
   ├─ Extract Bearer token
   ├─ Validate JWT:
   │  ├─ Signature: ✅ Valid (RS256)
   │  ├─ Expiry: ✅ Valid until 2025-10-09 10:00:00
   │  ├─ Issuer: ✅ control-center
   │  ├─ Audience: ✅ orchestrator
   │  └─ Revoked: ✅ Not revoked
   ├─ Build SecurityContext:
   │  ├─ user_id: "user-456"
   │  ├─ workspace: "production"
   │  ├─ permissions: ["read", "write", "delete"]
   │  ├─ mfa_verified: true
   │  └─ ip_address: 192.168.1.100
   ├─ Decision: ALLOW (valid token)
   └─ Continue →

4. MFA VERIFICATION MIDDLEWARE
   ├─ Check endpoint: DELETE /api/v1/servers/prod-srv-01
   ├─ Requires MFA: ✅ YES (DELETE operation)
   ├─ MFA status: ✅ Verified
   ├─ Decision: ALLOW (MFA verified)
   └─ Continue →

5. AUTHORIZATION MIDDLEWARE
   ├─ Build Cedar request:
   │  ├─ Principal: User("user-456")
   │  ├─ Action: Delete
   │  ├─ Resource: Server("prod-srv-01")
   │  └─ Context:
   │     ├─ mfa_verified: true
   │     ├─ ip_address: "192.168.1.100"
   │     ├─ time: 2025-10-08T14:30:00Z
   │     └─ workspace: "production"
   ├─ Evaluate Cedar policies:
   │  ├─ Policy 1: Allow if user.role == "admin" ✅
   │  ├─ Policy 2: Allow if mfa_verified == true ✅
   │  └─ Policy 3: Deny if not business_hours ❌
   ├─ Decision: ALLOW (2 allow, 1 deny = allow)
   ├─ Log to audit: Authorization GRANTED
   └─ Continue →

6. AUDIT LOGGING MIDDLEWARE
   ├─ Record:
   │  ├─ User: user-456 (IP: 192.168.1.100)
   │  ├─ Action: ServerDelete
   │  ├─ Resource: prod-srv-01
   │  ├─ Authorization: GRANTED
   │  ├─ MFA: Verified
   │  └─ Timestamp: 2025-10-08T14:30:00Z
   └─ Continue →

7. PROTECTED HANDLER
   ├─ Execute business logic
   ├─ Delete server prod-srv-01
   └─ Return: 200 OK

8. AUDIT LOGGING (Response)
   ├─ Update event:
   │  ├─ Status: 200 OK
   │  ├─ Duration: 1.234s
   │  └─ Result: SUCCESS
   └─ Write to audit log

9. CLIENT RESPONSE
   └─ 200 OK: Server deleted successfully

Configuration

Environment Variables

# JWT Configuration
JWT_ISSUER=control-center
JWT_AUDIENCE=orchestrator
PUBLIC_KEY_PATH=/path/to/keys/public.pem

# Cedar Policies
CEDAR_POLICIES_PATH=/path/to/policies

# Security Toggles
AUTH_ENABLED=true
AUTHZ_ENABLED=true
MFA_ENABLED=true

# Rate Limiting
RATE_LIMIT_MAX=100
RATE_LIMIT_WINDOW=60
RATE_LIMIT_EXEMPT_IPS=10.0.0.1,10.0.0.2

# Audit Logging
AUDIT_ENABLED=true
AUDIT_RETENTION_DAYS=365

Development Mode

For development/testing, all security can be disabled:

// In main.rs
let security = if env::var("DEVELOPMENT_MODE").unwrap_or("false".to_string()) == "true" {
    SecurityComponents::disabled(audit_logger.clone())
} else {
    SecurityComponents::initialize(security_config, audit_logger.clone()).await?
};

Testing

Integration Tests

Location: provisioning/platform/orchestrator/tests/security_integration_tests.rs

Test Coverage:

• ✅ Rate limiting enforcement
• ✅ Rate limit statistics
• ✅ Exempt IP handling
• ✅ Authentication missing token
• ✅ MFA verification for sensitive operations
• ✅ Cedar policy evaluation
• ✅ Complete security flow
• ✅ Security components initialization
• ✅ Configuration defaults

Lines of Code: 340

Run Tests:

cd provisioning/platform/orchestrator
cargo test security_integration_tests

File Summary

Benefits

Security

• ✅ Complete authentication flow with JWT validation
• ✅ MFA enforcement for sensitive operations
• ✅ Fine-grained authorization with Cedar policies
• ✅ Rate limiting prevents API abuse
• ✅ Complete audit trail for compliance

Architecture

• ✅ Modular middleware design
• ✅ Clear separation of concerns
• ✅ Reusable security components
• ✅ Easy to test and maintain
• ✅ Configuration-driven behavior

Operations

• ✅ Can enable/disable features independently
• ✅ Development mode for testing
• ✅ Comprehensive error messages
• ✅ Real-time statistics and monitoring
• ✅ Non-blocking audit logging

Future Enhancements

1. Token Refresh: Automatic token refresh before expiry
2. IP Whitelisting: Additional IP-based access control
3. Geolocation: Block requests from specific countries
4. Advanced Rate Limiting: Per-user, per-endpoint limits
5. Session Management: Track active sessions, force logout
6. 2FA Integration: Direct integration with TOTP/SMS providers
7. Policy Hot Reload: Update Cedar policies without restart
8. Metrics Dashboard: Real-time security metrics visualization

Related Documentation

• Cedar Policy Language
• JWT Token Management
• MFA Setup Guide
• Audit Log Format
• Rate Limiting Best Practices

Version History

Maintained By: Security Team Review Cycle: Quarterly Last Reviewed: 2025-10-08