# MFA Admin Setup Guide - Production Operations Manual\n\n**Document Version**: 1.0.0\n**Last Updated**: 2025-10-08\n**Target Audience**: Platform Administrators, Security Team\n**Prerequisites**: Control Center deployed, admin user created\n\n---\n\n## 📋 Table of Contents\n\n1. [Overview](#overview)\n2. [MFA Requirements](#mfa-requirements)\n3. [Admin Enrollment Process](#admin-enrollment-process)\n4. [TOTP Setup (Authenticator Apps)](#totp-setup-authenticator-apps)\n5. [WebAuthn Setup (Hardware Keys)](#webauthn-setup-hardware-keys)\n6. [Enforcing MFA via Cedar Policies](#enforcing-mfa-via-cedar-policies)\n7. [Backup Codes Management](#backup-codes-management)\n8. [Recovery Procedures](#recovery-procedures)\n9. [Troubleshooting](#troubleshooting)\n10. [Best Practices](#best-practices)\n11. [Audit and Compliance](#audit-and-compliance)\n\n---\n\n## Overview\n\n### What is MFA\n\nMulti-Factor Authentication (MFA) adds a second layer of security beyond passwords. Admins must provide:\n\n1. **Something they know**: Password\n2. **Something they have**: TOTP code (authenticator app) or WebAuthn device (YubiKey, Touch ID)\n\n### Why MFA for Admins\n\nAdministrators have elevated privileges including:\n\n- Server creation/deletion\n- Production deployments\n- Secret management\n- User management\n- Break-glass approval\n\n**MFA protects against**:\n\n- Password compromise (phishing, leaks, brute force)\n- Unauthorized access to critical systems\n- Compliance violations (SOC2, ISO 27001)\n\n### MFA Methods Supported\n\n| Method | Type | Examples | Recommended For |\n| -------- | ------ | ---------- | ----------------- |\n| **TOTP** | Software | Google Authenticator, Authy, 1Password | All admins (primary) |\n| **WebAuthn/FIDO2** | Hardware | YubiKey, Touch ID, Windows Hello | High-security admins |\n| **Backup Codes** | One-time | 10 single-use codes | Emergency recovery |\n\n---\n\n## MFA Requirements\n\n### Mandatory MFA Enforcement\n\n**All administrators MUST enable MFA** for:\n\n- Production environment access\n- Server creation/deletion operations\n- Deployment to production clusters\n- Secret access (KMS, dynamic secrets)\n- Break-glass approval\n- User management operations\n\n### Grace Period\n\n- **Development**: MFA optional (not recommended)\n- **Staging**: MFA recommended, not enforced\n- **Production**: MFA **mandatory** (enforced by Cedar policies)\n\n### Timeline for Rollout\n\n```\nWeek 1-2: Pilot Program\n  ├─ Platform admins enable MFA\n  ├─ Document issues and refine process\n  └─ Create training materials\n\nWeek 3-4: Full Deployment\n  ├─ All admins enable MFA\n  ├─ Cedar policies enforce MFA for production\n  └─ Monitor compliance\n\nWeek 5+: Maintenance\n  ├─ Regular MFA device audits\n  ├─ Backup code rotation\n  └─ User support for MFA issues\n```\n\n---\n\n## Admin Enrollment Process\n\n### Step 1: Initial Login (Password Only)\n\n```\n# Login with username/password\nprovisioning login --user admin@example.com --workspace production\n\n# Response (partial token, MFA not yet verified):\n{\n  "status": "mfa_required",\n  "partial_token": "eyJhbGci...",  # Limited access token\n  "message": "MFA enrollment required for production access"\n}\n```\n\n**Partial token limitations**:\n\n- Cannot access production resources\n- Can only access MFA enrollment endpoints\n- Expires in 15 minutes\n\n### Step 2: Choose MFA Method\n\n```\n# Check available MFA methods\nprovisioning mfa methods\n\n# Output:\nAvailable MFA Methods:\n  • TOTP (Authenticator apps) - Recommended for all users\n  • WebAuthn (Hardware keys) - Recommended for high-security roles\n  • Backup Codes - Emergency recovery only\n\n# Check current MFA status\nprovisioning mfa status\n\n# Output:\nMFA Status:\n  TOTP: Not enrolled\n  WebAuthn: Not enrolled\n  Backup Codes: Not generated\n  MFA Required: Yes (production workspace)\n```\n\n### Step 3: Enroll MFA Device\n\nChoose one or both methods (TOTP + WebAuthn recommended):\n\n- [TOTP Setup](#totp-setup-authenticator-apps)\n- [WebAuthn Setup](#webauthn-setup-hardware-keys)\n\n### Step 4: Verify and Activate\n\nAfter enrollment, login again with MFA:\n\n```\n# Login (returns partial token)\nprovisioning login --user admin@example.com --workspace production\n\n# Verify MFA code (returns full access token)\nprovisioning mfa verify 123456\n\n# Response:\n{\n  "status": "authenticated",\n  "access_token": "eyJhbGci...",      # Full access token (15 min)\n  "refresh_token": "eyJhbGci...",     # Refresh token (7 days)\n  "mfa_verified": true,\n  "expires_in": 900\n}\n```\n\n---\n\n## TOTP Setup (Authenticator Apps)\n\n### Supported Authenticator Apps\n\n| App | Platform | Notes |\n| ----- | ---------- | ------- |\n| **Google Authenticator** | iOS, Android | Simple, widely used |\n| **Authy** | iOS, Android, Desktop | Cloud backup, multi-device |\n| **1Password** | All platforms | Integrated with password manager |\n| **Microsoft Authenticator** | iOS, Android | Enterprise integration |\n| **Bitwarden** | All platforms | Open source |\n\n### Step-by-Step TOTP Enrollment\n\n#### 1. Initiate TOTP Enrollment\n\n```\nprovisioning mfa totp enroll\n```\n\n**Output**:\n\n```\n╔════════════════════════════════════════════════════════════╗\n║                   TOTP ENROLLMENT                          ║\n╚════════════════════════════════════════════════════════════╝\n\nScan this QR code with your authenticator app:\n\n█████████████████████████████████\n█████████████████████████████████\n████ ▄▄▄▄▄ █▀ █▀▀██ ▄▄▄▄▄ ████\n████ █   █ █▀▄ ▀ ▄█ █   █ ████\n████ █▄▄▄█ █ ▀▀ ▀▀█ █▄▄▄█ ████\n████▄▄▄▄▄▄▄█ █▀█ ▀ █▄▄▄▄▄▄████\n█████████████████████████████████\n█████████████████████████████████\n\nManual entry (if QR code doesn't work):\n  Secret: JBSWY3DPEHPK3PXP\n  Account: admin@example.com\n  Issuer: Provisioning Platform\n\nTOTP Configuration:\n  Algorithm: SHA1\n  Digits: 6\n  Period: 30 seconds\n```\n\n#### 2. Add to Authenticator App\n\n**Option A: Scan QR Code (Recommended)**\n\n1. Open authenticator app (Google Authenticator, Authy, etc.)\n2. Tap "+" or "Add Account"\n3. Select "Scan QR Code"\n4. Point camera at QR code displayed in terminal\n5. Account added automatically\n\n**Option B: Manual Entry**\n\n1. Open authenticator app\n2. Tap "+" or "Add Account"\n3. Select "Enter a setup key" or "Manual entry"\n4. Enter:\n   - **Account name**: <admin@example.com>\n   - **Key**: `JBSWY3DPEHPK3PXP` (secret shown above)\n   - **Type of key**: Time-based\n5. Save account\n\n#### 3. Verify TOTP Code\n\n```\n# Get current code from authenticator app (6 digits, changes every 30s)\n# Example code: 123456\n\nprovisioning mfa totp verify 123456\n```\n\n**Success Response**:\n\n```\n✓ TOTP verified successfully!\n\nBackup Codes (SAVE THESE SECURELY):\n  1. A3B9-C2D7-E1F4\n  2. G8H5-J6K3-L9M2\n  3. N4P7-Q1R8-S5T2\n  4. U6V3-W9X1-Y7Z4\n  5. A2B8-C5D1-E9F3\n  6. G7H4-J2K6-L8M1\n  7. N3P9-Q5R2-S7T4\n  8. U1V6-W3X8-Y2Z5\n  9. A9B4-C7D2-E5F1\n 10. G3H8-J1K5-L6M9\n\n⚠ Store backup codes in a secure location (password manager, encrypted file)\n⚠ Each code can only be used once\n⚠ These codes allow access if you lose your authenticator device\n\nTOTP enrollment complete. MFA is now active for your account.\n```\n\n#### 4. Save Backup Codes\n\n**Critical**: Store backup codes in a secure location:\n\n```\n# Copy backup codes to password manager or encrypted file\n# NEVER store in plaintext, email, or cloud storage\n\n# Example: Store in encrypted file\nprovisioning mfa backup-codes --save-encrypted ~/secure/mfa-backup-codes.enc\n\n# Or display again (requires existing MFA verification)\nprovisioning mfa backup-codes --show\n```\n\n#### 5. Test TOTP Login\n\n```\n# Logout to test full login flow\nprovisioning logout\n\n# Login with password (returns partial token)\nprovisioning login --user admin@example.com --workspace production\n\n# Get current TOTP code from authenticator app\n# Verify with TOTP code (returns full access token)\nprovisioning mfa verify 654321\n\n# ✓ Full access granted\n```\n\n---\n\n## WebAuthn Setup (Hardware Keys)\n\n### Supported WebAuthn Devices\n\n| Device Type | Examples | Security Level |\n| ------------- | ---------- | ---------------- |\n| **USB Security Keys** | YubiKey 5, SoloKey, Titan Key | Highest |\n| **NFC Keys** | YubiKey 5 NFC, Google Titan | High (mobile compatible) |\n| **Biometric** | Touch ID (macOS), Windows Hello, Face ID | High (convenience) |\n| **Platform Authenticators** | Built-in laptop/phone biometrics | Medium-High |\n\n### Step-by-Step WebAuthn Enrollment\n\n#### 1. Check WebAuthn Support\n\n```\n# Verify WebAuthn support on your system\nprovisioning mfa webauthn check\n\n# Output:\nWebAuthn Support:\n  ✓ Browser: Chrome 120.0 (WebAuthn supported)\n  ✓ Platform: macOS 14.0 (Touch ID available)\n  ✓ USB: YubiKey 5 NFC detected\n```\n\n#### 2. Initiate WebAuthn Registration\n\n```\nprovisioning mfa webauthn register --device-name "YubiKey-Admin-Primary"\n```\n\n**Output**:\n\n```\n╔════════════════════════════════════════════════════════════╗\n║               WEBAUTHN DEVICE REGISTRATION                 ║\n╚════════════════════════════════════════════════════════════╝\n\nDevice Name: YubiKey-Admin-Primary\nRelying Party: provisioning.example.com\n\n⚠ Please insert your security key and touch it when it blinks\n\nWaiting for device interaction...\n```\n\n#### 3. Complete Device Registration\n\n**For USB Security Keys (YubiKey, SoloKey)**:\n\n1. Insert USB key into computer\n2. Terminal shows "Touch your security key"\n3. Touch the gold/silver contact on the key (it will blink)\n4. Registration completes\n\n**For Touch ID (macOS)**:\n\n1. Terminal shows "Touch ID prompt will appear"\n2. Touch ID dialog appears on screen\n3. Place finger on Touch ID sensor\n4. Registration completes\n\n**For Windows Hello**:\n\n1. Terminal shows "Windows Hello prompt"\n2. Windows Hello biometric prompt appears\n3. Complete biometric scan (fingerprint/face)\n4. Registration completes\n\n**Success Response**:\n\n```\n✓ WebAuthn device registered successfully!\n\nDevice Details:\n  Name: YubiKey-Admin-Primary\n  Type: USB Security Key\n  AAGUID: 2fc0579f-8113-47ea-b116-bb5a8 d9202a\n  Credential ID: kZj8C3bx...\n  Registered: 2025-10-08T14:32:10Z\n\nYou can now use this device for authentication.\n```\n\n#### 4. Register Additional Devices (Optional)\n\n**Best Practice**: Register 2+ WebAuthn devices (primary + backup)\n\n```\n# Register backup YubiKey\nprovisioning mfa webauthn register --device-name "YubiKey-Admin-Backup"\n\n# Register Touch ID (for convenience on personal laptop)\nprovisioning mfa webauthn register --device-name "MacBook-TouchID"\n```\n\n#### 5. List Registered Devices\n\n```\nprovisioning mfa webauthn list\n\n# Output:\nRegistered WebAuthn Devices:\n\n  1. YubiKey-Admin-Primary (USB Security Key)\n     Registered: 2025-10-08T14:32:10Z\n     Last Used: 2025-10-08T14:32:10Z\n\n  2. YubiKey-Admin-Backup (USB Security Key)\n     Registered: 2025-10-08T14:35:22Z\n     Last Used: Never\n\n  3. MacBook-TouchID (Platform Authenticator)\n     Registered: 2025-10-08T14:40:15Z\n     Last Used: 2025-10-08T15:20:05Z\n\nTotal: 3 devices\n```\n\n#### 6. Test WebAuthn Login\n\n```\n# Logout to test\nprovisioning logout\n\n# Login with password (partial token)\nprovisioning login --user admin@example.com --workspace production\n\n# Authenticate with WebAuthn\nprovisioning mfa webauthn verify\n\n# Output:\n⚠ Insert and touch your security key\n[Touch YubiKey when it blinks]\n\n✓ WebAuthn verification successful\n✓ Full access granted\n```\n\n---\n\n## Enforcing MFA via Cedar Policies\n\n### Production MFA Enforcement Policy\n\n**Location**: `provisioning/config/cedar-policies/production.cedar`\n\n```\n// Production operations require MFA verification\npermit (\n  principal,\n  action in [\n    Action::"server:create",\n    Action::"server:delete",\n    Action::"cluster:deploy",\n    Action::"secret:read",\n    Action::"user:manage"\n  ],\n  resource in Environment::"production"\n) when {\n  // MFA MUST be verified\n  context.mfa_verified == true\n};\n\n// Admin role requires MFA for ALL production actions\npermit (\n  principal in Role::"Admin",\n  action,\n  resource in Environment::"production"\n) when {\n  context.mfa_verified == true\n};\n\n// Break-glass approval requires MFA\npermit (\n  principal,\n  action == Action::"break_glass:approve",\n  resource\n) when {\n  context.mfa_verified == true &&\n  principal.role in [Role::"Admin", Role::"SecurityLead"]\n};\n```\n\n### Development/Staging Policies (MFA Recommended, Not Required)\n\n**Location**: `provisioning/config/cedar-policies/development.cedar`\n\n```\n// Development: MFA recommended but not enforced\npermit (\n  principal,\n  action,\n  resource in Environment::"dev"\n) when {\n  // MFA not required for dev, but logged if missing\n  true\n};\n\n// Staging: MFA recommended for destructive operations\npermit (\n  principal,\n  action in [Action::"server:delete", Action::"cluster:delete"],\n  resource in Environment::"staging"\n) when {\n  // Allow without MFA but log warning\n  context.mfa_verified == true || context has mfa_warning_acknowledged\n};\n```\n\n### Policy Deployment\n\n```\n# Validate Cedar policies\nprovisioning cedar validate --policies config/cedar-policies/\n\n# Test policies with sample requests\nprovisioning cedar test --policies config/cedar-policies/ \\n  --test-file tests/cedar-test-cases.yaml\n\n# Deploy to production (requires MFA + approval)\nprovisioning cedar deploy production --policies config/cedar-policies/production.cedar\n\n# Verify policy is active\nprovisioning cedar status production\n```\n\n### Testing MFA Enforcement\n\n```\n# Test 1: Production access WITHOUT MFA (should fail)\nprovisioning login --user admin@example.com --workspace production\nprovisioning server create web-01 --plan medium --check\n\n# Expected: Authorization denied (MFA not verified)\n\n# Test 2: Production access WITH MFA (should succeed)\nprovisioning login --user admin@example.com --workspace production\nprovisioning mfa verify 123456\nprovisioning server create web-01 --plan medium --check\n\n# Expected: Server creation initiated\n```\n\n---\n\n## Backup Codes Management\n\n### Generating Backup Codes\n\nBackup codes are automatically generated during first MFA enrollment:\n\n```\n# View existing backup codes (requires MFA verification)\nprovisioning mfa backup-codes --show\n\n# Regenerate backup codes (invalidates old ones)\nprovisioning mfa backup-codes --regenerate\n\n# Output:\n⚠ WARNING: Regenerating backup codes will invalidate all existing codes.\nContinue? (yes/no): yes\n\nNew Backup Codes:\n  1. X7Y2-Z9A4-B6C1\n  2. D3E8-F5G2-H9J4\n  3. K6L1-M7N3-P8Q2\n  4. R4S9-T6U1-V3W7\n  5. X2Y5-Z8A3-B9C4\n  6. D7E1-F4G6-H2J8\n  7. K5L9-M3N6-P1Q4\n  8. R8S2-T5U7-V9W3\n  9. X4Y6-Z1A8-B3C5\n 10. D9E2-F7G4-H6J1\n\n✓ Backup codes regenerated successfully\n⚠ Save these codes in a secure location\n```\n\n### Using Backup Codes\n\n**When to use backup codes**:\n\n- Lost authenticator device (phone stolen, broken)\n- WebAuthn key not available (traveling, left at office)\n- Authenticator app not working (time sync issue)\n\n**Login with backup code**:\n\n```\n# Login (partial token)\nprovisioning login --user admin@example.com --workspace production\n\n# Use backup code instead of TOTP/WebAuthn\nprovisioning mfa verify-backup X7Y2-Z9A4-B6C1\n\n# Output:\n✓ Backup code verified\n⚠ Backup code consumed (9 remaining)\n⚠ Enroll a new MFA device as soon as possible\n✓ Full access granted (temporary)\n```\n\n### Backup Code Storage Best Practices\n\n**✅ DO**:\n\n- Store in password manager (1Password, Bitwarden, LastPass)\n- Print and store in physical safe\n- Encrypt and store in secure cloud storage (with encryption key stored separately)\n- Share with trusted IT team member (encrypted)\n\n**❌ DON'T**:\n\n- Email to yourself\n- Store in plaintext file on laptop\n- Save in browser notes/bookmarks\n- Share via Slack/Teams/unencrypted chat\n- Screenshot and save to Photos\n\n**Example: Encrypted Storage**:\n\n```\n# Encrypt backup codes with Age\nprovisioning mfa backup-codes --export | \\n  age -p -o ~/secure/mfa-backup-codes.age\n\n# Decrypt when needed\nage -d ~/secure/mfa-backup-codes.age\n```\n\n---\n\n## Recovery Procedures\n\n### Scenario 1: Lost Authenticator Device (TOTP)\n\n**Situation**: Phone stolen/broken, authenticator app not accessible\n\n**Recovery Steps**:\n\n```\n# Step 1: Use backup code to login\nprovisioning login --user admin@example.com --workspace production\nprovisioning mfa verify-backup X7Y2-Z9A4-B6C1\n\n# Step 2: Remove old TOTP enrollment\nprovisioning mfa totp unenroll\n\n# Step 3: Enroll new TOTP device\nprovisioning mfa totp enroll\n# [Scan QR code with new phone/authenticator app]\nprovisioning mfa totp verify 654321\n\n# Step 4: Generate new backup codes\nprovisioning mfa backup-codes --regenerate\n```\n\n### Scenario 2: Lost WebAuthn Key (YubiKey)\n\n**Situation**: YubiKey lost, stolen, or damaged\n\n**Recovery Steps**:\n\n```\n# Step 1: Login with alternative method (TOTP or backup code)\nprovisioning login --user admin@example.com --workspace production\nprovisioning mfa verify 123456  # TOTP from authenticator app\n\n# Step 2: List registered WebAuthn devices\nprovisioning mfa webauthn list\n\n# Step 3: Remove lost device\nprovisioning mfa webauthn remove "YubiKey-Admin-Primary"\n\n# Output:\n⚠ Remove WebAuthn device "YubiKey-Admin-Primary"?\nThis cannot be undone. (yes/no): yes\n\n✓ Device removed\n\n# Step 4: Register new WebAuthn device\nprovisioning mfa webauthn register --device-name "YubiKey-Admin-Replacement"\n```\n\n### Scenario 3: All MFA Methods Lost\n\n**Situation**: Lost phone (TOTP), lost YubiKey, no backup codes\n\n**Recovery Steps** (Requires Admin Assistance):\n\n```\n# User contacts Security Team / Platform Admin\n\n# Admin performs MFA reset (requires 2+ admin approval)\nprovisioning admin mfa-reset admin@example.com \\n  --reason "Employee lost all MFA devices (phone + YubiKey)" \\n  --ticket SUPPORT-12345\n\n# Output:\n⚠ MFA Reset Request Created\n\nReset Request ID: MFA-RESET-20251008-001\nUser: admin@example.com\nReason: Employee lost all MFA devices (phone + YubiKey)\nTicket: SUPPORT-12345\n\nRequired Approvals: 2\nApprovers: 0/2\n\n# Two other admins approve (with their own MFA)\nprovisioning admin mfa-reset approve MFA-RESET-20251008-001 \\n  --reason "Verified via video call + employee badge"\n\n# After 2 approvals, MFA is reset\n✓ MFA reset approved (2/2 approvals)\n✓ User admin@example.com can now re-enroll MFA devices\n\n# User re-enrolls TOTP and WebAuthn\nprovisioning mfa totp enroll\nprovisioning mfa webauthn register --device-name "YubiKey-New"\n```\n\n### Scenario 4: Backup Codes Depleted\n\n**Situation**: Used 9 out of 10 backup codes\n\n**Recovery Steps**:\n\n```\n# Login with last backup code\nprovisioning login --user admin@example.com --workspace production\nprovisioning mfa verify-backup D9E2-F7G4-H6J1\n\n# Output:\n⚠ WARNING: This is your LAST backup code!\n✓ Backup code verified\n⚠ Regenerate backup codes immediately!\n\n# Immediately regenerate backup codes\nprovisioning mfa backup-codes --regenerate\n\n# Save new codes securely\n```\n\n---\n\n## Troubleshooting\n\n### Issue 1: "Invalid TOTP code" Error\n\n**Symptoms**:\n\n```\nprovisioning mfa verify 123456\n✗ Error: Invalid TOTP code\n```\n\n**Possible Causes**:\n\n1. **Time sync issue** (most common)\n2. Wrong secret key entered during enrollment\n3. Code expired (30-second window)\n\n**Solutions**:\n\n```\n# Check time sync (device clock must be accurate)\n# macOS:\nsudo sntp -sS time.apple.com\n\n# Linux:\nsudo ntpdate pool.ntp.org\n\n# Verify TOTP configuration\nprovisioning mfa totp status\n\n# Output:\nTOTP Configuration:\n  Algorithm: SHA1\n  Digits: 6\n  Period: 30 seconds\n  Time Window: ±1 period (90 seconds total)\n\n# Check system time vs NTP\ndate && curl -s http://worldtimeapi.org/api/ip | grep datetime\n\n# If time is off by >30 seconds, sync time and retry\n```\n\n### Issue 2: WebAuthn Not Detected\n\n**Symptoms**:\n\n```\nprovisioning mfa webauthn register\n✗ Error: No WebAuthn authenticator detected\n```\n\n**Solutions**:\n\n```\n# Check USB connection (for hardware keys)\n# macOS:\nsystem_profiler SPUSBDataType | grep -i yubikey\n\n# Linux:\nlsusb | grep -i yubico\n\n# Check browser WebAuthn support\nprovisioning mfa webauthn check\n\n# Try different USB port (USB-A vs USB-C)\n\n# For Touch ID: Ensure finger is enrolled in System Preferences\n# For Windows Hello: Ensure biometrics are configured in Settings\n```\n\n### Issue 3: "MFA Required" Despite Verification\n\n**Symptoms**:\n\n```\nprovisioning server create web-01\n✗ Error: Authorization denied (MFA verification required)\n```\n\n**Cause**: Access token expired (15 min) or MFA verification not in token claims\n\n**Solution**:\n\n```\n# Check token expiration\nprovisioning auth status\n\n# Output:\nAuthentication Status:\n  Logged in: Yes\n  User: admin@example.com\n  Access Token: Expired (issued 16 minutes ago)\n  MFA Verified: Yes (but token expired)\n\n# Re-authenticate (will prompt for MFA again)\nprovisioning login --user admin@example.com --workspace production\nprovisioning mfa verify 654321\n\n# Verify MFA claim in token\nprovisioning auth decode-token\n\n# Output (JWT claims):\n{\n  "sub": "admin@example.com",\n  "role": "Admin",\n  "mfa_verified": true,  # ← Must be true\n  "mfa_method": "totp",\n  "iat": 1696766400,\n  "exp": 1696767300\n}\n```\n\n### Issue 4: QR Code Not Displaying\n\n**Symptoms**: QR code appears garbled or doesn't display in terminal\n\n**Solutions**:\n\n```\n# Use manual entry instead\nprovisioning mfa totp enroll --manual\n\n# Output (no QR code):\nManual TOTP Setup:\n  Secret: JBSWY3DPEHPK3PXP\n  Account: admin@example.com\n  Issuer: Provisioning Platform\n\nEnter this secret manually in your authenticator app.\n\n# Or export QR code to image file\nprovisioning mfa totp enroll --qr-image ~/mfa-qr.png\nopen ~/mfa-qr.png  # View in image viewer\n```\n\n### Issue 5: Backup Code Not Working\n\n**Symptoms**:\n\n```\nprovisioning mfa verify-backup X7Y2-Z9A4-B6C1\n✗ Error: Invalid or already used backup code\n```\n\n**Possible Causes**:\n\n1. Code already used (single-use only)\n2. Backup codes regenerated (old codes invalidated)\n3. Typo in code entry\n\n**Solutions**:\n\n```\n# Check backup code status (requires alternative login method)\nprovisioning mfa backup-codes --status\n\n# Output:\nBackup Codes Status:\n  Total Generated: 10\n  Used: 3\n  Remaining: 7\n  Last Used: 2025-10-05T10:15:30Z\n\n# Contact admin for MFA reset if all codes used\n# Or use alternative MFA method (TOTP, WebAuthn)\n```\n\n---\n\n## Best Practices\n\n### For Individual Admins\n\n#### 1. Use Multiple MFA Methods\n\n**✅ Recommended Setup**:\n\n- **Primary**: TOTP (Google Authenticator, Authy)\n- **Backup**: WebAuthn (YubiKey or Touch ID)\n- **Emergency**: Backup codes (stored securely)\n\n```\n# Enroll all three\nprovisioning mfa totp enroll\nprovisioning mfa webauthn register --device-name "YubiKey-Primary"\nprovisioning mfa backup-codes --save-encrypted ~/secure/codes.enc\n```\n\n#### 2. Secure Backup Code Storage\n\n```\n# Store in password manager (1Password example)\nprovisioning mfa backup-codes --show | \\n  op item create --category "Secure Note" \\n    --title "Provisioning MFA Backup Codes" \\n    --vault "Work"\n\n# Or encrypted file\nprovisioning mfa backup-codes --export | \\n  age -p -o ~/secure/mfa-backup-codes.age\n```\n\n#### 3. Regular Device Audits\n\n```\n# Monthly: Review registered devices\nprovisioning mfa devices --all\n\n# Remove unused/old devices\nprovisioning mfa webauthn remove "Old-YubiKey"\nprovisioning mfa totp remove "Old-Phone"\n```\n\n#### 4. Test Recovery Procedures\n\n```\n# Quarterly: Test backup code login\nprovisioning logout\nprovisioning login --user admin@example.com --workspace dev\nprovisioning mfa verify-backup [test-code]\n\n# Verify backup codes are accessible\ncat ~/secure/mfa-backup-codes.enc | age -d\n```\n\n### For Security Teams\n\n#### 1. MFA Enrollment Verification\n\n```\n# Generate MFA enrollment report\nprovisioning admin mfa-report --format csv > mfa-enrollment.csv\n\n# Output (CSV):\n# User,MFA_Enabled,TOTP,WebAuthn,Backup_Codes,Last_MFA_Login,Role\n# admin@example.com,Yes,Yes,Yes,10,2025-10-08T14:00:00Z,Admin\n# dev@example.com,No,No,No,0,Never,Developer\n```\n\n#### 2. Enforce MFA Deadlines\n\n```\n# Set MFA enrollment deadline\nprovisioning admin mfa-deadline set 2025-11-01 \\n  --roles Admin,Developer \\n  --environment production\n\n# Send reminder emails\nprovisioning admin mfa-remind \\n  --users-without-mfa \\n  --template "MFA enrollment required by Nov 1"\n```\n\n#### 3. Monitor MFA Usage\n\n```\n# Audit: Find production logins without MFA\nprovisioning audit query \\n  --action "auth:login" \\n  --filter 'mfa_verified == false && environment == "production"' \\n  --since 7d\n\n# Alert on repeated MFA failures\nprovisioning monitoring alert create \\n  --name "MFA Brute Force" \\n  --condition "mfa_failures > 5 in 5 min" \\n  --action "notify security-team"\n```\n\n#### 4. MFA Reset Policy\n\n**MFA Reset Requirements**:\n\n- User verification (video call + ID check)\n- Support ticket created (incident tracking)\n- 2+ admin approvals (different teams)\n- Time-limited reset window (24 hours)\n- Mandatory re-enrollment before production access\n\n```\n# MFA reset workflow\nprovisioning admin mfa-reset create user@example.com \\n  --reason "Lost all devices" \\n  --ticket SUPPORT-12345 \\n  --expires-in 24h\n\n# Requires 2 approvals\nprovisioning admin mfa-reset approve MFA-RESET-001\n```\n\n### For Platform Admins\n\n#### 1. Cedar Policy Best Practices\n\n```\n// Require MFA for high-risk actions\npermit (\n  principal,\n  action in [\n    Action::"server:delete",\n    Action::"cluster:delete",\n    Action::"secret:delete",\n    Action::"user:delete"\n  ],\n  resource\n) when {\n  context.mfa_verified == true &&\n  context.mfa_age_seconds < 300  // MFA verified within last 5 minutes\n};\n```\n\n#### 2. MFA Grace Periods (For Rollout)\n\n```\n# Development: No MFA required\nexport PROVISIONING_MFA_REQUIRED=false\n\n# Staging: MFA recommended (warnings only)\nexport PROVISIONING_MFA_REQUIRED=warn\n\n# Production: MFA mandatory (strict enforcement)\nexport PROVISIONING_MFA_REQUIRED=true\n```\n\n#### 3. Backup Admin Account\n\n**Emergency Admin** (break-glass scenario):\n\n- Separate admin account with MFA enrollment\n- Credentials stored in physical safe\n- Only used when primary admins locked out\n- Requires incident report after use\n\n```\n# Create emergency admin\nprovisioning admin create emergency-admin@example.com \\n  --role EmergencyAdmin \\n  --mfa-required true \\n  --max-concurrent-sessions 1\n\n# Print backup codes and store in safe\nprovisioning mfa backup-codes --show --user emergency-admin@example.com > emergency-codes.txt\n# [Print and store in physical safe]\n```\n\n---\n\n## Audit and Compliance\n\n### MFA Audit Logging\n\nAll MFA events are logged to the audit system:\n\n```\n# View MFA enrollment events\nprovisioning audit query \\n  --action-type "mfa:*" \\n  --since 30d\n\n# Output (JSON):\n[\n  {\n    "timestamp": "2025-10-08T14:32:10Z",\n    "action": "mfa:totp:enroll",\n    "user": "admin@example.com",\n    "result": "success",\n    "device_type": "totp",\n    "ip_address": "203.0.113.42"\n  },\n  {\n    "timestamp": "2025-10-08T14:35:22Z",\n    "action": "mfa:webauthn:register",\n    "user": "admin@example.com",\n    "result": "success",\n    "device_name": "YubiKey-Admin-Primary",\n    "ip_address": "203.0.113.42"\n  }\n]\n```\n\n### Compliance Reports\n\n#### SOC2 Compliance (Access Control)\n\n```\n# Generate SOC2 access control report\nprovisioning compliance report soc2 \\n  --control "CC6.1" \\n  --period "2025-Q3"\n\n# Output:\nSOC2 Trust Service Criteria - CC6.1 (Logical Access)\n\nMFA Enforcement:\n  ✓ MFA enabled for 100% of production admins (15/15)\n  ✓ MFA verified for 98.7% of production logins (2,453/2,485)\n  ✓ MFA policies enforced via Cedar authorization\n  ✓ Failed MFA attempts logged and monitored\n\nEvidence:\n  - Cedar policy: production.cedar (lines 15-25)\n  - Audit logs: mfa-verification-logs-2025-q3.json\n  - Enrollment report: mfa-enrollment-status.csv\n```\n\n#### ISO 27001 Compliance (A.9.4.2 - Secure Log-on)\n\n```\n# ISO 27001 A.9.4.2 compliance report\nprovisioning compliance report iso27001 \\n  --control "A.9.4.2" \\n  --format pdf \\n  --output iso27001-a942-mfa-report.pdf\n\n# Report Sections:\n# 1. MFA Implementation Details\n# 2. Enrollment Procedures\n# 3. Audit Trail\n# 4. Policy Enforcement\n# 5. Recovery Procedures\n```\n\n#### GDPR Compliance (MFA Data Handling)\n\n```\n# GDPR data subject request (MFA data export)\nprovisioning compliance gdpr export admin@example.com \\n  --include mfa\n\n# Output (JSON):\n{\n  "user": "admin@example.com",\n  "mfa_data": {\n    "totp_enrolled": true,\n    "totp_enrollment_date": "2025-10-08T14:32:10Z",\n    "webauthn_devices": [\n      {\n        "name": "YubiKey-Admin-Primary",\n        "registered": "2025-10-08T14:35:22Z",\n        "last_used": "2025-10-08T16:20:05Z"\n      }\n    ],\n    "backup_codes_remaining": 7,\n    "mfa_login_history": [...]  # Last 90 days\n  }\n}\n\n# GDPR deletion (MFA data removal after account deletion)\nprovisioning compliance gdpr delete admin@example.com --include-mfa\n```\n\n### MFA Metrics Dashboard\n\n```\n# Generate MFA metrics\nprovisioning admin mfa-metrics --period 30d\n\n# Output:\nMFA Metrics (Last 30 Days)\n\nEnrollment:\n  Total Users: 42\n  MFA Enabled: 38 (90.5%)\n  TOTP Only: 22 (57.9%)\n  WebAuthn Only: 3 (7.9%)\n  Both TOTP + WebAuthn: 13 (34.2%)\n  No MFA: 4 (9.5%) ⚠\n\nAuthentication:\n  Total Logins: 3,847\n  MFA Verified: 3,802 (98.8%)\n  MFA Failed: 45 (1.2%)\n  Backup Code Used: 7 (0.2%)\n\nDevices:\n  TOTP Devices: 35\n  WebAuthn Devices: 47\n  Backup Codes Remaining (avg): 8.3\n\nIncidents:\n  MFA Resets: 2\n  Lost Devices: 3\n  Lockouts: 1\n```\n\n---\n\n## Quick Reference Card\n\n### Daily Admin Operations\n\n```\n# Login with MFA\nprovisioning login --user admin@example.com --workspace production\nprovisioning mfa verify 123456\n\n# Check MFA status\nprovisioning mfa status\n\n# View registered devices\nprovisioning mfa devices\n```\n\n### MFA Management\n\n```\n# TOTP\nprovisioning mfa totp enroll              # Enroll TOTP\nprovisioning mfa totp verify 123456       # Verify TOTP code\nprovisioning mfa totp unenroll            # Remove TOTP\n\n# WebAuthn\nprovisioning mfa webauthn register --device-name "YubiKey"  # Register key\nprovisioning mfa webauthn list            # List devices\nprovisioning mfa webauthn remove "YubiKey"  # Remove device\n\n# Backup Codes\nprovisioning mfa backup-codes --show      # View codes\nprovisioning mfa backup-codes --regenerate  # Generate new codes\nprovisioning mfa verify-backup X7Y2-Z9A4-B6C1  # Use backup code\n```\n\n### Emergency Procedures\n\n```\n# Lost device recovery (use backup code)\nprovisioning login --user admin@example.com\nprovisioning mfa verify-backup [code]\nprovisioning mfa totp enroll  # Re-enroll new device\n\n# MFA reset (admin only)\nprovisioning admin mfa-reset user@example.com --reason "Lost all devices"\n\n# Check MFA compliance\nprovisioning admin mfa-report\n```\n\n---\n\n## Summary Checklist\n\n### For New Admins\n\n- [ ] Complete initial login with password\n- [ ] Enroll TOTP (Google Authenticator, Authy)\n- [ ] Verify TOTP code successfully\n- [ ] Save backup codes in password manager\n- [ ] Register WebAuthn device (YubiKey or Touch ID)\n- [ ] Test full login flow with MFA\n- [ ] Store backup codes in secure location\n- [ ] Verify production access works with MFA\n\n### For Security Team\n\n- [ ] Deploy Cedar MFA enforcement policies\n- [ ] Verify 100% admin MFA enrollment\n- [ ] Configure MFA audit logging\n- [ ] Setup MFA compliance reports (SOC2, ISO 27001)\n- [ ] Document MFA reset procedures\n- [ ] Train admins on MFA usage\n- [ ] Create emergency admin account (break-glass)\n- [ ] Schedule quarterly MFA audits\n\n### For Platform Team\n\n- [ ] Configure MFA settings in `config/mfa.toml`\n- [ ] Deploy Cedar policies with MFA requirements\n- [ ] Setup monitoring for MFA failures\n- [ ] Configure alerts for MFA bypass attempts\n- [ ] Document MFA architecture in ADR\n- [ ] Test MFA enforcement in all environments\n- [ ] Verify audit logs capture MFA events\n- [ ] Create runbooks for MFA incidents\n\n---\n\n## Support and Resources\n\n### Documentation\n\n- **MFA Implementation**: `/docs/architecture/MFA_IMPLEMENTATION_SUMMARY.md`\n- **Cedar Policies**: `/docs/operations/CEDAR_POLICIES_PRODUCTION_GUIDE.md`\n- **Break-Glass**: `/docs/operations/BREAK_GLASS_TRAINING_GUIDE.md`\n- **Audit Logging**: `/docs/architecture/AUDIT_LOGGING_IMPLEMENTATION.md`\n\n### Configuration Files\n\n- **MFA Config**: `provisioning/config/mfa.toml`\n- **Cedar Policies**: `provisioning/config/cedar-policies/production.cedar`\n- **Control Center**: `provisioning/platform/control-center/config.toml`\n\n### CLI Help\n\n```\nprovisioning mfa help          # MFA command help\nprovisioning mfa totp --help   # TOTP-specific help\nprovisioning mfa webauthn --help  # WebAuthn-specific help\n```\n\n### Contact\n\n- **Security Team**: <security@example.com>\n- **Platform Team**: <platform@example.com>\n- **Support Ticket**: <https://support.example.com>\n\n---\n\n**Document Status**: ✅ Complete\n**Review Date**: 2025-11-08\n**Maintained By**: Security Team, Platform Team
