# Break-Glass Emergency Access - Training Guide\n\n**Version**: 1.0.0\n**Date**: 2025-10-08\n**Audience**: Platform Administrators, SREs, Security Team\n**Training Duration**: 45-60 minutes\n**Certification**: Required annually\n\n---\n\n## 🚨 What is Break-Glass\n\n**Break-glass** is an emergency access procedure that allows authorized personnel to bypass normal security controls during critical incidents (for\nexample, production outages, security breaches, data loss).\n\n### Key Principles\n\n1. **Last Resort Only**: Use only when normal access is insufficient\n2. **Multi-Party Approval**: Requires 2+ approvers from different teams\n3. **Time-Limited**: Maximum 4 hours, auto-revokes\n4. **Enhanced Audit**: 7-year retention, immutable logs\n5. **Real-Time Alerts**: Security team notified immediately\n\n---\n\n## 📋 Table of Contents\n\n1. [When to Use Break-Glass](#when-to-use-break-glass)\n2. [When NOT to Use](#when-not-to-use)\n3. [Roles & Responsibilities](#roles--responsibilities)\n4. [Break-Glass Workflow](#break-glass-workflow)\n5. [Using the System](#using-the-system)\n6. [Examples](#examples)\n7. [Auditing & Compliance](#auditing--compliance)\n8. [Post-Incident Review](#post-incident-review)\n9. [FAQ](#faq)\n10. [Emergency Contacts](#emergency-contacts)\n\n---\n\n## When to Use Break-Glass\n\n### ✅ Valid Emergency Scenarios\n\n| Scenario | Example | Urgency |\n| ---------- | --------- | --------- |\n| **Production Outage** | Database cluster unresponsive, affecting all users | Critical |\n| **Security Incident** | Active breach detected, need immediate containment | Critical |\n| **Data Loss** | Accidental deletion of critical data, need restore | High |\n| **System Failure** | Infrastructure failure requiring emergency fixes | High |\n| **Locked Out** | Normal admin accounts compromised, need recovery | High |\n\n### Criteria Checklist\n\nUse break-glass if **ALL** apply:\n\n- [ ] Production systems affected OR security incident\n- [ ] Normal access insufficient OR unavailable\n- [ ] Immediate action required (cannot wait for approval process)\n- [ ] Clear justification for emergency access\n- [ ] Incident properly documented\n\n---\n\n## When NOT to Use\n\n### ❌ Invalid Scenarios (Do NOT Use Break-Glass)\n\n| Scenario | Why Not | Alternative |\n| ---------- | --------- | ------------- |\n| Forgot password | Not an emergency | Use password reset |\n| Routine maintenance | Can be scheduled | Use normal change process |\n| Convenience | Normal process "too slow" | Follow standard approval |\n| Deadline pressure | Business pressure ≠ emergency | Plan ahead |\n| Testing | Want to test emergency access | Use dev environment |\n\n### Consequences of Misuse\n\n- Immediate suspension of break-glass privileges\n- Security team investigation\n- Disciplinary action (up to termination)\n- All actions audited and reviewed\n\n---\n\n## Roles & Responsibilities\n\n### Requester\n\n**Who**: Platform Admin, SRE on-call, Security Officer\n**Responsibilities**:\n\n- Assess if situation warrants emergency access\n- Provide clear justification and reason\n- Document incident timeline\n- Use access only for stated purpose\n- Revoke access immediately after resolution\n\n### Approvers\n\n**Who**: 2+ from different teams (Security, Platform, Engineering Leadership)\n**Responsibilities**:\n\n- Verify emergency is genuine\n- Assess risk of granting access\n- Review requester's justification\n- Monitor usage during active session\n- Participate in post-incident review\n\n### Security Team\n\n**Who**: Security Operations team\n**Responsibilities**:\n\n- Monitor all break-glass activations (real-time)\n- Review audit logs during session\n- Alert on suspicious activity\n- Lead post-incident review\n- Update policies based on learnings\n\n---\n\n## Break-Glass Workflow\n\n### Phase 1: Request (5 minutes)\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ 1. Requester submits emergency access request          │\n│    - Reason: "Production database cluster down"        │\n│    - Justification: "Need direct SSH to diagnose"      │\n│    - Duration: 2 hours                                  │\n│    - Resources: ["database/*"]                          │\n└─────────────────────────────────────────────────────────┘\n                          ↓\n┌─────────────────────────────────────────────────────────┐\n│ 2. System creates request ID: BG-20251008-001          │\n│    - Sends notifications to approver pool               │\n│    - Starts approval timeout (1 hour)                   │\n└─────────────────────────────────────────────────────────┘\n```\n\n### Phase 2: Approval (10-15 minutes)\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ 3. First approver reviews request                      │\n│    - Verifies emergency is real                         │\n│    - Checks requester's justification                   │\n│    - Approves with reason                               │\n└─────────────────────────────────────────────────────────┘\n                          ↓\n┌─────────────────────────────────────────────────────────┐\n│ 4. Second approver (different team) reviews             │\n│    - Independent verification                            │\n│    - Approves with reason                               │\n└─────────────────────────────────────────────────────────┘\n                          ↓\n┌─────────────────────────────────────────────────────────┐\n│ 5. System validates approvals                           │\n│    - ✓ Min 2 approvers                                  │\n│    - ✓ Different teams                                  │\n│    - ✓ Within approval window                           │\n│    - Status → APPROVED                                  │\n└─────────────────────────────────────────────────────────┘\n```\n\n### Phase 3: Activation (1-2 minutes)\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ 6. Requester activates approved session                │\n│    - Receives emergency JWT token                       │\n│    - Token valid for 2 hours (or requested duration)    │\n│    - All actions logged with session ID                 │\n└─────────────────────────────────────────────────────────┘\n                          ↓\n┌─────────────────────────────────────────────────────────┐\n│ 7. Security team notified                               │\n│    - Real-time alert: "Break-glass activated"           │\n│    - Monitoring dashboard shows active session          │\n└─────────────────────────────────────────────────────────┘\n```\n\n### Phase 4: Usage (Variable)\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ 8. Requester performs emergency actions                │\n│    - Uses emergency token for access                    │\n│    - Every action audited                               │\n│    - Security team monitors in real-time                │\n└─────────────────────────────────────────────────────────┘\n                          ↓\n┌─────────────────────────────────────────────────────────┐\n│ 9. Background monitoring                                │\n│    - Checks for suspicious activity                     │\n│    - Enforces inactivity timeout (30 min)               │\n│    - Alerts on unusual patterns                         │\n└─────────────────────────────────────────────────────────┘\n```\n\n### Phase 5: Revocation (Immediate)\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ 10. Session ends (one of):                             │\n│     - Manual revocation by requester                    │\n│     - Expiration (max 4 hours)                          │\n│     - Inactivity timeout (30 minutes)                   │\n│     - Security team revocation                          │\n└─────────────────────────────────────────────────────────┘\n                          ↓\n┌─────────────────────────────────────────────────────────┐\n│ 11. System audit                                        │\n│     - All actions logged (7-year retention)             │\n│     - Incident report generated                         │\n│     - Post-incident review scheduled                    │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Using the System\n\n### CLI Commands\n\n#### 1. Request Emergency Access\n\n```\nprovisioning break-glass request \\n  "Production database cluster unresponsive" \\n  --justification "Need direct SSH access to diagnose PostgreSQL failure. \\n  Monitoring shows cluster down. Application offline affecting 10,000+ users." \\n  --resources '["database/*", "server/db-*"]' \\n  --duration 2hr\n\n# Output:\n# ✓ Break-glass request created\n# Request ID: BG-20251008-001\n# Status: Pending Approval\n# Approvers needed: 2\n# Expires: 2025-10-08 11:30:00 (1 hour)\n#\n# Notifications sent to:\n# - security-team@example.com\n# - platform-admin@example.com\n```\n\n#### 2. Approve Request (Approver)\n\n```\n# First approver (Security team)\nprovisioning break-glass approve BG-20251008-001 \\n  --reason "Emergency verified via incident INC-2025-234. Database cluster confirmed down, affecting production."\n\n# Output:\n# ✓ Approval granted\n# Approver: alice@example.com (Security Team)\n# Approvals: 1/2\n# Status: Pending (need 1 more approval)\n```\n\n```\n# Second approver (Platform team)\nprovisioning break-glass approve BG-20251008-001 \\n  --reason "Confirmed with monitoring. PostgreSQL master node unreachable. Emergency access justified."\n\n# Output:\n# ✓ Approval granted\n# Approver: bob@example.com (Platform Team)\n# Approvals: 2/2\n# Status: APPROVED\n#\n# Requester can now activate session\n```\n\n#### 3. Activate Session\n\n```\nprovisioning break-glass activate BG-20251008-001\n\n# Output:\n# ✓ Emergency session activated\n# Session ID: BGS-20251008-001\n# Token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...\n# Expires: 2025-10-08 12:30:00 (2 hours)\n# Max inactivity: 30 minutes\n#\n# ⚠️  WARNING ⚠️\n# - All actions are logged and monitored\n# - Security team has been notified\n# - Session will auto-revoke after 2 hours\n# - Use ONLY for stated emergency purpose\n#\n# Export token:\nexport EMERGENCY_TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."\n```\n\n#### 4. Use Emergency Access\n\n```\n# SSH to database server\nprovisioning ssh connect db-master-01 \\n  --token $EMERGENCY_TOKEN\n\n# Execute emergency commands\nsudo systemctl status postgresql\nsudo tail -f /var/log/postgresql/postgresql.log\n\n# Diagnose issue...\n# Fix issue...\n```\n\n#### 5. Revoke Session\n\n```\n# When done, immediately revoke\nprovisioning break-glass revoke BGS-20251008-001 \\n  --reason "Database cluster restored. PostgreSQL master node restarted successfully. All services online."\n\n# Output:\n# ✓ Emergency session revoked\n# Duration: 47 minutes\n# Actions performed: 23\n# Audit log: /var/log/provisioning/break-glass/BGS-20251008-001.json\n#\n# Post-incident review scheduled: 2025-10-09 10:00am\n```\n\n### Web UI (Control Center)\n\n#### Request Flow\n\n1. **Navigate**: Control Center → Security → Break-Glass\n2. **Click**: "Request Emergency Access"\n3. **Fill Form**:\n   - Reason: "Production database cluster down"\n   - Justification: (detailed description)\n   - Duration: 2 hours\n   - Resources: Select from dropdown or wildcard\n4. **Submit**: Request sent to approvers\n\n#### Approver Flow\n\n1. **Receive**: Email/Slack notification\n2. **Navigate**: Control Center → Break-Glass → Pending Requests\n3. **Review**: Request details, reason, justification\n4. **Decision**: Approve or Deny\n5. **Reason**: Provide approval/denial reason\n\n#### Monitor Active Sessions\n\n1. **Navigate**: Control Center → Security → Break-Glass → Active Sessions\n2. **View**: Real-time dashboard of active sessions\n   - Who, What, When, How long\n   - Actions performed (live)\n   - Inactivity timer\n3. **Revoke**: Emergency revoke button (if needed)\n\n---\n\n## Examples\n\n### Example 1: Production Database Outage\n\n**Scenario**: PostgreSQL cluster unresponsive, affecting all users\n\n**Request**:\n\n```\nprovisioning break-glass request \\n  "Production PostgreSQL cluster completely unresponsive" \\n  --justification "Database cluster (3 nodes) not responding. \\n  All services offline, 10,000+ users affected. Need SSH to diagnose. \\n  Monitoring shows all nodes down. Last state: replication failure during backup." \\n  --resources '["database/*", "server/db-prod-*"]' \\n  --duration 2hr\n```\n\n**Approval 1** (Security):\n> "Verified incident INC-2025-234. Database monitoring confirms cluster down. Application completely offline. Emergency justified."\n\n**Approval 2** (Platform):\n> "Confirmed. PostgreSQL master and replicas unreachable. On-call SRE needs immediate access. Approved."\n\n**Actions Taken**:\n\n1. SSH to db-prod-01, db-prod-02, db-prod-03\n2. Check PostgreSQL status: `systemctl status postgresql`\n3. Review logs: `/var/log/postgresql/`\n4. Diagnose: Disk full on master node\n5. Fix: Clear old WAL files, restart PostgreSQL\n6. Verify: Cluster restored, replication working\n7. Revoke access\n\n**Outcome**: Cluster restored in 47 minutes. Root cause: Backup retention not working.\n\n---\n\n### Example 2: Security Incident\n\n**Scenario**: Suspicious activity detected, need immediate containment\n\n**Request**:\n\n```\nprovisioning break-glass request \\n  "Active security breach detected - need immediate containment" \\n  --justification "IDS alerts show unauthorized access from IP 203.0.113.42 to API. \\n  Multiple failed sudo attempts. Isolate affected servers and investigate. \\n  Potential data exfiltration in progress." \\n  --resources '["server/api-prod-*", "firewall/*", "network/*"]' \\n  --duration 4hr\n```\n\n**Approval 1** (Security):\n> "Security incident SI-2025-089 confirmed. IDS shows sustained attack from external IP. Immediate containment required. Approved."\n\n**Approval 2** (Engineering Director):\n> "Concur with security assessment. Production impact acceptable vs risk of data breach. Approved."\n\n**Actions Taken**:\n\n1. Firewall block on 203.0.113.42\n2. Isolate affected API servers\n3. Snapshot servers for forensics\n4. Review access logs\n5. Identify compromised service account\n6. Rotate credentials\n7. Restore from clean backup\n8. Re-enable servers with patched vulnerability\n\n**Outcome**: Breach contained in 3h 15 min. No data loss. Vulnerability patched across fleet.\n\n---\n\n### Example 3: Accidental Data Deletion\n\n**Scenario**: Critical production data accidentally deleted\n\n**Request**:\n\n```\nprovisioning break-glass request \\n  "Critical customer data accidentally deleted from production" \\n  --justification "Database migration script ran against production instead of staging. \\n  50,000+ customer records deleted. Need immediate restore from backup. \\n  Normal restore requires 4-6 hours for approval. Time-critical window." \\n  --resources '["database/customers", "backup/*"]' \\n  --duration 3hr\n```\n\n**Approval 1** (Platform):\n> "Verified data deletion in production database. 50,284 records deleted at 10:42am. Backup available from 10:00am (42 minutes ago). Time-critical\nrestore needed. Approved."\n\n**Approval 2** (Security):\n> "Risk assessment: Restore from trusted backup less risky than data loss. Emergency justified. Ensure post-incident review of deployment process.\nApproved."\n\n**Actions Taken**:\n\n1. Stop application writes to affected tables\n2. Identify latest good backup (10:00am)\n3. Restore deleted records from backup\n4. Verify data integrity\n5. Compare record counts\n6. Re-enable application writes\n7. Notify affected users (if any noticed)\n\n**Outcome**: Data restored in 1h 38 min. Only 42 minutes of data lost (from backup to deletion). Zero customer impact.\n\n---\n\n## Auditing & Compliance\n\n### What is Logged\n\nEvery break-glass session logs:\n\n1. **Request Details**:\n   - Requester identity\n   - Reason and justification\n   - Requested resources\n   - Requested duration\n   - Timestamp\n\n2. **Approval Process**:\n   - Each approver identity\n   - Approval/denial reason\n   - Approval timestamp\n   - Team affiliation\n\n3. **Session Activity**:\n   - Activation timestamp\n   - Every action performed\n   - Resources accessed\n   - Commands executed\n   - Inactivity periods\n\n4. **Revocation**:\n   - Revocation reason\n   - Who revoked (system or manual)\n   - Total duration\n   - Final status\n\n### Retention\n\n- **Break-glass logs**: 7 years (immutable)\n- **Cannot be deleted**: Only anonymized for GDPR\n- **Exported to SIEM**: Real-time\n\n### Compliance Reports\n\n```\n# Generate break-glass usage report\nprovisioning break-glass audit \\n  --from "2025-01-01" \\n  --to "2025-12-31" \\n  --format pdf \\n  --output break-glass-2025-report.pdf\n\n# Report includes:\n# - Total break-glass activations\n# - Average duration\n# - Most common reasons\n# - Approval times\n# - Incidents resolved\n# - Misuse incidents (if any)\n```\n\n---\n\n## Post-Incident Review\n\n### Within 24 Hours\n\n**Required attendees**:\n\n- Requester\n- Approvers\n- Security team\n- Incident commander\n\n**Agenda**:\n\n1. **Timeline Review**: What happened, when\n2. **Actions Taken**: What was done with emergency access\n3. **Outcome**: Was issue resolved? Any side effects?\n4. **Process**: Did break-glass work as intended?\n5. **Lessons Learned**: What can be improved?\n\n### Review Checklist\n\n- [ ] Was break-glass appropriate for this incident?\n- [ ] Were approvals granted timely?\n- [ ] Was access used only for stated purpose?\n- [ ] Were any security policies violated?\n- [ ] Could incident be prevented in future?\n- [ ] Do we need policy updates?\n- [ ] Do we need system changes?\n\n### Output\n\n**Incident Report**:\n\n```\n# Break-Glass Incident Report: BG-20251008-001\n\n**Incident**: Production database cluster outage\n**Duration**: 47 minutes\n**Impact**: 10,000+ users, complete service outage\n\n## Timeline\n- 10:15: Incident detected\n- 10:17: Break-glass requested\n- 10:25: Approved (2/2)\n- 10:27: Activated\n- 11:02: Database restored\n- 11:04: Session revoked\n\n## Actions Taken\n1. SSH access to database servers\n2. Diagnosed disk full issue\n3. Cleared old WAL files\n4. Restarted PostgreSQL\n5. Verified replication\n\n## Root Cause\nBackup retention job failed silently for 2 weeks, causing WAL files to accumulate until disk full.\n\n## Prevention\n- ✅ Add disk space monitoring alerts\n- ✅ Fix backup retention job\n- ✅ Test recovery procedures\n- ✅ Implement WAL archiving to S3\n\n## Break-Glass Assessment\n- ✓ Appropriate use\n- ✓ Timely approvals\n- ✓ No policy violations\n- ✓ Access revoked promptly\n```\n\n---\n\n## FAQ\n\n### Q: How quickly can break-glass be activated\n\n**A**: Typically 15-20 minutes:\n\n- 5 min: Request submission\n- 10 min: Approvals (2 people)\n- 2 min: Activation\n\nIn extreme emergencies, approvers can be on standby.\n\n### Q: Can I use break-glass for scheduled maintenance\n\n**A**: No. Break-glass is for emergencies only. Schedule maintenance through normal change process.\n\n### Q: What if I can't get 2 approvers\n\n**A**: System requires 2 approvers from different teams. If unavailable:\n\n1. Escalate to on-call manager\n2. Contact security team directly\n3. Use emergency contact list\n\n### Q: Can approvers be from the same team\n\n**A**: No. System enforces team diversity to prevent collusion.\n\n### Q: What if security team revokes my session\n\n**A**: Security team can revoke for:\n\n- Suspicious activity\n- Policy violation\n- Incident resolved\n- Misuse detected\n\nYou'll receive immediate notification. Contact security team for details.\n\n### Q: Can I extend an active session\n\n**A**: No. Maximum duration is 4 hours. If you need more time, submit a new request with updated justification.\n\n### Q: What happens if I forget to revoke\n\n**A**: Session auto-revokes after:\n\n- Maximum duration (4 hours), OR\n- Inactivity timeout (30 minutes)\n\nAlways manually revoke when done.\n\n### Q: Is break-glass monitored\n\n**A**: Yes. Security team monitors in real-time:\n\n- Session activation alerts\n- Action logging\n- Suspicious activity detection\n- Compliance verification\n\n### Q: Can I practice break-glass\n\n**A**: Yes, in **development environment only**:\n\n```\nPROVISIONING_ENV=dev provisioning break-glass request "Test emergency access procedure"\n```\n\nNever practice in staging or production.\n\n---\n\n## Emergency Contacts\n\n### During Incident\n\n| Role | Contact | Response Time |\n| ------ | --------- | --------------- |\n| **Security On-Call** | +1-555-SECURITY | 5 minutes |\n| **Platform On-Call** | +1-555-PLATFORM | 5 minutes |\n| **Engineering Director** | +1-555-ENG-DIR | 15 minutes |\n\n### Escalation Path\n\n1. **L1**: On-call SRE\n2. **L2**: Platform team lead\n3. **L3**: Engineering manager\n4. **L4**: Director of Engineering\n5. **L5**: CTO\n\n### Communication Channels\n\n- **Incident Slack**: `#incidents`\n- **Security Slack**: `#security-alerts`\n- **Email**: `security-team@example.com`\n- **PagerDuty**: Break-glass policy\n\n---\n\n## Training Certification\n\n**I certify that I have**:\n\n- [ ] Read and understood this training guide\n- [ ] Understand when to use (and not use) break-glass\n- [ ] Know the approval workflow\n- [ ] Can use the CLI commands\n- [ ] Understand auditing and compliance requirements\n- [ ] Will follow post-incident review process\n\n**Signature**: _________________________\n**Date**: _________________________\n**Next Training Due**: _________________________ (1 year)\n\n---\n\n**Version**: 1.0.0\n**Maintained By**: Security Team\n**Last Updated**: 2025-10-08\n**Next Review**: 2026-10-08
