# Break-Glass Emergency Access - Training Guide

**Version**: 1.0.0
**Date**: 2025-10-08
**Audience**: Platform Administrators, SREs, Security Team
**Training Duration**: 45-60 minutes
**Certification**: Required annually

---

## 🚨 What is Break-Glass

**Break-glass** is an emergency access procedure that allows authorized personnel to bypass normal security controls during critical incidents (for example, production outages, security breaches, data loss).

### Key Principles

1. **Last Resort Only**: Use only when normal access is insufficient
2. **Multi-Party Approval**: Requires 2+ approvers from different teams
3. **Time-Limited**: Maximum 4 hours, auto-revokes
4. **Enhanced Audit**: 7-year retention, immutable logs
5. **Real-Time Alerts**: Security team notified immediately

---

## 📋 Table of Contents

1. [When to Use Break-Glass](#when-to-use-break-glass)
2. [When NOT to Use](#when-not-to-use)
3. [Roles & Responsibilities](#roles--responsibilities)
4. [Break-Glass Workflow](#break-glass-workflow)
5. [Using the System](#using-the-system)
6. [Examples](#examples)
7. [Auditing & Compliance](#auditing--compliance)
8. [Post-Incident Review](#post-incident-review)
9. [FAQ](#faq)
10. [Emergency Contacts](#emergency-contacts)

---

## When to Use Break-Glass

### ✅ Valid Emergency Scenarios

| Scenario | Example | Urgency |
|----------|---------|---------|
| **Production Outage** | Database cluster unresponsive, affecting all users | Critical |
| **Security Incident** | Active breach detected, need immediate containment | Critical |
| **Data Loss** | Accidental deletion of critical data, need restore | High |
| **System Failure** | Infrastructure failure requiring emergency fixes | High |
| **Locked Out** | Normal admin accounts compromised, need recovery | High |

### Criteria Checklist

Use break-glass if **ALL** apply:

- [ ] Production systems affected OR security incident
- [ ] Normal access insufficient OR unavailable
- [ ] Immediate action required (cannot wait for approval process)
- [ ] Clear justification for emergency access
- [ ] Incident properly documented

---

## When NOT to Use

### ❌ Invalid Scenarios (Do NOT Use Break-Glass)

| Scenario | Why Not | Alternative |
|----------|---------|-------------|
| Forgot password | Not an emergency | Use password reset |
| Routine maintenance | Can be scheduled | Use normal change process |
| Convenience | Normal process "too slow" | Follow standard approval |
| Deadline pressure | Business pressure ≠ emergency | Plan ahead |
| Testing | Want to test emergency access | Use dev environment |

### Consequences of Misuse

- Immediate suspension of break-glass privileges
- Security team investigation
- Disciplinary action (up to termination)
- All actions audited and reviewed

---

## Roles & Responsibilities

### Requester

**Who**: Platform Admin, SRE on-call, Security Officer
**Responsibilities**:

- Assess if situation warrants emergency access
- Provide clear justification and reason
- Document incident timeline
- Use access only for stated purpose
- Revoke access immediately after resolution

### Approvers

**Who**: 2+ from different teams (Security, Platform, Engineering Leadership)
**Responsibilities**:

- Verify emergency is genuine
- Assess risk of granting access
- Review requester's justification
- Monitor usage during active session
- Participate in post-incident review

### Security Team

**Who**: Security Operations team
**Responsibilities**:

- Monitor all break-glass activations (real-time)
- Review audit logs during session
- Alert on suspicious activity
- Lead post-incident review
- Update policies based on learnings

---

## Break-Glass Workflow

### Phase 1: Request (5 minutes)

```plaintext
┌─────────────────────────────────────────────────────────┐
│ 1. Requester submits emergency access request          │
│    - Reason: "Production database cluster down"        │
│    - Justification: "Need direct SSH to diagnose"      │
│    - Duration: 2 hours                                  │
│    - Resources: ["database/*"]                          │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ 2. System creates request ID: BG-20251008-001          │
│    - Sends notifications to approver pool               │
│    - Starts approval timeout (1 hour)                   │
└─────────────────────────────────────────────────────────┘
```plaintext

### Phase 2: Approval (10-15 minutes)

```plaintext
┌─────────────────────────────────────────────────────────┐
│ 3. First approver reviews request                      │
│    - Verifies emergency is real                         │
│    - Checks requester's justification                   │
│    - Approves with reason                               │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ 4. Second approver (different team) reviews             │
│    - Independent verification                            │
│    - Approves with reason                               │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ 5. System validates approvals                           │
│    - ✓ Min 2 approvers                                  │
│    - ✓ Different teams                                  │
│    - ✓ Within approval window                           │
│    - Status → APPROVED                                  │
└─────────────────────────────────────────────────────────┘
```plaintext

### Phase 3: Activation (1-2 minutes)

```plaintext
┌─────────────────────────────────────────────────────────┐
│ 6. Requester activates approved session                │
│    - Receives emergency JWT token                       │
│    - Token valid for 2 hours (or requested duration)    │
│    - All actions logged with session ID                 │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ 7. Security team notified                               │
│    - Real-time alert: "Break-glass activated"           │
│    - Monitoring dashboard shows active session          │
└─────────────────────────────────────────────────────────┘
```plaintext

### Phase 4: Usage (Variable)

```plaintext
┌─────────────────────────────────────────────────────────┐
│ 8. Requester performs emergency actions                │
│    - Uses emergency token for access                    │
│    - Every action audited                               │
│    - Security team monitors in real-time                │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ 9. Background monitoring                                │
│    - Checks for suspicious activity                     │
│    - Enforces inactivity timeout (30 min)               │
│    - Alerts on unusual patterns                         │
└─────────────────────────────────────────────────────────┘
```plaintext

### Phase 5: Revocation (Immediate)

```plaintext
┌─────────────────────────────────────────────────────────┐
│ 10. Session ends (one of):                             │
│     - Manual revocation by requester                    │
│     - Expiration (max 4 hours)                          │
│     - Inactivity timeout (30 minutes)                   │
│     - Security team revocation                          │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ 11. System audit                                        │
│     - All actions logged (7-year retention)             │
│     - Incident report generated                         │
│     - Post-incident review scheduled                    │
└─────────────────────────────────────────────────────────┘
```plaintext

---

## Using the System

### CLI Commands

#### 1. Request Emergency Access

```bash
provisioning break-glass request \
  "Production database cluster unresponsive" \
  --justification "Need direct SSH access to diagnose PostgreSQL failure. All monitoring shows cluster down. Application completely offline affecting 10,000+ users." \
  --resources '["database/*", "server/db-*"]' \
  --duration 2hr

# Output:
# ✓ Break-glass request created
# Request ID: BG-20251008-001
# Status: Pending Approval
# Approvers needed: 2
# Expires: 2025-10-08 11:30:00 (1 hour)
#
# Notifications sent to:
# - security-team@example.com
# - platform-admin@example.com
```plaintext

#### 2. Approve Request (Approver)

```bash
# First approver (Security team)
provisioning break-glass approve BG-20251008-001 \
  --reason "Emergency verified via incident INC-2025-234. Database cluster confirmed down, affecting production."

# Output:
# ✓ Approval granted
# Approver: alice@example.com (Security Team)
# Approvals: 1/2
# Status: Pending (need 1 more approval)
```plaintext

```bash
# Second approver (Platform team)
provisioning break-glass approve BG-20251008-001 \
  --reason "Confirmed with monitoring. PostgreSQL master node unreachable. Emergency access justified."

# Output:
# ✓ Approval granted
# Approver: bob@example.com (Platform Team)
# Approvals: 2/2
# Status: APPROVED
#
# Requester can now activate session
```plaintext

#### 3. Activate Session

```bash
provisioning break-glass activate BG-20251008-001

# Output:
# ✓ Emergency session activated
# Session ID: BGS-20251008-001
# Token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...
# Expires: 2025-10-08 12:30:00 (2 hours)
# Max inactivity: 30 minutes
#
# ⚠️  WARNING ⚠️
# - All actions are logged and monitored
# - Security team has been notified
# - Session will auto-revoke after 2 hours
# - Use ONLY for stated emergency purpose
#
# Export token:
export EMERGENCY_TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
```plaintext

#### 4. Use Emergency Access

```bash
# SSH to database server
provisioning ssh connect db-master-01 \
  --token $EMERGENCY_TOKEN

# Execute emergency commands
sudo systemctl status postgresql
sudo tail -f /var/log/postgresql/postgresql.log

# Diagnose issue...
# Fix issue...
```plaintext

#### 5. Revoke Session

```bash
# When done, immediately revoke
provisioning break-glass revoke BGS-20251008-001 \
  --reason "Database cluster restored. PostgreSQL master node restarted successfully. All services online."

# Output:
# ✓ Emergency session revoked
# Duration: 47 minutes
# Actions performed: 23
# Audit log: /var/log/provisioning/break-glass/BGS-20251008-001.json
#
# Post-incident review scheduled: 2025-10-09 10:00am
```plaintext

### Web UI (Control Center)

#### Request Flow

1. **Navigate**: Control Center → Security → Break-Glass
2. **Click**: "Request Emergency Access"
3. **Fill Form**:
   - Reason: "Production database cluster down"
   - Justification: (detailed description)
   - Duration: 2 hours
   - Resources: Select from dropdown or wildcard
4. **Submit**: Request sent to approvers

#### Approver Flow

1. **Receive**: Email/Slack notification
2. **Navigate**: Control Center → Break-Glass → Pending Requests
3. **Review**: Request details, reason, justification
4. **Decision**: Approve or Deny
5. **Reason**: Provide approval/denial reason

#### Monitor Active Sessions

1. **Navigate**: Control Center → Security → Break-Glass → Active Sessions
2. **View**: Real-time dashboard of active sessions
   - Who, What, When, How long
   - Actions performed (live)
   - Inactivity timer
3. **Revoke**: Emergency revoke button (if needed)

---

## Examples

### Example 1: Production Database Outage

**Scenario**: PostgreSQL cluster unresponsive, affecting all users

**Request**:

```bash
provisioning break-glass request \
  "Production PostgreSQL cluster completely unresponsive" \
  --justification "Database cluster (3 nodes) not responding. All application services offline. 10,000+ users affected. Need direct SSH to diagnose and restore. Monitoring shows all nodes down. Last known state: replication failure during routine backup." \
  --resources '["database/*", "server/db-prod-*"]' \
  --duration 2hr
```plaintext

**Approval 1** (Security):
> "Verified incident INC-2025-234. Database monitoring confirms cluster down. Application completely offline. Emergency justified."

**Approval 2** (Platform):
> "Confirmed. PostgreSQL master and replicas unreachable. On-call SRE needs immediate access. Approved."

**Actions Taken**:

1. SSH to db-prod-01, db-prod-02, db-prod-03
2. Check PostgreSQL status: `systemctl status postgresql`
3. Review logs: `/var/log/postgresql/`
4. Diagnose: Disk full on master node
5. Fix: Clear old WAL files, restart PostgreSQL
6. Verify: Cluster restored, replication working
7. Revoke access

**Outcome**: Cluster restored in 47 minutes. Root cause: Backup retention not working.

---

### Example 2: Security Incident

**Scenario**: Suspicious activity detected, need immediate containment

**Request**:

```bash
provisioning break-glass request \
  "Active security breach detected - need immediate containment" \
  --justification "IDS alerts show unauthorized access from IP 203.0.113.42 to production API servers. Multiple failed sudo attempts. Need to isolate affected servers and investigate. Potential data exfiltration in progress." \
  --resources '["server/api-prod-*", "firewall/*", "network/*"]' \
  --duration 4hr
```plaintext

**Approval 1** (Security):
> "Security incident SI-2025-089 confirmed. IDS shows sustained attack from external IP. Immediate containment required. Approved."

**Approval 2** (Engineering Director):
> "Concur with security assessment. Production impact acceptable vs risk of data breach. Approved."

**Actions Taken**:

1. Firewall block on 203.0.113.42
2. Isolate affected API servers
3. Snapshot servers for forensics
4. Review access logs
5. Identify compromised service account
6. Rotate credentials
7. Restore from clean backup
8. Re-enable servers with patched vulnerability

**Outcome**: Breach contained in 3h 15 min. No data loss. Vulnerability patched across fleet.

---

### Example 3: Accidental Data Deletion

**Scenario**: Critical production data accidentally deleted

**Request**:

```bash
provisioning break-glass request \
  "Critical customer data accidentally deleted from production" \
  --justification "Database migration script ran against production instead of staging. Deleted 50,000+ customer records. Need immediate restore from backup before data loss is noticed. Normal restore process requires change approval (4-6 hours). Data loss window critical." \
  --resources '["database/customers", "backup/*"]' \
  --duration 3hr
```plaintext

**Approval 1** (Platform):
> "Verified data deletion in production database. 50,284 records deleted at 10:42am. Backup available from 10:00am (42 minutes ago). Time-critical restore needed. Approved."

**Approval 2** (Security):
> "Risk assessment: Restore from trusted backup less risky than data loss. Emergency justified. Ensure post-incident review of deployment process. Approved."

**Actions Taken**:

1. Stop application writes to affected tables
2. Identify latest good backup (10:00am)
3. Restore deleted records from backup
4. Verify data integrity
5. Compare record counts
6. Re-enable application writes
7. Notify affected users (if any noticed)

**Outcome**: Data restored in 1h 38 min. Only 42 minutes of data lost (from backup to deletion). Zero customer impact.

---

## Auditing & Compliance

### What is Logged

Every break-glass session logs:

1. **Request Details**:
   - Requester identity
   - Reason and justification
   - Requested resources
   - Requested duration
   - Timestamp

2. **Approval Process**:
   - Each approver identity
   - Approval/denial reason
   - Approval timestamp
   - Team affiliation

3. **Session Activity**:
   - Activation timestamp
   - Every action performed
   - Resources accessed
   - Commands executed
   - Inactivity periods

4. **Revocation**:
   - Revocation reason
   - Who revoked (system or manual)
   - Total duration
   - Final status

### Retention

- **Break-glass logs**: 7 years (immutable)
- **Cannot be deleted**: Only anonymized for GDPR
- **Exported to SIEM**: Real-time

### Compliance Reports

```bash
# Generate break-glass usage report
provisioning break-glass audit \
  --from "2025-01-01" \
  --to "2025-12-31" \
  --format pdf \
  --output break-glass-2025-report.pdf

# Report includes:
# - Total break-glass activations
# - Average duration
# - Most common reasons
# - Approval times
# - Incidents resolved
# - Misuse incidents (if any)
```plaintext

---

## Post-Incident Review

### Within 24 Hours

**Required attendees**:

- Requester
- Approvers
- Security team
- Incident commander

**Agenda**:

1. **Timeline Review**: What happened, when
2. **Actions Taken**: What was done with emergency access
3. **Outcome**: Was issue resolved? Any side effects?
4. **Process**: Did break-glass work as intended?
5. **Lessons Learned**: What can be improved?

### Review Checklist

- [ ] Was break-glass appropriate for this incident?
- [ ] Were approvals granted timely?
- [ ] Was access used only for stated purpose?
- [ ] Were any security policies violated?
- [ ] Could incident be prevented in future?
- [ ] Do we need policy updates?
- [ ] Do we need system changes?

### Output

**Incident Report**:

```markdown
# Break-Glass Incident Report: BG-20251008-001

**Incident**: Production database cluster outage
**Duration**: 47 minutes
**Impact**: 10,000+ users, complete service outage

## Timeline
- 10:15: Incident detected
- 10:17: Break-glass requested
- 10:25: Approved (2/2)
- 10:27: Activated
- 11:02: Database restored
- 11:04: Session revoked

## Actions Taken
1. SSH access to database servers
2. Diagnosed disk full issue
3. Cleared old WAL files
4. Restarted PostgreSQL
5. Verified replication

## Root Cause
Backup retention job failed silently for 2 weeks, causing WAL files to accumulate until disk full.

## Prevention
- ✅ Add disk space monitoring alerts
- ✅ Fix backup retention job
- ✅ Test recovery procedures
- ✅ Implement WAL archiving to S3

## Break-Glass Assessment
- ✓ Appropriate use
- ✓ Timely approvals
- ✓ No policy violations
- ✓ Access revoked promptly
```plaintext

---

## FAQ

### Q: How quickly can break-glass be activated

**A**: Typically 15-20 minutes:

- 5 min: Request submission
- 10 min: Approvals (2 people)
- 2 min: Activation

In extreme emergencies, approvers can be on standby.

### Q: Can I use break-glass for scheduled maintenance

**A**: No. Break-glass is for emergencies only. Schedule maintenance through normal change process.

### Q: What if I can't get 2 approvers

**A**: System requires 2 approvers from different teams. If unavailable:

1. Escalate to on-call manager
2. Contact security team directly
3. Use emergency contact list

### Q: Can approvers be from the same team

**A**: No. System enforces team diversity to prevent collusion.

### Q: What if security team revokes my session

**A**: Security team can revoke for:

- Suspicious activity
- Policy violation
- Incident resolved
- Misuse detected

You'll receive immediate notification. Contact security team for details.

### Q: Can I extend an active session

**A**: No. Maximum duration is 4 hours. If you need more time, submit a new request with updated justification.

### Q: What happens if I forget to revoke

**A**: Session auto-revokes after:

- Maximum duration (4 hours), OR
- Inactivity timeout (30 minutes)

Always manually revoke when done.

### Q: Is break-glass monitored

**A**: Yes. Security team monitors in real-time:

- Session activation alerts
- Action logging
- Suspicious activity detection
- Compliance verification

### Q: Can I practice break-glass

**A**: Yes, in **development environment only**:

```bash
PROVISIONING_ENV=dev provisioning break-glass request "Test emergency access procedure"
```plaintext

Never practice in staging or production.

---

## Emergency Contacts

### During Incident

| Role | Contact | Response Time |
|------|---------|---------------|
| **Security On-Call** | +1-555-SECURITY | 5 minutes |
| **Platform On-Call** | +1-555-PLATFORM | 5 minutes |
| **Engineering Director** | +1-555-ENG-DIR | 15 minutes |

### Escalation Path

1. **L1**: On-call SRE
2. **L2**: Platform team lead
3. **L3**: Engineering manager
4. **L4**: Director of Engineering
5. **L5**: CTO

### Communication Channels

- **Incident Slack**: `#incidents`
- **Security Slack**: `#security-alerts`
- **Email**: `security-team@example.com`
- **PagerDuty**: Break-glass policy

---

## Training Certification

**I certify that I have**:

- [ ] Read and understood this training guide
- [ ] Understand when to use (and not use) break-glass
- [ ] Know the approval workflow
- [ ] Can use the CLI commands
- [ ] Understand auditing and compliance requirements
- [ ] Will follow post-incident review process

**Signature**: _________________________
**Date**: _________________________
**Next Training Due**: _________________________ (1 year)

---

**Version**: 1.0.0
**Maintained By**: Security Team
**Last Updated**: 2025-10-08
**Next Review**: 2026-10-08
