695 lines
17 KiB
Markdown
695 lines
17 KiB
Markdown
# Deployment Runbook
|
|
|
|
Step-by-step procedures for deploying VAPORA to staging and production environments.
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
For experienced operators:
|
|
|
|
```bash
|
|
# Validate in CI/CD
|
|
# Download artifacts
|
|
# Review dry-run
|
|
# Apply: kubectl apply -f configmap.yaml deployment.yaml
|
|
# Monitor: kubectl logs -f deployment/vapora-backend -n vapora
|
|
# Verify: curl http://localhost:8001/health
|
|
```
|
|
|
|
For complete steps, continue reading.
|
|
|
|
---
|
|
|
|
## Before Starting
|
|
|
|
✅ **Prerequisites Completed**:
|
|
- [ ] Pre-deployment checklist completed
|
|
- [ ] Artifacts generated and validated
|
|
- [ ] Staging deployment verified
|
|
- [ ] Team ready and monitoring
|
|
- [ ] Maintenance window announced
|
|
|
|
✅ **Access Verified**:
|
|
- [ ] kubectl configured for target cluster
|
|
- [ ] Can list nodes: `kubectl get nodes`
|
|
- [ ] Can access namespace: `kubectl get namespace vapora`
|
|
|
|
❌ **If any prerequisite missing**: Go back to pre-deployment checklist
|
|
|
|
---
|
|
|
|
## Phase 1: Pre-Flight (5 minutes)
|
|
|
|
### 1.1 Verify Current State
|
|
|
|
```bash
|
|
# Set context
|
|
export CLUSTER=production # or staging
|
|
export NAMESPACE=vapora
|
|
|
|
# Verify cluster access
|
|
kubectl cluster-info
|
|
kubectl get nodes
|
|
|
|
# Output should show:
|
|
# NAME STATUS ROLES AGE
|
|
# node-1 Ready worker 30d
|
|
# node-2 Ready worker 25d
|
|
```
|
|
|
|
**What to look for:**
|
|
- ✓ All nodes in "Ready" state
|
|
- ✓ No "NotReady" or "Unknown" nodes
|
|
- If issues: Don't proceed, investigate node health
|
|
|
|
### 1.2 Check Current Deployments
|
|
|
|
```bash
|
|
# Get current deployment status
|
|
kubectl get deployments -n $NAMESPACE -o wide
|
|
kubectl get pods -n $NAMESPACE
|
|
|
|
# Output example:
|
|
# NAME READY UP-TO-DATE AVAILABLE
|
|
# vapora-backend 3/3 3 3
|
|
# vapora-agents 2/2 2 2
|
|
# vapora-llm-router 2/2 2 2
|
|
```
|
|
|
|
**What to look for:**
|
|
- ✓ All deployments showing correct replica count
|
|
- ✓ All pods in "Running" state
|
|
- ❌ If pods in "CrashLoopBackOff" or "Pending": Investigate before proceeding
|
|
|
|
### 1.3 Record Current Versions
|
|
|
|
```bash
|
|
# Get current image versions (baseline for rollback)
|
|
kubectl get deployments -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}'
|
|
|
|
# Expected output:
|
|
# vapora-backend vapora/backend:v1.2.0
|
|
# vapora-agents vapora/agents:v1.2.0
|
|
# vapora-llm-router vapora/llm-router:v1.2.0
|
|
```
|
|
|
|
**Record these for rollback**: Keep this output visible
|
|
|
|
### 1.4 Get Current Revision Numbers
|
|
|
|
```bash
|
|
# For each deployment, get rollout history
|
|
for deployment in vapora-backend vapora-agents vapora-llm-router; do
|
|
echo "=== $deployment ==="
|
|
kubectl rollout history deployment/$deployment -n $NAMESPACE | tail -5
|
|
done
|
|
|
|
# Output example:
|
|
# REVISION CHANGE-CAUSE
|
|
# 42 Deployment rolled out
|
|
# 43 Deployment rolled out
|
|
# 44 (current)
|
|
```
|
|
|
|
**Record the highest revision number for each** - this is your rollback reference
|
|
|
|
### 1.5 Check Cluster Resources
|
|
|
|
```bash
|
|
# Verify cluster has capacity for new deployment
|
|
kubectl top nodes
|
|
kubectl describe nodes | grep -A 5 "Allocated resources"
|
|
|
|
# Example - check memory/CPU availability
|
|
# Requested: 8200m (41%)
|
|
# Limits: 16400m (82%)
|
|
```
|
|
|
|
**What to look for:**
|
|
- ✓ Less than 80% resource utilization
|
|
- ❌ If above 85%: Insufficient capacity, don't proceed
|
|
|
|
---
|
|
|
|
## Phase 2: Configuration Deployment (3 minutes)
|
|
|
|
### 2.1 Apply ConfigMap
|
|
|
|
The ConfigMap contains all application configuration.
|
|
|
|
```bash
|
|
# First: Dry-run to verify no syntax errors
|
|
kubectl apply -f configmap.yaml --dry-run=server -n $NAMESPACE
|
|
|
|
# Should output:
|
|
# configmap/vapora-config configured (server dry run)
|
|
|
|
# Check for any warnings or errors in output
|
|
# If errors, stop and fix the YAML before proceeding
|
|
```
|
|
|
|
**Troubleshooting**:
|
|
- "error validating": YAML syntax error - fix and retry
|
|
- "field is immutable": Can't change certain ConfigMap fields - delete and recreate
|
|
- "resourceQuotaExceeded": Namespace quota exceeded - contact cluster admin
|
|
|
|
### 2.2 Apply ConfigMap for Real
|
|
|
|
```bash
|
|
# Apply the actual ConfigMap
|
|
kubectl apply -f configmap.yaml -n $NAMESPACE
|
|
|
|
# Output:
|
|
# configmap/vapora-config configured
|
|
|
|
# Verify it was applied
|
|
kubectl get configmap -n $NAMESPACE vapora-config -o yaml | head -20
|
|
|
|
# Check for your new values in the output
|
|
```
|
|
|
|
**Verify ConfigMap is correct**:
|
|
```bash
|
|
# Extract specific values to verify
|
|
kubectl get configmap vapora-config -n $NAMESPACE -o jsonpath='{.data.vapora\.toml}' | grep "database_url" | head -1
|
|
|
|
# Should show the correct database URL
|
|
```
|
|
|
|
### 2.3 Annotate ConfigMap
|
|
|
|
Record when this config was deployed for audit trail:
|
|
|
|
```bash
|
|
kubectl annotate configmap vapora-config \
|
|
-n $NAMESPACE \
|
|
deployment.timestamp="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
|
|
deployment.commit="$(git rev-parse HEAD | cut -c1-8)" \
|
|
deployment.branch="$(git rev-parse --abbrev-ref HEAD)" \
|
|
--overwrite
|
|
|
|
# Verify annotation was added
|
|
kubectl get configmap vapora-config -n $NAMESPACE -o yaml | grep "deployment\."
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 3: Deployment Update (5 minutes)
|
|
|
|
### 3.1 Dry-Run Deployment
|
|
|
|
Always dry-run first to catch issues:
|
|
|
|
```bash
|
|
# Run deployment dry-run
|
|
kubectl apply -f deployment.yaml --dry-run=server -n $NAMESPACE
|
|
|
|
# Output should show what will be updated:
|
|
# deployment.apps/vapora-backend configured (server dry run)
|
|
# deployment.apps/vapora-agents configured (server dry run)
|
|
# deployment.apps/vapora-llm-router configured (server dry run)
|
|
```
|
|
|
|
**Check for warnings**:
|
|
- "imagePullBackOff": Docker image doesn't exist
|
|
- "insufficient quota": Resource limits exceeded
|
|
- "nodeAffinity": Pod can't be placed on any node
|
|
|
|
### 3.2 Apply Deployments
|
|
|
|
```bash
|
|
# Apply the actual deployments
|
|
kubectl apply -f deployment.yaml -n $NAMESPACE
|
|
|
|
# Output:
|
|
# deployment.apps/vapora-backend configured
|
|
# deployment.apps/vapora-agents configured
|
|
# deployment.apps/vapora-llm-router configured
|
|
```
|
|
|
|
**Verify deployments updated**:
|
|
```bash
|
|
# Check that new rollout was initiated
|
|
kubectl get deployments -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.observedGeneration}{"\n"}{end}'
|
|
|
|
# Compare with recorded versions - should be incremented
|
|
```
|
|
|
|
### 3.3 Monitor Rollout Progress
|
|
|
|
Watch the deployment rollout status:
|
|
|
|
```bash
|
|
# For each deployment, monitor the rollout
|
|
for deployment in vapora-backend vapora-agents vapora-llm-router; do
|
|
echo "Waiting for $deployment..."
|
|
kubectl rollout status deployment/$deployment \
|
|
-n $NAMESPACE \
|
|
--timeout=5m
|
|
echo "$deployment ready"
|
|
done
|
|
```
|
|
|
|
**What to look for** (per pod update):
|
|
```
|
|
Waiting for rollout to finish: 2 of 3 updated replicas are available...
|
|
Waiting for rollout to finish: 2 of 3 updated replicas are available...
|
|
Waiting for rollout to finish: 3 of 3 updated replicas are available...
|
|
deployment "vapora-backend" successfully rolled out
|
|
```
|
|
|
|
**Expected time: 2-3 minutes per deployment**
|
|
|
|
### 3.4 Watch Pod Updates (in separate terminal)
|
|
|
|
While rollout completes, monitor pods:
|
|
|
|
```bash
|
|
# Watch pods being updated in real-time
|
|
kubectl get pods -n $NAMESPACE -w
|
|
|
|
# Output shows updates like:
|
|
# NAME READY STATUS
|
|
# vapora-backend-abc123-def45 1/1 Running
|
|
# vapora-backend-xyz789-old-pod 1/1 Running ← old pod still running
|
|
# vapora-backend-abc123-new-pod 0/1 Pending ← new pod starting
|
|
# vapora-backend-abc123-new-pod 0/1 ContainerCreating
|
|
# vapora-backend-abc123-new-pod 1/1 Running ← new pod ready
|
|
# vapora-backend-xyz789-old-pod 1/1 Terminating ← old pod being removed
|
|
```
|
|
|
|
**What to look for:**
|
|
- ✓ New pods starting (Pending → ContainerCreating → Running)
|
|
- ✓ Each new pod reaches Running state
|
|
- ✓ Old pods gradually terminating
|
|
- ❌ Pod stuck in "CrashLoopBackOff": Stop, check logs, might need rollback
|
|
|
|
---
|
|
|
|
## Phase 4: Verification (5 minutes)
|
|
|
|
### 4.1 Verify All Pods Running
|
|
|
|
```bash
|
|
# Check all pods are ready
|
|
kubectl get pods -n $NAMESPACE
|
|
|
|
# Expected output:
|
|
# NAME READY STATUS
|
|
# vapora-backend-<hash>-1 1/1 Running
|
|
# vapora-backend-<hash>-2 1/1 Running
|
|
# vapora-backend-<hash>-3 1/1 Running
|
|
# vapora-agents-<hash>-1 1/1 Running
|
|
# vapora-agents-<hash>-2 1/1 Running
|
|
# vapora-llm-router-<hash>-1 1/1 Running
|
|
# vapora-llm-router-<hash>-2 1/1 Running
|
|
```
|
|
|
|
**Verification**:
|
|
```bash
|
|
# All pods should show READY=1/1
|
|
# All pods should show STATUS=Running
|
|
# No pods should be in Pending, CrashLoopBackOff, or Error state
|
|
|
|
# Quick check:
|
|
READY=$(kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c "True")
|
|
TOTAL=$(kubectl get pods -n $NAMESPACE --no-headers | wc -l)
|
|
|
|
echo "Ready pods: $READY / $TOTAL"
|
|
|
|
# Should show: Ready pods: 7 / 7 (or your expected pod count)
|
|
```
|
|
|
|
### 4.2 Check Pod Logs for Errors
|
|
|
|
```bash
|
|
# Check logs from the last minute for errors
|
|
for pod in $(kubectl get pods -n $NAMESPACE -o name); do
|
|
echo "=== $pod ==="
|
|
kubectl logs $pod -n $NAMESPACE --since=1m 2>&1 | grep -i "error\|exception\|fatal" | head -3
|
|
done
|
|
|
|
# If errors found:
|
|
# 1. Note which pods have errors
|
|
# 2. Get full log: kubectl logs <pod> -n $NAMESPACE
|
|
# 3. Decide: can proceed or need to rollback
|
|
```
|
|
|
|
### 4.3 Verify Service Endpoints
|
|
|
|
```bash
|
|
# Check services are exposing pods correctly
|
|
kubectl get endpoints -n $NAMESPACE
|
|
|
|
# Expected output:
|
|
# NAME ENDPOINTS
|
|
# vapora-backend 10.1.2.3:8001,10.1.2.4:8001,10.1.2.5:8001
|
|
# vapora-agents 10.1.2.6:8002,10.1.2.7:8002
|
|
# vapora-llm-router 10.1.2.8:8003,10.1.2.9:8003
|
|
```
|
|
|
|
**Verification**:
|
|
- ✓ Each service has multiple endpoints (not empty)
|
|
- ✓ Endpoints match running pods
|
|
- ❌ If empty endpoints: Service can't route traffic
|
|
|
|
### 4.4 Health Check Endpoints
|
|
|
|
```bash
|
|
# Port-forward to access services locally
|
|
kubectl port-forward -n $NAMESPACE svc/vapora-backend 8001:8001 &
|
|
|
|
# Wait a moment for port-forward to establish
|
|
sleep 2
|
|
|
|
# Check backend health
|
|
curl -v http://localhost:8001/health
|
|
|
|
# Expected response:
|
|
# HTTP/1.1 200 OK
|
|
# {...healthy response...}
|
|
|
|
# Check other endpoints
|
|
curl http://localhost:8001/api/projects -H "Authorization: Bearer test-token"
|
|
```
|
|
|
|
**Expected responses**:
|
|
- `/health`: 200 OK with health data
|
|
- `/api/projects`: 200 OK with projects list
|
|
- `/metrics`: 200 OK with Prometheus metrics
|
|
|
|
**If connection refused**:
|
|
```bash
|
|
# Check if port-forward working
|
|
ps aux | grep "port-forward"
|
|
|
|
# Restart port-forward
|
|
pkill -f "port-forward svc/vapora-backend"
|
|
kubectl port-forward -n $NAMESPACE svc/vapora-backend 8001:8001 &
|
|
```
|
|
|
|
### 4.5 Check Metrics
|
|
|
|
```bash
|
|
# Monitor resource usage of deployed pods
|
|
kubectl top pods -n $NAMESPACE
|
|
|
|
# Expected output:
|
|
# NAME CPU(cores) MEMORY(Mi)
|
|
# vapora-backend-abc123 250m 512Mi
|
|
# vapora-backend-def456 280m 498Mi
|
|
# vapora-agents-ghi789 300m 256Mi
|
|
```
|
|
|
|
**Verification**:
|
|
- ✓ CPU usage within expected range (typically 100-500m per pod)
|
|
- ✓ Memory usage within expected range (typically 200-512Mi)
|
|
- ❌ If any pod at 100% CPU/Memory: Performance issue, monitor closely
|
|
|
|
---
|
|
|
|
## Phase 5: Validation (3 minutes)
|
|
|
|
### 5.1 Run Smoke Tests (if available)
|
|
|
|
```bash
|
|
# If your project has smoke tests:
|
|
kubectl exec -it deployment/vapora-backend -n $NAMESPACE -- \
|
|
sh -c "curl http://localhost:8001/health && echo 'Health check passed'"
|
|
|
|
# Or run from your local machine:
|
|
./scripts/smoke-tests.sh --endpoint http://localhost:8001
|
|
```
|
|
|
|
### 5.2 Check for Errors in Logs
|
|
|
|
```bash
|
|
# Look at logs from all pods since deployment started
|
|
for deployment in vapora-backend vapora-agents vapora-llm-router; do
|
|
echo "=== Checking $deployment ==="
|
|
kubectl logs deployment/$deployment -n $NAMESPACE --since=5m 2>&1 | \
|
|
grep -i "error\|exception\|failed" | wc -l
|
|
done
|
|
|
|
# If any errors found:
|
|
# 1. Get detailed logs
|
|
# 2. Determine if critical or expected errors
|
|
# 3. Decide to proceed or rollback
|
|
```
|
|
|
|
### 5.3 Compare Against Baseline Metrics
|
|
|
|
Compare current metrics with pre-deployment baseline:
|
|
|
|
```bash
|
|
# Current metrics
|
|
echo "=== Current ==="
|
|
kubectl top nodes
|
|
kubectl top pods -n $NAMESPACE | head -5
|
|
|
|
# Compare with recorded baseline
|
|
# If similar: ✓ Good
|
|
# If significantly higher: ⚠️ Watch for issues
|
|
# If error rates high: ❌ Consider rollback
|
|
```
|
|
|
|
### 5.4 Check for Recent Events/Warnings
|
|
|
|
```bash
|
|
# Look for any cluster events in the last 5 minutes
|
|
kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -20
|
|
|
|
# Watch for:
|
|
# - Warning: FailedScheduling (pod won't fit)
|
|
# - Warning: PullImageError (image doesn't exist)
|
|
# - Warning: ImagePullBackOff (can't download image)
|
|
# - Error: ExceededQuota (resource limits)
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 6: Communication (1 minute)
|
|
|
|
### 6.1 Post Deployment Complete
|
|
|
|
```
|
|
Post message to #deployments:
|
|
|
|
🚀 DEPLOYMENT COMPLETE
|
|
|
|
Deployment: VAPORA Core Services
|
|
Mode: Enterprise
|
|
Duration: 8 minutes
|
|
Status: ✅ Successful
|
|
|
|
Deployed:
|
|
- vapora-backend (v1.2.1)
|
|
- vapora-agents (v1.2.1)
|
|
- vapora-llm-router (v1.2.1)
|
|
|
|
Verification:
|
|
✓ All pods running
|
|
✓ Health checks passing
|
|
✓ No error logs
|
|
✓ Metrics normal
|
|
|
|
Next steps:
|
|
- Monitor #alerts for any issues
|
|
- Check dashboards every 5 minutes for 30 min
|
|
- Review logs if any issues detected
|
|
|
|
Questions? @on-call-engineer
|
|
```
|
|
|
|
### 6.2 Update Status Page
|
|
|
|
```
|
|
If using public status page:
|
|
|
|
UPDATE: Maintenance Complete
|
|
|
|
VAPORA services have been successfully updated
|
|
and are now operating normally.
|
|
|
|
All systems monitoring nominal.
|
|
```
|
|
|
|
### 6.3 Notify Stakeholders
|
|
|
|
- [ ] Send message to support team: "Deployment complete, all systems normal"
|
|
- [ ] Post in #product: "Backend updated to v1.2.1, new features available"
|
|
- [ ] Update ticket/issue with deployment completion time and status
|
|
|
|
---
|
|
|
|
## Phase 7: Post-Deployment Monitoring (Ongoing)
|
|
|
|
### 7.1 First 5 Minutes: Watch Closely
|
|
|
|
```bash
|
|
# Keep watching for any issues
|
|
watch kubectl get pods -n $NAMESPACE
|
|
watch kubectl top pods -n $NAMESPACE
|
|
watch kubectl logs -f deployment/vapora-backend -n $NAMESPACE
|
|
```
|
|
|
|
**Watch for:**
|
|
- Pod restarts (RESTARTS counter increasing)
|
|
- Increased error logs
|
|
- Resource usage spikes
|
|
- Service unreachability
|
|
|
|
### 7.2 First 30 Minutes: Monitor Dashboard
|
|
|
|
Keep dashboard visible showing:
|
|
- Pod health status
|
|
- CPU/Memory usage per pod
|
|
- Request latency (if available)
|
|
- Error rate
|
|
- Recent logs
|
|
|
|
**Alert triggers for immediate action:**
|
|
- Any pod restarting repeatedly
|
|
- Error rate above 5%
|
|
- Latency above 2x normal
|
|
- Pod stuck in Pending state
|
|
|
|
### 7.3 First 2 Hours: Regular Checks
|
|
|
|
```bash
|
|
# Every 10 minutes:
|
|
1. kubectl get pods -n $NAMESPACE
|
|
2. kubectl top pods -n $NAMESPACE
|
|
3. Check error logs: grep -i error from recent logs
|
|
4. Check alerts dashboard
|
|
```
|
|
|
|
**If issues detected**, proceed to Incident Response Runbook
|
|
|
|
### 7.4 After 2 Hours: Normal Monitoring
|
|
|
|
Return to standard monitoring procedures. Deployment complete.
|
|
|
|
---
|
|
|
|
## If Issues Detected: Quick Rollback
|
|
|
|
If problems occur at any point:
|
|
|
|
```bash
|
|
# IMMEDIATE: Rollback (1 minute)
|
|
for deployment in vapora-backend vapora-agents vapora-llm-router; do
|
|
kubectl rollout undo deployment/$deployment -n $NAMESPACE &
|
|
done
|
|
wait
|
|
|
|
# Verify rollback completing:
|
|
kubectl rollout status deployment/vapora-backend -n $NAMESPACE --timeout=5m
|
|
|
|
# Confirm services recovering:
|
|
curl http://localhost:8001/health
|
|
|
|
# Post to #deployments:
|
|
# 🔙 ROLLBACK EXECUTED
|
|
# Issue detected, services rolled back to previous version
|
|
# All pods should be recovering now
|
|
```
|
|
|
|
See [Rollback Runbook](./rollback-runbook.md) for detailed procedures.
|
|
|
|
---
|
|
|
|
## Common Issues & Solutions
|
|
|
|
### Issue: Pod stuck in ImagePullBackOff
|
|
|
|
**Cause**: Docker image doesn't exist or can't be downloaded
|
|
|
|
**Solution**:
|
|
```bash
|
|
# Check pod events
|
|
kubectl describe pod <pod-name> -n $NAMESPACE
|
|
|
|
# Check image registry access
|
|
kubectl get secret -n $NAMESPACE
|
|
|
|
# Either:
|
|
1. Verify image name is correct in deployment.yaml
|
|
2. Push missing image to registry
|
|
3. Rollback deployment
|
|
```
|
|
|
|
### Issue: Pod stuck in CrashLoopBackOff
|
|
|
|
**Cause**: Application crashing on startup
|
|
|
|
**Solution**:
|
|
```bash
|
|
# Get pod logs
|
|
kubectl logs <pod-name> -n $NAMESPACE --previous
|
|
|
|
# Fix typically requires config change:
|
|
1. Fix ConfigMap issue
|
|
2. Re-apply ConfigMap: kubectl apply -f configmap.yaml
|
|
3. Trigger pod restart: kubectl rollout restart deployment/<name>
|
|
|
|
# Or rollback if unclear
|
|
```
|
|
|
|
### Issue: Pod in Pending state
|
|
|
|
**Cause**: Node doesn't have capacity or resources
|
|
|
|
**Solution**:
|
|
```bash
|
|
# Describe pod to see why
|
|
kubectl describe pod <pod-name> -n $NAMESPACE
|
|
|
|
# Check for "Insufficient cpu", "Insufficient memory"
|
|
kubectl top nodes
|
|
|
|
# Either:
|
|
1. Scale down other workloads
|
|
2. Increase node count
|
|
3. Reduce resource requirements in deployment.yaml and redeploy
|
|
```
|
|
|
|
### Issue: Service endpoints empty
|
|
|
|
**Cause**: Pods not passing health checks
|
|
|
|
**Solution**:
|
|
```bash
|
|
# Check pod logs for errors
|
|
kubectl logs <pod-name> -n $NAMESPACE
|
|
|
|
# Check pod readiness probe failures
|
|
kubectl describe pod <pod-name> -n $NAMESPACE | grep -A 5 "Readiness"
|
|
|
|
# Fix configuration or rollback
|
|
```
|
|
|
|
---
|
|
|
|
## Completion Checklist
|
|
|
|
- [ ] All pods running and ready
|
|
- [ ] Health endpoints responding
|
|
- [ ] No error logs
|
|
- [ ] Metrics normal
|
|
- [ ] Deployment communication posted
|
|
- [ ] Status page updated
|
|
- [ ] Stakeholders notified
|
|
- [ ] Monitoring enabled for next 2 hours
|
|
- [ ] Ticket/issue updated with completion details
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
- Continue monitoring per [Monitoring Runbook](./monitoring-runbook.md)
|
|
- If issues arise, follow [Incident Response Runbook](./incident-response-runbook.md)
|
|
- Document lessons learned
|
|
- Update runbooks if procedures need improvement
|