# Troubleshooting

Common issues, debugging procedures, and resolution strategies for the Provisioning platform.

## Quick Diagnosis

Run platform diagnostics:

```bash
# Comprehensive health check
provisioning diagnose

# Check specific component
provisioning diagnose --component orchestrator

# Generate diagnostic report
provisioning diagnose --report /tmp/diagnostics.txt
```

## Common Issues

### Services Won't Start

**Symptom**: Service fails to start or crashes immediately

**Diagnosis**:

```bash
# Check service status
systemctl status provisioning-orchestrator

# View recent logs
journalctl -u provisioning-orchestrator -n 100 --no-pager

# Check configuration
provisioning validate config
```

**Common Causes**:

1. Port already in use

```bash
# Find process using port
lsof -i :8080

# Kill conflicting process or change port in config
```

2. Configuration error

```bash
# Validate configuration
provisioning validate config --strict

# Check for syntax errors
nickel typecheck /etc/provisioning/config.ncl
```

3. Missing dependencies

```bash
# Check binary dependencies
ldd /usr/local/bin/provisioning-orchestrator

# Install missing libraries
sudo apt install <missing-library>
```

4. Permission issues

```bash
# Fix ownership
sudo chown -R provisioning:provisioning /var/lib/provisioning
sudo chown -R provisioning:provisioning /etc/provisioning

# Fix permissions
sudo chmod 750 /var/lib/provisioning
sudo chmod 640 /etc/provisioning/*.toml
```

### Database Connection Failures

**Symptom**: Services can't connect to SurrealDB

**Diagnosis**:

```bash
# Check database status
systemctl status surrealdb

# Test database connectivity
curl  [http://localhost:8000/health](http://localhost:8000/health)

# Check database logs
journalctl -u surrealdb -n 50
```

**Resolution**:

```bash
# Restart database
sudo systemctl restart surrealdb

# Verify connection string in config
provisioning config get database.url

# Test manual connection
surreal sql --conn  [http://localhost:8000](http://localhost:8000) --user root --pass root
```

### High Resource Usage

**Symptom**: Service consuming excessive CPU or memory

**Diagnosis**:

```bash
# Monitor resource usage
top -p $(pgrep provisioning-orchestrator)

# Detailed metrics
provisioning platform metrics --service orchestrator

# Check for resource leaks
```

**Resolution**:

```bash
# Adjust worker threads
provisioning config set execution.worker_threads 4

# Reduce parallel tasks
provisioning config set execution.max_parallel_tasks 50

# Increase memory limit
sudo systemctl set-property provisioning-orchestrator MemoryMax=8G

# Restart service
sudo systemctl restart provisioning-orchestrator
```

### Workflow Failures

**Symptom**: Workflows fail or hang

**Diagnosis**:

```bash
# List failed workflows
provisioning workflow list --status failed

# View workflow details
provisioning workflow show <workflow-id>

# Check workflow logs
provisioning workflow logs <workflow-id>

# Inspect checkpoint state
provisioning workflow checkpoints <workflow-id>
```

**Common Issues**:

1. Provider API errors

```bash
# Check provider credentials
provisioning provider validate upcloud

# Test provider connectivity
provisioning provider test upcloud
```

2. Dependency resolution failures

```bash
# Validate infrastructure schema
provisioning validate infra my-cluster.ncl

# Check task service dependencies
provisioning taskserv deps kubernetes
```

3. Timeout issues

```bash
# Increase timeout
provisioning config set workflows.task_timeout 600

# Enable detailed logging
provisioning config set logging.level debug
```

### Network Connectivity Issues

**Symptom**: Can't reach external services or cloud providers

**Diagnosis**:

```bash
# Test network connectivity
ping -c 3 upcloud.com

# Check DNS resolution
nslookup api.upcloud.com

# Test HTTPS connectivity
curl -v  [https://api.upcloud.com](https://api.upcloud.com)

# Check proxy settings
env | grep -i proxy
```

**Resolution**:

```bash
# Configure proxy if needed
export HTTPS_PROXY= [http://proxy.example.com:8080](http://proxy.example.com:8080)
provisioning config set network.proxy  [http://proxy.example.com:8080](http://proxy.example.com:8080)

# Verify firewall rules
sudo ufw status

# Check routing
ip route show
```

### Authentication Failures

**Symptom**: API requests fail with 401 Unauthorized

**Diagnosis**:

```bash
# Check JWT token
provisioning auth status

# Verify user credentials
provisioning auth whoami

# Check authentication logs
journalctl -u provisioning-control-center | grep "auth"
```

**Resolution**:

```bash
# Refresh authentication token
provisioning auth login --username admin

# Reset user password
provisioning auth reset-password --username admin

# Verify MFA configuration
provisioning auth mfa status
```

## Debugging Workflows

### Enable Debug Logging

```bash
# Enable debug mode
export PROVISIONING_LOG_LEVEL=debug
provisioning workflow create my-cluster --debug

# Or in configuration
provisioning config set logging.level debug
sudo systemctl restart provisioning-orchestrator
```

### Workflow State Inspection

```bash
# View workflow state
provisioning workflow state <workflow-id>

# Export workflow state to JSON
provisioning workflow state <workflow-id> --format json > workflow-state.json

# Inspect checkpoints
provisioning workflow checkpoints <workflow-id>
```

### Manual Workflow Retry

```bash
# Retry failed workflow from last checkpoint
provisioning workflow retry <workflow-id>

# Retry from specific checkpoint
provisioning workflow retry <workflow-id> --from-checkpoint 3

# Force retry (skip validation)
provisioning workflow retry <workflow-id> --force
```

## Performance Troubleshooting

### Slow Workflow Execution

**Diagnosis**:

```bash
# Profile workflow execution
provisioning workflow profile <workflow-id>

# Identify bottlenecks
provisioning workflow analyze <workflow-id>
```

**Optimization**:

```bash
# Increase parallelism
provisioning config set execution.max_parallel_tasks 200

# Optimize database queries
provisioning database analyze

# Add caching
provisioning config set cache.enabled true
```

### Database Performance Issues

**Diagnosis**:

```bash
# Check database metrics
curl  [http://localhost:8000/metrics](http://localhost:8000/metrics)

# Identify slow queries
provisioning database slow-queries

# Check connection pool
provisioning database pool-status
```

**Optimization**:

```bash
# Increase connection pool
provisioning config set database.max_connections 200

# Add indexes
provisioning database create-indexes

# Optimize vacuum settings
provisioning database vacuum
```

## Log Analysis

### Centralized Log Viewing

```bash
# View all platform logs
journalctl -u provisioning-* -f

# Filter by severity
journalctl -u provisioning-* -p err

# Export logs for analysis
journalctl -u provisioning-* --since "1 hour ago" > /tmp/logs.txt
```

### Structured Log Queries

Using Loki with LogQL:

```logql
# Find errors in orchestrator
{job="provisioning-orchestrator"} | = "ERROR"

# Workflow failures
{job="provisioning-orchestrator"} | json | status="failed"

# API request latency over 1s
{job="provisioning-control-center"} | json | duration > 1
```

### Log Correlation

```bash
# Correlate logs by request ID
journalctl -u provisioning-* | grep "request_id=abc123"

# Trace workflow execution
provisioning workflow trace <workflow-id>
```

## Advanced Debugging

### Enable Rust Backtrace

```bash
# Enable backtrace for Rust services
export RUST_BACKTRACE=1
sudo systemctl restart provisioning-orchestrator

# Full backtrace
export RUST_BACKTRACE=full
```

### Core Dump Analysis

```bash
# Enable core dumps
sudo sysctl -w kernel.core_pattern=/var/crash/core.%e.%p
ulimit -c unlimited

# Analyze core dump
sudo coredumpctl list
sudo coredumpctl debug <pid>

# In gdb:
(gdb) bt
(gdb) info threads
(gdb) thread apply all bt
```

### Network Traffic Analysis

```bash
# Capture API traffic
sudo tcpdump -i any -w /tmp/api-traffic.pcap port 8080

# Analyze with tshark
tshark -r /tmp/api-traffic.pcap -Y "http"
```

## Getting Help

### Collect Diagnostic Information

```bash
# Generate comprehensive diagnostic report
provisioning diagnose --full --output /tmp/diagnostics.tar.gz

# Report includes:
# - Service status
# - Configuration files
# - Recent logs (last 1000 lines per service)
# - Resource usage metrics
# - Database status
# - Network connectivity tests
# - Workflow states
```

### Support Channels

1. Check documentation: `provisioning help <topic>`
2. Search logs: `journalctl -u provisioning-*`
3. Review monitoring dashboards: [http://localhost:3000](http://localhost:3000)
4. Run diagnostics: `provisioning diagnose`
5. Contact support with diagnostic report

## Preventive Measures

- Enable comprehensive monitoring and alerting
- Implement regular health checks
- Maintain up-to-date documentation
- Test disaster recovery procedures monthly
- Keep platform and dependencies updated
- Review logs regularly for warning signs
- Monitor resource utilization trends
- Validate configuration changes before applying

## Related Documentation

- [Service Management](service-management.md) - Service lifecycle
- [Monitoring](monitoring.md) - Observability setup
- [Platform Health](platform-health.md) - Health checks
- [Backup & Recovery](backup-recovery.md) - Recovery procedures
