Rustelo/scripts/databases/DATABASE_SCRIPTS.md

# Database Management Scripts

This directory contains a comprehensive set of shell scripts for managing your Rustelo application's database. These scripts provide convenient commands for all database operations including setup, backup, monitoring, migrations, and utilities.

## Overview

The database management system consists of several specialized scripts, each handling different aspects of database operations:

- **`db.sh`** - Master script that provides easy access to all database tools
- **`db-setup.sh`** - Database setup and initialization 
- **`db-backup.sh`** - Backup and restore operations
- **`db-monitor.sh`** - Monitoring and health checks
- **`db-migrate.sh`** - Migration management with advanced features
- **`db-utils.sh`** - Database utilities and maintenance tasks

## Quick Start

### Master Script (`db.sh`)

The master script provides a centralized interface to all database operations:

```bash
# Quick status check
./scripts/db.sh status

# Complete health check
./scripts/db.sh health

# Create backup
./scripts/db.sh backup

# Run migrations
./scripts/db.sh migrate

# Optimize database
./scripts/db.sh optimize
```

### Category-based Commands

Use the master script with categories for specific operations:

```bash
# Database setup
./scripts/db.sh setup create
./scripts/db.sh setup migrate
./scripts/db.sh setup seed

# Backup operations
./scripts/db.sh backup create
./scripts/db.sh backup restore --file backup.sql
./scripts/db.sh backup list

# Monitoring
./scripts/db.sh monitor health
./scripts/db.sh monitor connections
./scripts/db.sh monitor performance

# Migration management
./scripts/db.sh migrate create --name add_users
./scripts/db.sh migrate run
./scripts/db.sh migrate rollback --steps 1

# Utilities
./scripts/db.sh utils size
./scripts/db.sh utils tables
./scripts/db.sh utils optimize
```

## Individual Scripts

### Database Setup (`db-setup.sh`)

Handles database initialization and basic operations:

```bash
# Full setup (create + migrate + seed)
./scripts/db-setup.sh setup

# Individual operations
./scripts/db-setup.sh create
./scripts/db-setup.sh migrate
./scripts/db-setup.sh seed
./scripts/db-setup.sh reset --force

# Database-specific setup
./scripts/db-setup.sh postgres
./scripts/db-setup.sh sqlite
```

**Features:**
- Automatic environment detection
- Support for PostgreSQL and SQLite
- Seed data management
- Database reset with safety checks
- Environment variable management

### Database Backup (`db-backup.sh`)

Comprehensive backup and restore functionality:

```bash
# Create backups
./scripts/db-backup.sh backup                          # Full backup
./scripts/db-backup.sh backup --compress               # Compressed backup
./scripts/db-backup.sh backup --schema-only            # Schema only
./scripts/db-backup.sh backup --tables users,content   # Specific tables

# Restore operations
./scripts/db-backup.sh restore --file backup.sql
./scripts/db-backup.sh restore --file backup.sql --force

# Backup management
./scripts/db-backup.sh list                            # List backups
./scripts/db-backup.sh clean --keep-days 7             # Clean old backups
```

**Features:**
- Multiple backup formats (SQL, custom, tar)
- Compression support
- Selective table backup
- Automatic backup cleanup
- Backup validation
- Database cloning capabilities

### Database Monitoring (`db-monitor.sh`)

Real-time monitoring and health checks:

```bash
# Health checks
./scripts/db-monitor.sh health                         # Complete health check
./scripts/db-monitor.sh status                         # Quick status
./scripts/db-monitor.sh connections                    # Active connections
./scripts/db-monitor.sh performance                    # Performance metrics

# Monitoring
./scripts/db-monitor.sh monitor --interval 30          # Continuous monitoring
./scripts/db-monitor.sh slow-queries                   # Slow query analysis
./scripts/db-monitor.sh locks                          # Database locks

# Maintenance
./scripts/db-monitor.sh vacuum                         # Database maintenance
./scripts/db-monitor.sh analyze                        # Update statistics
./scripts/db-monitor.sh report                         # Generate report
```

**Features:**
- Real-time connection monitoring
- Performance metrics tracking
- Slow query detection
- Lock analysis
- Disk usage monitoring
- Memory usage tracking
- Automated maintenance tasks
- Comprehensive reporting

### Database Migration (`db-migrate.sh`)

Advanced migration management system:

```bash
# Migration status
./scripts/db-migrate.sh status                         # Show migration status
./scripts/db-migrate.sh pending                        # List pending migrations
./scripts/db-migrate.sh applied                        # List applied migrations

# Running migrations
./scripts/db-migrate.sh run                            # Run all pending
./scripts/db-migrate.sh run --version 003              # Run to specific version
./scripts/db-migrate.sh dry-run                        # Preview changes

# Creating migrations
./scripts/db-migrate.sh create --name add_user_preferences
./scripts/db-migrate.sh create --name migrate_users --type data
./scripts/db-migrate.sh create --template create-table

# Rollback operations
./scripts/db-migrate.sh rollback --steps 1             # Rollback last migration
./scripts/db-migrate.sh rollback --steps 3             # Rollback 3 migrations

# Validation
./scripts/db-migrate.sh validate                       # Validate all migrations
```

**Features:**
- Migration version control
- Rollback capabilities
- Migration templates
- Dry-run mode
- Migration validation
- Automatic rollback script generation
- Lock-based migration safety
- Comprehensive migration history

### Database Utilities (`db-utils.sh`)

Comprehensive database utilities and maintenance:

```bash
# Database information
./scripts/db-utils.sh size                             # Database size info
./scripts/db-utils.sh tables                           # Table information
./scripts/db-utils.sh tables --table users             # Specific table info
./scripts/db-utils.sh indexes                          # Index information
./scripts/db-utils.sh constraints                      # Table constraints

# User and session management
./scripts/db-utils.sh users                            # Database users
./scripts/db-utils.sh sessions                         # Active sessions
./scripts/db-utils.sh queries                          # Running queries
./scripts/db-utils.sh kill-query --query-id 12345      # Kill specific query

# Maintenance operations
./scripts/db-utils.sh optimize                         # Optimize database
./scripts/db-utils.sh reindex                          # Rebuild indexes
./scripts/db-utils.sh check-integrity                  # Integrity check
./scripts/db-utils.sh cleanup                          # Clean temporary data

# Data analysis
./scripts/db-utils.sh duplicate-data --table users     # Find duplicates
./scripts/db-utils.sh table-stats --table users        # Detailed table stats
./scripts/db-utils.sh benchmark                        # Performance benchmarks
```

**Features:**
- Comprehensive database analysis
- User and session management
- Query monitoring and termination
- Database optimization
- Integrity checking
- Duplicate data detection
- Performance benchmarking
- Automated cleanup tasks

## Configuration

### Environment Variables

The scripts use the following environment variables from your `.env` file:

```env
# Database Configuration
DATABASE_URL=postgresql://user:password@localhost:5432/database_name
# or
DATABASE_URL=sqlite://data/database.db

# Environment
ENVIRONMENT=dev
```

### Script Configuration

Each script has configurable parameters:

```bash
# Common options
--env ENV           # Environment (dev/prod)
--force             # Skip confirmations
--quiet             # Suppress verbose output
--debug             # Enable debug output
--dry-run           # Show what would be done

# Backup-specific
--compress          # Compress backup files
--keep-days N       # Retention period for backups

# Monitoring-specific
--interval N        # Monitoring interval in seconds
--threshold-conn N  # Connection alert threshold
--continuous        # Run continuously

# Migration-specific
--version VERSION   # Target migration version
--steps N          # Number of migration steps
--template NAME    # Migration template name
```

## Database Support

### PostgreSQL

Full support for PostgreSQL features:
- Connection pooling monitoring
- Query performance analysis
- Index usage statistics
- Lock detection and resolution
- User and permission management
- Extension management
- Advanced backup formats

### SQLite

Optimized support for SQLite:
- File-based operations
- Integrity checking
- Vacuum and analyze operations
- Backup and restore
- Schema analysis

## Safety Features

### Confirmation Prompts

Destructive operations require confirmation:
- Database reset
- Data truncation
- Migration rollback
- Backup restoration

### Dry-Run Mode

Preview changes before execution:
```bash
./scripts/db-migrate.sh run --dry-run
./scripts/db-backup.sh backup --dry-run
./scripts/db-utils.sh optimize --dry-run
```

### Locking Mechanism

Migration operations use locks to prevent concurrent execution:
- Automatic lock acquisition
- Lock timeout handling
- Process ID tracking
- Graceful lock release

### Backup Safety

Automatic backup creation before destructive operations:
- Pre-rollback backups
- Pre-reset backups
- Backup validation
- Checksums for integrity

## Error Handling

### Robust Error Detection

Scripts include comprehensive error checking:
- Database connectivity verification
- File existence validation
- Permission checking
- SQL syntax validation

### Graceful Recovery

Automatic recovery mechanisms:
- Transaction rollback on failure
- Lock release on interruption
- Temporary file cleanup
- Error state recovery

## Integration

### CI/CD Integration

Scripts are designed for automation:
```bash
# In CI/CD pipeline
./scripts/db.sh setup create --force --quiet
./scripts/db.sh migrate run --force
./scripts/db.sh utils check-integrity
```

### Monitoring Integration

Easy integration with monitoring systems:
```bash
# Health check endpoint
./scripts/db.sh monitor health --format json

# Metrics collection
./scripts/db.sh monitor performance --format csv
```

## Advanced Usage

### Custom Migration Templates

Create custom migration templates in `migration_templates/`:

```sql
-- migration_templates/add-audit-columns.sql
-- Add audit columns to a table
ALTER TABLE ${TABLE_NAME}
ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
ADD COLUMN updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
ADD COLUMN created_by VARCHAR(255),
ADD COLUMN updated_by VARCHAR(255);
```

### Scheduled Operations

Set up automated database maintenance:
```bash
# Crontab entry for nightly optimization
0 2 * * * cd /path/to/project && ./scripts/db.sh utils optimize --quiet

# Weekly backup
0 0 * * 0 cd /path/to/project && ./scripts/db.sh backup create --compress --quiet
```

### Performance Tuning

Use monitoring data for optimization:
```bash
# Identify slow queries
./scripts/db.sh monitor slow-queries

# Analyze index usage
./scripts/db.sh utils indexes

# Check table statistics
./scripts/db.sh utils table-stats --table high_traffic_table
```

## Troubleshooting

### Common Issues

1. **Connection Errors**
   ```bash
   # Test connection
   ./scripts/db.sh utils connection-test
   
   # Check database status
   ./scripts/db.sh status
   ```

2. **Migration Failures**
   ```bash
   # Check migration status
   ./scripts/db.sh migrate status
   
   # Validate migrations
   ./scripts/db.sh migrate validate
   
   # Rollback if needed
   ./scripts/db.sh migrate rollback --steps 1
   ```

3. **Performance Issues**
   ```bash
   # Check database health
   ./scripts/db.sh monitor health
   
   # Analyze performance
   ./scripts/db.sh monitor performance
   
   # Optimize database
   ./scripts/db.sh utils optimize
   ```

### Debug Mode

Enable debug output for troubleshooting:
```bash
./scripts/db.sh setup migrate --debug
./scripts/db.sh backup create --debug
```

### Log Files

Scripts generate logs in the `logs/` directory:
- `migration.log` - Migration operations
- `backup.log` - Backup operations
- `monitoring.log` - Monitoring data

## Best Practices

### Regular Maintenance

1. **Daily**: Health checks and monitoring
2. **Weekly**: Backups and cleanup
3. **Monthly**: Full optimization and analysis

### Development Workflow

1. Create feature branch
2. Generate migration: `./scripts/db.sh migrate create --name feature_name`
3. Test migration: `./scripts/db.sh migrate dry-run`
4. Run migration: `./scripts/db.sh migrate run`
5. Verify changes: `./scripts/db.sh monitor health`

### Production Deployment

1. Backup before deployment: `./scripts/db.sh backup create`
2. Run migrations: `./scripts/db.sh migrate run --env prod`
3. Verify deployment: `./scripts/db.sh monitor health --env prod`
4. Monitor performance: `./scripts/db.sh monitor performance --env prod`

## Security Considerations

### Environment Variables

- Store sensitive data in `.env` files
- Use different credentials for each environment
- Regularly rotate database passwords
- Limit database user privileges

### Script Permissions

```bash
# Set appropriate permissions
chmod 750 scripts/db*.sh
chown app:app scripts/db*.sh
```

### Access Control

- Limit script execution to authorized users
- Use sudo for production operations
- Audit script usage
- Monitor database access

## Support

For issues or questions:
1. Check the script help: `./scripts/db.sh --help`
2. Review the logs in the `logs/` directory
3. Run diagnostics: `./scripts/db.sh monitor health`
4. Test connectivity: `./scripts/db.sh utils connection-test`

## Contributing

To add new database management features:
1. Follow the existing script structure
2. Add comprehensive error handling
3. Include help documentation
4. Add safety checks for destructive operations
5. Test with both PostgreSQL and SQLite
6. Update this documentation