Rustelo/scripts/databases/DATABASE_SCRIPTS.md

Database Management Scripts

This directory contains a comprehensive set of shell scripts for managing your Rustelo application's database. These scripts provide convenient commands for all database operations including setup, backup, monitoring, migrations, and utilities.

Overview

The database management system consists of several specialized scripts, each handling different aspects of database operations:

• db.sh - Master script that provides easy access to all database tools
• db-setup.sh - Database setup and initialization
• db-backup.sh - Backup and restore operations
• db-monitor.sh - Monitoring and health checks
• db-migrate.sh - Migration management with advanced features
• db-utils.sh - Database utilities and maintenance tasks

Quick Start

Master Script (db.sh)

The master script provides a centralized interface to all database operations:

# Quick status check
./scripts/db.sh status

# Complete health check
./scripts/db.sh health

# Create backup
./scripts/db.sh backup

# Run migrations
./scripts/db.sh migrate

# Optimize database
./scripts/db.sh optimize

Category-based Commands

Use the master script with categories for specific operations:

# Database setup
./scripts/db.sh setup create
./scripts/db.sh setup migrate
./scripts/db.sh setup seed

# Backup operations
./scripts/db.sh backup create
./scripts/db.sh backup restore --file backup.sql
./scripts/db.sh backup list

# Monitoring
./scripts/db.sh monitor health
./scripts/db.sh monitor connections
./scripts/db.sh monitor performance

# Migration management
./scripts/db.sh migrate create --name add_users
./scripts/db.sh migrate run
./scripts/db.sh migrate rollback --steps 1

# Utilities
./scripts/db.sh utils size
./scripts/db.sh utils tables
./scripts/db.sh utils optimize

Individual Scripts

Database Setup (db-setup.sh)

Handles database initialization and basic operations:

# Full setup (create + migrate + seed)
./scripts/db-setup.sh setup

# Individual operations
./scripts/db-setup.sh create
./scripts/db-setup.sh migrate
./scripts/db-setup.sh seed
./scripts/db-setup.sh reset --force

# Database-specific setup
./scripts/db-setup.sh postgres
./scripts/db-setup.sh sqlite

Features:

• Automatic environment detection
• Support for PostgreSQL and SQLite
• Seed data management
• Database reset with safety checks
• Environment variable management

Database Backup (db-backup.sh)

Comprehensive backup and restore functionality:

# Create backups
./scripts/db-backup.sh backup                          # Full backup
./scripts/db-backup.sh backup --compress               # Compressed backup
./scripts/db-backup.sh backup --schema-only            # Schema only
./scripts/db-backup.sh backup --tables users,content   # Specific tables

# Restore operations
./scripts/db-backup.sh restore --file backup.sql
./scripts/db-backup.sh restore --file backup.sql --force

# Backup management
./scripts/db-backup.sh list                            # List backups
./scripts/db-backup.sh clean --keep-days 7             # Clean old backups

Features:

• Multiple backup formats (SQL, custom, tar)
• Compression support
• Selective table backup
• Automatic backup cleanup
• Backup validation
• Database cloning capabilities

Database Monitoring (db-monitor.sh)

Real-time monitoring and health checks:

# Health checks
./scripts/db-monitor.sh health                         # Complete health check
./scripts/db-monitor.sh status                         # Quick status
./scripts/db-monitor.sh connections                    # Active connections
./scripts/db-monitor.sh performance                    # Performance metrics

# Monitoring
./scripts/db-monitor.sh monitor --interval 30          # Continuous monitoring
./scripts/db-monitor.sh slow-queries                   # Slow query analysis
./scripts/db-monitor.sh locks                          # Database locks

# Maintenance
./scripts/db-monitor.sh vacuum                         # Database maintenance
./scripts/db-monitor.sh analyze                        # Update statistics
./scripts/db-monitor.sh report                         # Generate report

Features:

• Real-time connection monitoring
• Performance metrics tracking
• Slow query detection
• Lock analysis
• Disk usage monitoring
• Memory usage tracking
• Automated maintenance tasks
• Comprehensive reporting

Database Migration (db-migrate.sh)

Advanced migration management system:

# Migration status
./scripts/db-migrate.sh status                         # Show migration status
./scripts/db-migrate.sh pending                        # List pending migrations
./scripts/db-migrate.sh applied                        # List applied migrations

# Running migrations
./scripts/db-migrate.sh run                            # Run all pending
./scripts/db-migrate.sh run --version 003              # Run to specific version
./scripts/db-migrate.sh dry-run                        # Preview changes

# Creating migrations
./scripts/db-migrate.sh create --name add_user_preferences
./scripts/db-migrate.sh create --name migrate_users --type data
./scripts/db-migrate.sh create --template create-table

# Rollback operations
./scripts/db-migrate.sh rollback --steps 1             # Rollback last migration
./scripts/db-migrate.sh rollback --steps 3             # Rollback 3 migrations

# Validation
./scripts/db-migrate.sh validate                       # Validate all migrations

Features:

• Migration version control
• Rollback capabilities
• Migration templates
• Dry-run mode
• Migration validation
• Automatic rollback script generation
• Lock-based migration safety
• Comprehensive migration history

Database Utilities (db-utils.sh)

Comprehensive database utilities and maintenance:

# Database information
./scripts/db-utils.sh size                             # Database size info
./scripts/db-utils.sh tables                           # Table information
./scripts/db-utils.sh tables --table users             # Specific table info
./scripts/db-utils.sh indexes                          # Index information
./scripts/db-utils.sh constraints                      # Table constraints

# User and session management
./scripts/db-utils.sh users                            # Database users
./scripts/db-utils.sh sessions                         # Active sessions
./scripts/db-utils.sh queries                          # Running queries
./scripts/db-utils.sh kill-query --query-id 12345      # Kill specific query

# Maintenance operations
./scripts/db-utils.sh optimize                         # Optimize database
./scripts/db-utils.sh reindex                          # Rebuild indexes
./scripts/db-utils.sh check-integrity                  # Integrity check
./scripts/db-utils.sh cleanup                          # Clean temporary data

# Data analysis
./scripts/db-utils.sh duplicate-data --table users     # Find duplicates
./scripts/db-utils.sh table-stats --table users        # Detailed table stats
./scripts/db-utils.sh benchmark                        # Performance benchmarks

Features:

• Comprehensive database analysis
• User and session management
• Query monitoring and termination
• Database optimization
• Integrity checking
• Duplicate data detection
• Performance benchmarking
• Automated cleanup tasks

Configuration

Environment Variables

The scripts use the following environment variables from your .env file:

# Database Configuration
DATABASE_URL=postgresql://user:password@localhost:5432/database_name
# or
DATABASE_URL=sqlite://data/database.db

# Environment
ENVIRONMENT=dev

Script Configuration

Each script has configurable parameters:

# Common options
--env ENV           # Environment (dev/prod)
--force             # Skip confirmations
--quiet             # Suppress verbose output
--debug             # Enable debug output
--dry-run           # Show what would be done

# Backup-specific
--compress          # Compress backup files
--keep-days N       # Retention period for backups

# Monitoring-specific
--interval N        # Monitoring interval in seconds
--threshold-conn N  # Connection alert threshold
--continuous        # Run continuously

# Migration-specific
--version VERSION   # Target migration version
--steps N          # Number of migration steps
--template NAME    # Migration template name

Database Support

PostgreSQL

Full support for PostgreSQL features:

• Connection pooling monitoring
• Query performance analysis
• Index usage statistics
• Lock detection and resolution
• User and permission management
• Extension management
• Advanced backup formats

SQLite

Optimized support for SQLite:

• File-based operations
• Integrity checking
• Vacuum and analyze operations
• Backup and restore
• Schema analysis

Safety Features

Confirmation Prompts

Destructive operations require confirmation:

• Database reset
• Data truncation
• Migration rollback
• Backup restoration

Dry-Run Mode

Preview changes before execution:

./scripts/db-migrate.sh run --dry-run
./scripts/db-backup.sh backup --dry-run
./scripts/db-utils.sh optimize --dry-run

Locking Mechanism

Migration operations use locks to prevent concurrent execution:

• Automatic lock acquisition
• Lock timeout handling
• Process ID tracking
• Graceful lock release

Backup Safety

Automatic backup creation before destructive operations:

• Pre-rollback backups
• Pre-reset backups
• Backup validation
• Checksums for integrity

Error Handling

Robust Error Detection

Scripts include comprehensive error checking:

• Database connectivity verification
• File existence validation
• Permission checking
• SQL syntax validation

Graceful Recovery

Automatic recovery mechanisms:

• Transaction rollback on failure
• Lock release on interruption
• Temporary file cleanup
• Error state recovery

Integration

CI/CD Integration

Scripts are designed for automation:

# In CI/CD pipeline
./scripts/db.sh setup create --force --quiet
./scripts/db.sh migrate run --force
./scripts/db.sh utils check-integrity

Monitoring Integration

Easy integration with monitoring systems:

# Health check endpoint
./scripts/db.sh monitor health --format json

# Metrics collection
./scripts/db.sh monitor performance --format csv

Advanced Usage

Custom Migration Templates

Create custom migration templates in migration_templates/:

-- migration_templates/add-audit-columns.sql
-- Add audit columns to a table
ALTER TABLE ${TABLE_NAME}
ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
ADD COLUMN updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
ADD COLUMN created_by VARCHAR(255),
ADD COLUMN updated_by VARCHAR(255);

Scheduled Operations

Set up automated database maintenance:

# Crontab entry for nightly optimization
0 2 * * * cd /path/to/project && ./scripts/db.sh utils optimize --quiet

# Weekly backup
0 0 * * 0 cd /path/to/project && ./scripts/db.sh backup create --compress --quiet

Performance Tuning

Use monitoring data for optimization:

# Identify slow queries
./scripts/db.sh monitor slow-queries

# Analyze index usage
./scripts/db.sh utils indexes

# Check table statistics
./scripts/db.sh utils table-stats --table high_traffic_table

Troubleshooting

Common Issues

1. Connection Errors

   # Test connection
   ./scripts/db.sh utils connection-test
   
   # Check database status
   ./scripts/db.sh status
   

2. Migration Failures

   # Check migration status
   ./scripts/db.sh migrate status
   
   # Validate migrations
   ./scripts/db.sh migrate validate
   
   # Rollback if needed
   ./scripts/db.sh migrate rollback --steps 1
   

3. Performance Issues

   # Check database health
   ./scripts/db.sh monitor health
   
   # Analyze performance
   ./scripts/db.sh monitor performance
   
   # Optimize database
   ./scripts/db.sh utils optimize
   

Debug Mode

Enable debug output for troubleshooting:

./scripts/db.sh setup migrate --debug
./scripts/db.sh backup create --debug

Log Files

Scripts generate logs in the logs/ directory:

• migration.log - Migration operations
• backup.log - Backup operations
• monitoring.log - Monitoring data

Best Practices

Regular Maintenance

1. Daily: Health checks and monitoring
2. Weekly: Backups and cleanup
3. Monthly: Full optimization and analysis

Development Workflow

1. Create feature branch
2. Generate migration: ./scripts/db.sh migrate create --name feature_name
3. Test migration: ./scripts/db.sh migrate dry-run
4. Run migration: ./scripts/db.sh migrate run
5. Verify changes: ./scripts/db.sh monitor health

Production Deployment

1. Backup before deployment: ./scripts/db.sh backup create
2. Run migrations: ./scripts/db.sh migrate run --env prod
3. Verify deployment: ./scripts/db.sh monitor health --env prod
4. Monitor performance: ./scripts/db.sh monitor performance --env prod

Security Considerations

Environment Variables

• Store sensitive data in .env files
• Use different credentials for each environment
• Regularly rotate database passwords
• Limit database user privileges

Script Permissions

# Set appropriate permissions
chmod 750 scripts/db*.sh
chown app:app scripts/db*.sh

Access Control

• Limit script execution to authorized users
• Use sudo for production operations
• Audit script usage
• Monitor database access

Support

For issues or questions:

1. Check the script help: ./scripts/db.sh --help
2. Review the logs in the logs/ directory
3. Run diagnostics: ./scripts/db.sh monitor health
4. Test connectivity: ./scripts/db.sh utils connection-test

Contributing

To add new database management features:

1. Follow the existing script structure
2. Add comprehensive error handling
3. Include help documentation
4. Add safety checks for destructive operations
5. Test with both PostgreSQL and SQLite
6. Update this documentation