095fd89ff7
Some checks failed
CI/CD Pipeline / Test Suite (push) Has been cancelled
CI/CD Pipeline / Security Audit (push) Has been cancelled
CI/CD Pipeline / Build Docker Image (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Performance Benchmarks (push) Has been cancelled
CI/CD Pipeline / Cleanup (push) Has been cancelled
533 lines
14 KiB
Markdown
533 lines
14 KiB
Markdown
# Database Management Scripts
|
|
|
|
This directory contains a comprehensive set of shell scripts for managing your Rustelo application's database. These scripts provide convenient commands for all database operations including setup, backup, monitoring, migrations, and utilities.
|
|
|
|
## Overview
|
|
|
|
The database management system consists of several specialized scripts, each handling different aspects of database operations:
|
|
|
|
- **`db.sh`** - Master script that provides easy access to all database tools
|
|
- **`db-setup.sh`** - Database setup and initialization
|
|
- **`db-backup.sh`** - Backup and restore operations
|
|
- **`db-monitor.sh`** - Monitoring and health checks
|
|
- **`db-migrate.sh`** - Migration management with advanced features
|
|
- **`db-utils.sh`** - Database utilities and maintenance tasks
|
|
|
|
## Quick Start
|
|
|
|
### Master Script (`db.sh`)
|
|
|
|
The master script provides a centralized interface to all database operations:
|
|
|
|
```bash
|
|
# Quick status check
|
|
./scripts/db.sh status
|
|
|
|
# Complete health check
|
|
./scripts/db.sh health
|
|
|
|
# Create backup
|
|
./scripts/db.sh backup
|
|
|
|
# Run migrations
|
|
./scripts/db.sh migrate
|
|
|
|
# Optimize database
|
|
./scripts/db.sh optimize
|
|
```
|
|
|
|
### Category-based Commands
|
|
|
|
Use the master script with categories for specific operations:
|
|
|
|
```bash
|
|
# Database setup
|
|
./scripts/db.sh setup create
|
|
./scripts/db.sh setup migrate
|
|
./scripts/db.sh setup seed
|
|
|
|
# Backup operations
|
|
./scripts/db.sh backup create
|
|
./scripts/db.sh backup restore --file backup.sql
|
|
./scripts/db.sh backup list
|
|
|
|
# Monitoring
|
|
./scripts/db.sh monitor health
|
|
./scripts/db.sh monitor connections
|
|
./scripts/db.sh monitor performance
|
|
|
|
# Migration management
|
|
./scripts/db.sh migrate create --name add_users
|
|
./scripts/db.sh migrate run
|
|
./scripts/db.sh migrate rollback --steps 1
|
|
|
|
# Utilities
|
|
./scripts/db.sh utils size
|
|
./scripts/db.sh utils tables
|
|
./scripts/db.sh utils optimize
|
|
```
|
|
|
|
## Individual Scripts
|
|
|
|
### Database Setup (`db-setup.sh`)
|
|
|
|
Handles database initialization and basic operations:
|
|
|
|
```bash
|
|
# Full setup (create + migrate + seed)
|
|
./scripts/db-setup.sh setup
|
|
|
|
# Individual operations
|
|
./scripts/db-setup.sh create
|
|
./scripts/db-setup.sh migrate
|
|
./scripts/db-setup.sh seed
|
|
./scripts/db-setup.sh reset --force
|
|
|
|
# Database-specific setup
|
|
./scripts/db-setup.sh postgres
|
|
./scripts/db-setup.sh sqlite
|
|
```
|
|
|
|
**Features:**
|
|
- Automatic environment detection
|
|
- Support for PostgreSQL and SQLite
|
|
- Seed data management
|
|
- Database reset with safety checks
|
|
- Environment variable management
|
|
|
|
### Database Backup (`db-backup.sh`)
|
|
|
|
Comprehensive backup and restore functionality:
|
|
|
|
```bash
|
|
# Create backups
|
|
./scripts/db-backup.sh backup # Full backup
|
|
./scripts/db-backup.sh backup --compress # Compressed backup
|
|
./scripts/db-backup.sh backup --schema-only # Schema only
|
|
./scripts/db-backup.sh backup --tables users,content # Specific tables
|
|
|
|
# Restore operations
|
|
./scripts/db-backup.sh restore --file backup.sql
|
|
./scripts/db-backup.sh restore --file backup.sql --force
|
|
|
|
# Backup management
|
|
./scripts/db-backup.sh list # List backups
|
|
./scripts/db-backup.sh clean --keep-days 7 # Clean old backups
|
|
```
|
|
|
|
**Features:**
|
|
- Multiple backup formats (SQL, custom, tar)
|
|
- Compression support
|
|
- Selective table backup
|
|
- Automatic backup cleanup
|
|
- Backup validation
|
|
- Database cloning capabilities
|
|
|
|
### Database Monitoring (`db-monitor.sh`)
|
|
|
|
Real-time monitoring and health checks:
|
|
|
|
```bash
|
|
# Health checks
|
|
./scripts/db-monitor.sh health # Complete health check
|
|
./scripts/db-monitor.sh status # Quick status
|
|
./scripts/db-monitor.sh connections # Active connections
|
|
./scripts/db-monitor.sh performance # Performance metrics
|
|
|
|
# Monitoring
|
|
./scripts/db-monitor.sh monitor --interval 30 # Continuous monitoring
|
|
./scripts/db-monitor.sh slow-queries # Slow query analysis
|
|
./scripts/db-monitor.sh locks # Database locks
|
|
|
|
# Maintenance
|
|
./scripts/db-monitor.sh vacuum # Database maintenance
|
|
./scripts/db-monitor.sh analyze # Update statistics
|
|
./scripts/db-monitor.sh report # Generate report
|
|
```
|
|
|
|
**Features:**
|
|
- Real-time connection monitoring
|
|
- Performance metrics tracking
|
|
- Slow query detection
|
|
- Lock analysis
|
|
- Disk usage monitoring
|
|
- Memory usage tracking
|
|
- Automated maintenance tasks
|
|
- Comprehensive reporting
|
|
|
|
### Database Migration (`db-migrate.sh`)
|
|
|
|
Advanced migration management system:
|
|
|
|
```bash
|
|
# Migration status
|
|
./scripts/db-migrate.sh status # Show migration status
|
|
./scripts/db-migrate.sh pending # List pending migrations
|
|
./scripts/db-migrate.sh applied # List applied migrations
|
|
|
|
# Running migrations
|
|
./scripts/db-migrate.sh run # Run all pending
|
|
./scripts/db-migrate.sh run --version 003 # Run to specific version
|
|
./scripts/db-migrate.sh dry-run # Preview changes
|
|
|
|
# Creating migrations
|
|
./scripts/db-migrate.sh create --name add_user_preferences
|
|
./scripts/db-migrate.sh create --name migrate_users --type data
|
|
./scripts/db-migrate.sh create --template create-table
|
|
|
|
# Rollback operations
|
|
./scripts/db-migrate.sh rollback --steps 1 # Rollback last migration
|
|
./scripts/db-migrate.sh rollback --steps 3 # Rollback 3 migrations
|
|
|
|
# Validation
|
|
./scripts/db-migrate.sh validate # Validate all migrations
|
|
```
|
|
|
|
**Features:**
|
|
- Migration version control
|
|
- Rollback capabilities
|
|
- Migration templates
|
|
- Dry-run mode
|
|
- Migration validation
|
|
- Automatic rollback script generation
|
|
- Lock-based migration safety
|
|
- Comprehensive migration history
|
|
|
|
### Database Utilities (`db-utils.sh`)
|
|
|
|
Comprehensive database utilities and maintenance:
|
|
|
|
```bash
|
|
# Database information
|
|
./scripts/db-utils.sh size # Database size info
|
|
./scripts/db-utils.sh tables # Table information
|
|
./scripts/db-utils.sh tables --table users # Specific table info
|
|
./scripts/db-utils.sh indexes # Index information
|
|
./scripts/db-utils.sh constraints # Table constraints
|
|
|
|
# User and session management
|
|
./scripts/db-utils.sh users # Database users
|
|
./scripts/db-utils.sh sessions # Active sessions
|
|
./scripts/db-utils.sh queries # Running queries
|
|
./scripts/db-utils.sh kill-query --query-id 12345 # Kill specific query
|
|
|
|
# Maintenance operations
|
|
./scripts/db-utils.sh optimize # Optimize database
|
|
./scripts/db-utils.sh reindex # Rebuild indexes
|
|
./scripts/db-utils.sh check-integrity # Integrity check
|
|
./scripts/db-utils.sh cleanup # Clean temporary data
|
|
|
|
# Data analysis
|
|
./scripts/db-utils.sh duplicate-data --table users # Find duplicates
|
|
./scripts/db-utils.sh table-stats --table users # Detailed table stats
|
|
./scripts/db-utils.sh benchmark # Performance benchmarks
|
|
```
|
|
|
|
**Features:**
|
|
- Comprehensive database analysis
|
|
- User and session management
|
|
- Query monitoring and termination
|
|
- Database optimization
|
|
- Integrity checking
|
|
- Duplicate data detection
|
|
- Performance benchmarking
|
|
- Automated cleanup tasks
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
The scripts use the following environment variables from your `.env` file:
|
|
|
|
```env
|
|
# Database Configuration
|
|
DATABASE_URL=postgresql://user:password@localhost:5432/database_name
|
|
# or
|
|
DATABASE_URL=sqlite://data/database.db
|
|
|
|
# Environment
|
|
ENVIRONMENT=dev
|
|
```
|
|
|
|
### Script Configuration
|
|
|
|
Each script has configurable parameters:
|
|
|
|
```bash
|
|
# Common options
|
|
--env ENV # Environment (dev/prod)
|
|
--force # Skip confirmations
|
|
--quiet # Suppress verbose output
|
|
--debug # Enable debug output
|
|
--dry-run # Show what would be done
|
|
|
|
# Backup-specific
|
|
--compress # Compress backup files
|
|
--keep-days N # Retention period for backups
|
|
|
|
# Monitoring-specific
|
|
--interval N # Monitoring interval in seconds
|
|
--threshold-conn N # Connection alert threshold
|
|
--continuous # Run continuously
|
|
|
|
# Migration-specific
|
|
--version VERSION # Target migration version
|
|
--steps N # Number of migration steps
|
|
--template NAME # Migration template name
|
|
```
|
|
|
|
## Database Support
|
|
|
|
### PostgreSQL
|
|
|
|
Full support for PostgreSQL features:
|
|
- Connection pooling monitoring
|
|
- Query performance analysis
|
|
- Index usage statistics
|
|
- Lock detection and resolution
|
|
- User and permission management
|
|
- Extension management
|
|
- Advanced backup formats
|
|
|
|
### SQLite
|
|
|
|
Optimized support for SQLite:
|
|
- File-based operations
|
|
- Integrity checking
|
|
- Vacuum and analyze operations
|
|
- Backup and restore
|
|
- Schema analysis
|
|
|
|
## Safety Features
|
|
|
|
### Confirmation Prompts
|
|
|
|
Destructive operations require confirmation:
|
|
- Database reset
|
|
- Data truncation
|
|
- Migration rollback
|
|
- Backup restoration
|
|
|
|
### Dry-Run Mode
|
|
|
|
Preview changes before execution:
|
|
```bash
|
|
./scripts/db-migrate.sh run --dry-run
|
|
./scripts/db-backup.sh backup --dry-run
|
|
./scripts/db-utils.sh optimize --dry-run
|
|
```
|
|
|
|
### Locking Mechanism
|
|
|
|
Migration operations use locks to prevent concurrent execution:
|
|
- Automatic lock acquisition
|
|
- Lock timeout handling
|
|
- Process ID tracking
|
|
- Graceful lock release
|
|
|
|
### Backup Safety
|
|
|
|
Automatic backup creation before destructive operations:
|
|
- Pre-rollback backups
|
|
- Pre-reset backups
|
|
- Backup validation
|
|
- Checksums for integrity
|
|
|
|
## Error Handling
|
|
|
|
### Robust Error Detection
|
|
|
|
Scripts include comprehensive error checking:
|
|
- Database connectivity verification
|
|
- File existence validation
|
|
- Permission checking
|
|
- SQL syntax validation
|
|
|
|
### Graceful Recovery
|
|
|
|
Automatic recovery mechanisms:
|
|
- Transaction rollback on failure
|
|
- Lock release on interruption
|
|
- Temporary file cleanup
|
|
- Error state recovery
|
|
|
|
## Integration
|
|
|
|
### CI/CD Integration
|
|
|
|
Scripts are designed for automation:
|
|
```bash
|
|
# In CI/CD pipeline
|
|
./scripts/db.sh setup create --force --quiet
|
|
./scripts/db.sh migrate run --force
|
|
./scripts/db.sh utils check-integrity
|
|
```
|
|
|
|
### Monitoring Integration
|
|
|
|
Easy integration with monitoring systems:
|
|
```bash
|
|
# Health check endpoint
|
|
./scripts/db.sh monitor health --format json
|
|
|
|
# Metrics collection
|
|
./scripts/db.sh monitor performance --format csv
|
|
```
|
|
|
|
## Advanced Usage
|
|
|
|
### Custom Migration Templates
|
|
|
|
Create custom migration templates in `migration_templates/`:
|
|
|
|
```sql
|
|
-- migration_templates/add-audit-columns.sql
|
|
-- Add audit columns to a table
|
|
ALTER TABLE ${TABLE_NAME}
|
|
ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
ADD COLUMN updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
ADD COLUMN created_by VARCHAR(255),
|
|
ADD COLUMN updated_by VARCHAR(255);
|
|
```
|
|
|
|
### Scheduled Operations
|
|
|
|
Set up automated database maintenance:
|
|
```bash
|
|
# Crontab entry for nightly optimization
|
|
0 2 * * * cd /path/to/project && ./scripts/db.sh utils optimize --quiet
|
|
|
|
# Weekly backup
|
|
0 0 * * 0 cd /path/to/project && ./scripts/db.sh backup create --compress --quiet
|
|
```
|
|
|
|
### Performance Tuning
|
|
|
|
Use monitoring data for optimization:
|
|
```bash
|
|
# Identify slow queries
|
|
./scripts/db.sh monitor slow-queries
|
|
|
|
# Analyze index usage
|
|
./scripts/db.sh utils indexes
|
|
|
|
# Check table statistics
|
|
./scripts/db.sh utils table-stats --table high_traffic_table
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Connection Errors**
|
|
```bash
|
|
# Test connection
|
|
./scripts/db.sh utils connection-test
|
|
|
|
# Check database status
|
|
./scripts/db.sh status
|
|
```
|
|
|
|
2. **Migration Failures**
|
|
```bash
|
|
# Check migration status
|
|
./scripts/db.sh migrate status
|
|
|
|
# Validate migrations
|
|
./scripts/db.sh migrate validate
|
|
|
|
# Rollback if needed
|
|
./scripts/db.sh migrate rollback --steps 1
|
|
```
|
|
|
|
3. **Performance Issues**
|
|
```bash
|
|
# Check database health
|
|
./scripts/db.sh monitor health
|
|
|
|
# Analyze performance
|
|
./scripts/db.sh monitor performance
|
|
|
|
# Optimize database
|
|
./scripts/db.sh utils optimize
|
|
```
|
|
|
|
### Debug Mode
|
|
|
|
Enable debug output for troubleshooting:
|
|
```bash
|
|
./scripts/db.sh setup migrate --debug
|
|
./scripts/db.sh backup create --debug
|
|
```
|
|
|
|
### Log Files
|
|
|
|
Scripts generate logs in the `logs/` directory:
|
|
- `migration.log` - Migration operations
|
|
- `backup.log` - Backup operations
|
|
- `monitoring.log` - Monitoring data
|
|
|
|
## Best Practices
|
|
|
|
### Regular Maintenance
|
|
|
|
1. **Daily**: Health checks and monitoring
|
|
2. **Weekly**: Backups and cleanup
|
|
3. **Monthly**: Full optimization and analysis
|
|
|
|
### Development Workflow
|
|
|
|
1. Create feature branch
|
|
2. Generate migration: `./scripts/db.sh migrate create --name feature_name`
|
|
3. Test migration: `./scripts/db.sh migrate dry-run`
|
|
4. Run migration: `./scripts/db.sh migrate run`
|
|
5. Verify changes: `./scripts/db.sh monitor health`
|
|
|
|
### Production Deployment
|
|
|
|
1. Backup before deployment: `./scripts/db.sh backup create`
|
|
2. Run migrations: `./scripts/db.sh migrate run --env prod`
|
|
3. Verify deployment: `./scripts/db.sh monitor health --env prod`
|
|
4. Monitor performance: `./scripts/db.sh monitor performance --env prod`
|
|
|
|
## Security Considerations
|
|
|
|
### Environment Variables
|
|
|
|
- Store sensitive data in `.env` files
|
|
- Use different credentials for each environment
|
|
- Regularly rotate database passwords
|
|
- Limit database user privileges
|
|
|
|
### Script Permissions
|
|
|
|
```bash
|
|
# Set appropriate permissions
|
|
chmod 750 scripts/db*.sh
|
|
chown app:app scripts/db*.sh
|
|
```
|
|
|
|
### Access Control
|
|
|
|
- Limit script execution to authorized users
|
|
- Use sudo for production operations
|
|
- Audit script usage
|
|
- Monitor database access
|
|
|
|
## Support
|
|
|
|
For issues or questions:
|
|
1. Check the script help: `./scripts/db.sh --help`
|
|
2. Review the logs in the `logs/` directory
|
|
3. Run diagnostics: `./scripts/db.sh monitor health`
|
|
4. Test connectivity: `./scripts/db.sh utils connection-test`
|
|
|
|
## Contributing
|
|
|
|
To add new database management features:
|
|
1. Follow the existing script structure
|
|
2. Add comprehensive error handling
|
|
3. Include help documentation
|
|
4. Add safety checks for destructive operations
|
|
5. Test with both PostgreSQL and SQLite
|
|
6. Update this documentation |