provisioning/docs/src/development/dev-workspace-management.md

# Workspace Management Guide

This document provides comprehensive guidance on setting up and using development workspaces, including the path resolution system, testing
infrastructure, and workspace tools usage.

## Table of Contents

1. [Overview](#overview)
2. [Workspace Architecture](#workspace-architecture)
3. [Setup and Initialization](#setup-and-initialization)
4. [Path Resolution System](#path-resolution-system)
5. [Configuration Management](#configuration-management)
6. [Extension Development](#extension-development)
7. [Runtime Management](#runtime-management)
8. [Health Monitoring](#health-monitoring)
9. [Backup and Restore](#backup-and-restore)
10. [Troubleshooting](#troubleshooting)

## Overview

The workspace system provides isolated development environments for the provisioning project, enabling:

- **User Isolation**: Each developer has their own workspace with isolated runtime data
- **Configuration Cascading**: Hierarchical configuration from workspace to core system
- **Extension Development**: Template-based extension development with testing
- **Path Resolution**: Smart path resolution with workspace-aware fallbacks
- **Health Monitoring**: Comprehensive health checks with automatic repairs
- **Backup/Restore**: Complete workspace backup and restore capabilities

**Location**: `/workspace/`
**Main Tool**: `workspace/tools/workspace.nu`

## Workspace Architecture

### Directory Structure

```plaintext
workspace/
├── config/                          # Development configuration
│   ├── dev-defaults.toml            # Development environment defaults
│   ├── test-defaults.toml           # Testing environment configuration
│   ├── local-overrides.toml.example # User customization template
│   └── {user}.toml                  # User-specific configurations
├── extensions/                      # Extension development
│   ├── providers/                   # Custom provider extensions
│   │   ├── template/                # Provider development template
│   │   └── {user}/                  # User-specific providers
│   ├── taskservs/                   # Custom task service extensions
│   │   ├── template/                # Task service template
│   │   └── {user}/                  # User-specific task services
│   └── clusters/                    # Custom cluster extensions
│       ├── template/                # Cluster template
│       └── {user}/                  # User-specific clusters
├── infra/                          # Development infrastructure
│   ├── examples/                   # Example infrastructures
│   │   ├── minimal/                # Minimal learning setup
│   │   ├── development/            # Full development environment
│   │   └── testing/                # Testing infrastructure
│   ├── local/                      # Local development setups
│   └── {user}/                     # User-specific infrastructures
├── lib/                            # Workspace libraries
│   └── path-resolver.nu            # Path resolution system
├── runtime/                        # Runtime data (per-user isolation)
│   ├── workspaces/{user}/          # User workspace data
│   ├── cache/{user}/               # User-specific cache
│   ├── state/{user}/               # User state management
│   ├── logs/{user}/                # User application logs
│   └── data/{user}/                # User database files
└── tools/                          # Workspace management tools
    ├── workspace.nu                # Main workspace interface
    ├── init-workspace.nu           # Workspace initialization
    ├── workspace-health.nu         # Health monitoring
    ├── backup-workspace.nu         # Backup management
    ├── restore-workspace.nu        # Restore functionality
    ├── reset-workspace.nu          # Workspace reset
    └── runtime-manager.nu          # Runtime data management
```

### Component Integration

**Workspace → Core Integration**:

- Workspace paths take priority over core paths
- Extensions discovered automatically from workspace
- Configuration cascades from workspace to core defaults
- Runtime data completely isolated per user

**Development Workflow**:

1. **Initialize** personal workspace
2. **Configure** development environment
3. **Develop** extensions and infrastructure
4. **Test** locally with isolated environment
5. **Deploy** to shared infrastructure

## Setup and Initialization

### Quick Start

```bash
# Navigate to workspace
cd workspace/tools

# Initialize workspace with defaults
nu workspace.nu init

# Initialize with specific options
nu workspace.nu init --user-name developer --infra-name my-dev-infra
```

### Complete Initialization

```bash
# Full initialization with all options
nu workspace.nu init \
    --user-name developer \
    --infra-name development-env \
    --workspace-type development \
    --template full \
    --overwrite \
    --create-examples
```

**Initialization Parameters**:

- `--user-name`: User identifier (defaults to `$env.USER`)
- `--infra-name`: Infrastructure name for this workspace
- `--workspace-type`: Type (`development`, `testing`, `production`)
- `--template`: Template to use (`minimal`, `full`, `custom`)
- `--overwrite`: Overwrite existing workspace
- `--create-examples`: Create example configurations and infrastructure

### Post-Initialization Setup

**Verify Installation**:

```bash
# Check workspace health
nu workspace.nu health --detailed

# Show workspace status
nu workspace.nu status --detailed

# List workspace contents
nu workspace.nu list
```

**Configure Development Environment**:

```bash
# Create user-specific configuration
cp workspace/config/local-overrides.toml.example workspace/config/$USER.toml

# Edit configuration
$EDITOR workspace/config/$USER.toml
```

## Path Resolution System

The workspace implements a sophisticated path resolution system that prioritizes workspace paths while providing fallbacks to core system paths.

### Resolution Hierarchy

**Resolution Order**:

1. **Workspace User Paths**: `workspace/{type}/{user}/{name}`
2. **Workspace Shared Paths**: `workspace/{type}/{name}`
3. **Workspace Templates**: `workspace/{type}/template/{name}`
4. **Core System Paths**: `core/{type}/{name}` (fallback)

### Using Path Resolution

```nushell
# Import path resolver
use workspace/lib/path-resolver.nu

# Resolve configuration with workspace awareness
let config_path = (path-resolver resolve_path "config" "user" --workspace-user "developer")

# Resolve with automatic fallback to core
let extension_path = (path-resolver resolve_path "extensions" "custom-provider" --fallback-to-core)

# Create missing directories during resolution
let new_path = (path-resolver resolve_path "infra" "my-infra" --create-missing)
```

### Configuration Resolution

**Hierarchical Configuration Loading**:

```nushell
# Resolve configuration with full hierarchy
let config = (path-resolver resolve_config "user" --workspace-user "developer")

# Load environment-specific configuration
let dev_config = (path-resolver resolve_config "development" --workspace-user "developer")

# Get merged configuration with all overrides
let merged = (path-resolver resolve_config "merged" --workspace-user "developer" --include-overrides)
```

### Extension Discovery

**Automatic Extension Discovery**:

```nushell
# Find custom provider extension
let provider = (path-resolver resolve_extension "providers" "my-aws-provider")

# Discover all available task services
let taskservs = (path-resolver list_extensions "taskservs" --include-core)

# Find cluster definition
let cluster = (path-resolver resolve_extension "clusters" "development-cluster")
```

### Health Checking

**Workspace Health Validation**:

```nushell
# Check workspace health with automatic fixes
let health = (path-resolver check_workspace_health --workspace-user "developer" --fix-issues)

# Validate path resolution chain
let validation = (path-resolver validate_paths --workspace-user "developer" --repair-broken)

# Check runtime directories
let runtime_status = (path-resolver check_runtime_health --workspace-user "developer")
```

## Configuration Management

### Configuration Hierarchy

**Configuration Cascade**:

1. **User Configuration**: `workspace/config/{user}.toml`
2. **Environment Defaults**: `workspace/config/{env}-defaults.toml`
3. **Workspace Defaults**: `workspace/config/dev-defaults.toml`
4. **Core System Defaults**: `config.defaults.toml`

### Environment-Specific Configuration

**Development Environment** (`workspace/config/dev-defaults.toml`):

```toml
[core]
name = "provisioning-dev"
version = "dev-${git.branch}"

[development]
auto_reload = true
verbose_logging = true
experimental_features = true
hot_reload_templates = true

[http]
use_curl = false
timeout = 30
retry_count = 3

[cache]
enabled = true
ttl = 300
refresh_interval = 60

[logging]
level = "debug"
file_rotation = true
max_size = "10 MB"
```

**Testing Environment** (`workspace/config/test-defaults.toml`):

```toml
[core]
name = "provisioning-test"
version = "test-${build.timestamp}"

[testing]
mock_providers = true
ephemeral_resources = true
parallel_tests = true
cleanup_after_test = true

[http]
use_curl = true
timeout = 10
retry_count = 1

[cache]
enabled = false
mock_responses = true

[logging]
level = "info"
test_output = true
```

### User Configuration Example

**User-Specific Configuration** (`workspace/config/{user}.toml`):

```toml
[core]
name = "provisioning-${workspace.user}"
version = "1.0.0-dev"

[infra]
current = "${workspace.user}-development"
default_provider = "upcloud"

[workspace]
user = "developer"
type = "development"
infra_name = "developer-dev"

[development]
preferred_editor = "code"
auto_backup = true
backup_interval = "1h"

[paths]
# Custom paths for this user
templates = "~/custom-templates"
extensions = "~/my-extensions"

[git]
auto_commit = false
commit_message_template = "[${workspace.user}] ${change.type}: ${change.description}"

[notifications]
slack_webhook = "https://hooks.slack.com/..."
email = "developer@company.com"
```

### Configuration Commands

**Workspace Configuration Management**:

```bash
# Show current configuration
nu workspace.nu config show

# Validate configuration
nu workspace.nu config validate --user-name developer

# Edit user configuration
nu workspace.nu config edit --user-name developer

# Show configuration hierarchy
nu workspace.nu config hierarchy --user-name developer

# Merge configurations for debugging
nu workspace.nu config merge --user-name developer --output merged-config.toml
```

## Extension Development

### Extension Types

The workspace provides templates and tools for developing three types of extensions:

1. **Providers**: Cloud provider implementations
2. **Task Services**: Infrastructure service components
3. **Clusters**: Complete deployment solutions

### Provider Extension Development

**Create New Provider**:

```bash
# Copy template
cp -r workspace/extensions/providers/template workspace/extensions/providers/my-provider

# Initialize provider
cd workspace/extensions/providers/my-provider
nu init.nu --provider-name my-provider --author developer
```

**Provider Structure**:

```plaintext
workspace/extensions/providers/my-provider/
├── kcl/
│   ├── provider.ncl          # Provider configuration schema
│   ├── server.ncl            # Server configuration
│   └── version.ncl           # Version management
├── nulib/
│   ├── provider.nu         # Main provider implementation
│   ├── servers.nu          # Server management
│   └── auth.nu             # Authentication handling
├── templates/
│   ├── server.j2           # Server configuration template
│   └── network.j2          # Network configuration template
├── tests/
│   ├── unit/               # Unit tests
│   └── integration/        # Integration tests
└── README.md
```

**Test Provider**:

```bash
# Run provider tests
nu workspace/extensions/providers/my-provider/nulib/provider.nu test

# Test with dry-run
nu workspace/extensions/providers/my-provider/nulib/provider.nu create-server --dry-run

# Integration test
nu workspace/extensions/providers/my-provider/tests/integration/basic-test.nu
```

### Task Service Extension Development

**Create New Task Service**:

```bash
# Copy template
cp -r workspace/extensions/taskservs/template workspace/extensions/taskservs/my-service

# Initialize service
cd workspace/extensions/taskservs/my-service
nu init.nu --service-name my-service --service-type database
```

**Task Service Structure**:

```plaintext
workspace/extensions/taskservs/my-service/
├── kcl/
│   ├── taskserv.ncl          # Service configuration schema
│   ├── version.ncl           # Version configuration with GitHub integration
│   └── kcl.mod             # KCL module dependencies
├── nushell/
│   ├── taskserv.nu         # Main service implementation
│   ├── install.nu          # Installation logic
│   ├── uninstall.nu        # Removal logic
│   └── check-updates.nu    # Version checking
├── templates/
│   ├── config.j2           # Service configuration template
│   ├── systemd.j2          # Systemd service template
│   └── compose.j2          # Docker Compose template
└── manifests/
    ├── deployment.yaml     # Kubernetes deployment
    └── service.yaml        # Kubernetes service
```

### Cluster Extension Development

**Create New Cluster**:

```bash
# Copy template
cp -r workspace/extensions/clusters/template workspace/extensions/clusters/my-cluster

# Initialize cluster
cd workspace/extensions/clusters/my-cluster
nu init.nu --cluster-name my-cluster --cluster-type web-stack
```

**Testing Extensions**:

```bash
# Test extension syntax
nu workspace.nu tools validate-extension providers/my-provider

# Run extension tests
nu workspace.nu tools test-extension taskservs/my-service

# Integration test with infrastructure
nu workspace.nu tools deploy-test clusters/my-cluster --infra test-env
```

## Runtime Management

### Runtime Data Organization

**Per-User Isolation**:

```plaintext
runtime/
├── workspaces/
│   ├── developer/          # Developer's workspace data
│   │   ├── current-infra   # Current infrastructure context
│   │   ├── settings.toml   # Runtime settings
│   │   └── extensions/     # Extension runtime data
│   └── tester/             # Tester's workspace data
├── cache/
│   ├── developer/          # Developer's cache
│   │   ├── providers/      # Provider API cache
│   │   ├── images/         # Container image cache
│   │   └── downloads/      # Downloaded artifacts
│   └── tester/             # Tester's cache
├── state/
│   ├── developer/          # Developer's state
│   │   ├── deployments/    # Deployment state
│   │   └── workflows/      # Workflow state
│   └── tester/             # Tester's state
├── logs/
│   ├── developer/          # Developer's logs
│   │   ├── provisioning.log
│   │   ├── orchestrator.log
│   │   └── extensions/
│   └── tester/             # Tester's logs
└── data/
    ├── developer/          # Developer's data
    │   ├── database.db     # Local database
    │   └── backups/        # Local backups
    └── tester/             # Tester's data
```

### Runtime Management Commands

**Initialize Runtime Environment**:

```bash
# Initialize for current user
nu workspace/tools/runtime-manager.nu init

# Initialize for specific user
nu workspace/tools/runtime-manager.nu init --user-name developer
```

**Runtime Cleanup**:

```bash
# Clean cache older than 30 days
nu workspace/tools/runtime-manager.nu cleanup --type cache --age 30d

# Clean logs with rotation
nu workspace/tools/runtime-manager.nu cleanup --type logs --rotate

# Clean temporary files
nu workspace/tools/runtime-manager.nu cleanup --type temp --force
```

**Log Management**:

```bash
# View recent logs
nu workspace/tools/runtime-manager.nu logs --action tail --lines 100

# Follow logs in real-time
nu workspace/tools/runtime-manager.nu logs --action tail --follow

# Rotate large log files
nu workspace/tools/runtime-manager.nu logs --action rotate

# Archive old logs
nu workspace/tools/runtime-manager.nu logs --action archive --older-than 7d
```

**Cache Management**:

```bash
# Show cache statistics
nu workspace/tools/runtime-manager.nu cache --action stats

# Optimize cache
nu workspace/tools/runtime-manager.nu cache --action optimize

# Clear specific cache
nu workspace/tools/runtime-manager.nu cache --action clear --type providers

# Refresh cache
nu workspace/tools/runtime-manager.nu cache --action refresh --selective
```

**Monitoring**:

```bash
# Monitor runtime usage
nu workspace/tools/runtime-manager.nu monitor --duration 5m --interval 30s

# Check disk usage
nu workspace/tools/runtime-manager.nu monitor --type disk

# Monitor active processes
nu workspace/tools/runtime-manager.nu monitor --type processes --workspace-user developer
```

## Health Monitoring

### Health Check System

The workspace provides comprehensive health monitoring with automatic repair capabilities.

**Health Check Components**:

- **Directory Structure**: Validates workspace directory integrity
- **Configuration Files**: Checks configuration syntax and completeness
- **Runtime Environment**: Validates runtime data and permissions
- **Extension Status**: Checks extension functionality
- **Resource Usage**: Monitors disk space and memory usage
- **Integration Status**: Tests integration with core system

### Health Commands

**Basic Health Check**:

```bash
# Quick health check
nu workspace.nu health

# Detailed health check with all components
nu workspace.nu health --detailed

# Health check with automatic fixes
nu workspace.nu health --fix-issues

# Export health report
nu workspace.nu health --report-format json > health-report.json
```

**Component-Specific Health Checks**:

```bash
# Check directory structure
nu workspace/tools/workspace-health.nu check-directories --workspace-user developer

# Validate configuration files
nu workspace/tools/workspace-health.nu check-config --workspace-user developer

# Check runtime environment
nu workspace/tools/workspace-health.nu check-runtime --workspace-user developer

# Test extension functionality
nu workspace/tools/workspace-health.nu check-extensions --workspace-user developer
```

### Health Monitoring Output

**Example Health Report**:

```json
{
  "workspace_health": {
    "user": "developer",
    "timestamp": "2025-09-25T14:30:22Z",
    "overall_status": "healthy",
    "checks": {
      "directories": {
        "status": "healthy",
        "issues": [],
        "auto_fixed": []
      },
      "configuration": {
        "status": "warning",
        "issues": [
          "User configuration missing default provider"
        ],
        "auto_fixed": [
          "Created missing user configuration file"
        ]
      },
      "runtime": {
        "status": "healthy",
        "disk_usage": "1.2 GB",
        "cache_size": "450 MB",
        "log_size": "120 MB"
      },
      "extensions": {
        "status": "healthy",
        "providers": 2,
        "taskservs": 5,
        "clusters": 1
      }
    },
    "recommendations": [
      "Consider cleaning cache (>400 MB)",
      "Rotate logs (>100 MB)"
    ]
  }
}
```

### Automatic Fixes

**Auto-Fix Capabilities**:

- **Missing Directories**: Creates missing workspace directories
- **Broken Symlinks**: Repairs or removes broken symbolic links
- **Configuration Issues**: Creates missing configuration files with defaults
- **Permission Problems**: Fixes file and directory permissions
- **Corrupted Cache**: Clears and rebuilds corrupted cache entries
- **Log Rotation**: Rotates large log files automatically

## Backup and Restore

### Backup System

**Backup Components**:

- **Configuration**: All workspace configuration files
- **Extensions**: Custom extensions and templates
- **Runtime Data**: User-specific runtime data (optional)
- **Logs**: Application logs (optional)
- **Cache**: Cache data (optional)

### Backup Commands

**Create Backup**:

```bash
# Basic backup
nu workspace.nu backup

# Backup with auto-generated name
nu workspace.nu backup --auto-name

# Comprehensive backup including logs and cache
nu workspace.nu backup --auto-name --include-logs --include-cache

# Backup specific components
nu workspace.nu backup --components config,extensions --name my-backup
```

**Backup Options**:

- `--auto-name`: Generate timestamp-based backup name
- `--include-logs`: Include application logs
- `--include-cache`: Include cache data
- `--components`: Specify components to backup
- `--compress`: Create compressed backup archive
- `--encrypt`: Encrypt backup with age/sops
- `--remote`: Upload to remote storage (S3, etc.)

### Restore System

**List Available Backups**:

```bash
# List all backups
nu workspace.nu restore --list-backups

# List backups with details
nu workspace.nu restore --list-backups --detailed

# Show backup contents
nu workspace.nu restore --show-contents --backup-name workspace-developer-20250925_143022
```

**Restore Operations**:

```bash
# Restore latest backup
nu workspace.nu restore --latest

# Restore specific backup
nu workspace.nu restore --backup-name workspace-developer-20250925_143022

# Selective restore
nu workspace.nu restore --selective --backup-name my-backup

# Restore to different user
nu workspace.nu restore --backup-name my-backup --restore-to different-user
```

**Advanced Restore Options**:

- `--selective`: Choose components to restore interactively
- `--restore-to`: Restore to different user workspace
- `--merge`: Merge with existing workspace (don't overwrite)
- `--dry-run`: Show what would be restored without doing it
- `--verify`: Verify backup integrity before restore

### Reset and Cleanup

**Workspace Reset**:

```bash
# Reset with backup
nu workspace.nu reset --backup-first

# Reset keeping configuration
nu workspace.nu reset --backup-first --keep-config

# Complete reset (dangerous)
nu workspace.nu reset --force --no-backup
```

**Cleanup Operations**:

```bash
# Clean old data with dry-run
nu workspace.nu cleanup --type old --age 14d --dry-run

# Clean cache forcefully
nu workspace.nu cleanup --type cache --force

# Clean specific user data
nu workspace.nu cleanup --user-name old-user --type all
```

## Troubleshooting

### Common Issues

#### Workspace Not Found

**Error**: `Workspace for user 'developer' not found`

```bash
# Solution: Initialize workspace
nu workspace.nu init --user-name developer
```

#### Path Resolution Errors

**Error**: `Path resolution failed for config/user`

```bash
# Solution: Fix with health check
nu workspace.nu health --fix-issues

# Manual fix
nu workspace/lib/path-resolver.nu resolve_path "config" "user" --create-missing
```

#### Configuration Errors

**Error**: `Invalid configuration syntax in user.toml`

```bash
# Solution: Validate and fix configuration
nu workspace.nu config validate --user-name developer

# Reset to defaults
cp workspace/config/local-overrides.toml.example workspace/config/developer.toml
```

#### Runtime Issues

**Error**: `Runtime directory permissions error`

```bash
# Solution: Reinitialize runtime
nu workspace/tools/runtime-manager.nu init --user-name developer --force

# Fix permissions manually
chmod -R 755 workspace/runtime/workspaces/developer
```

#### Extension Issues

**Error**: `Extension 'my-provider' not found or invalid`

```bash
# Solution: Validate extension
nu workspace.nu tools validate-extension providers/my-provider

# Reinitialize extension from template
cp -r workspace/extensions/providers/template workspace/extensions/providers/my-provider
```

### Debug Mode

**Enable Debug Logging**:

```bash
# Set debug environment
export PROVISIONING_DEBUG=true
export PROVISIONING_LOG_LEVEL=debug
export PROVISIONING_WORKSPACE_USER=developer

# Run with debug
nu workspace.nu health --detailed
```

### Performance Issues

**Slow Operations**:

```bash
# Check disk space
df -h workspace/

# Check runtime data size
du -h workspace/runtime/workspaces/developer/

# Optimize workspace
nu workspace.nu cleanup --type cache
nu workspace/tools/runtime-manager.nu cache --action optimize
```

### Recovery Procedures

**Corrupted Workspace**:

```bash
# 1. Backup current state
nu workspace.nu backup --name corrupted-backup --force

# 2. Reset workspace
nu workspace.nu reset --backup-first

# 3. Restore from known good backup
nu workspace.nu restore --latest-known-good

# 4. Validate health
nu workspace.nu health --detailed --fix-issues
```

**Data Loss Prevention**:

- Enable automatic backups: `backup_interval = "1h"` in user config
- Use version control for custom extensions
- Regular health checks: `nu workspace.nu health`
- Monitor disk space and set up alerts

This workspace management system provides a robust foundation for development while maintaining isolation and providing comprehensive tools for
maintenance and troubleshooting.