# Provisioning Daemon Feature

High-performance background service that accelerates Nushell script execution by maintaining a persistent Rust process.

## Overview

The `provisioning-daemon` is a Rust microservice that eliminates the startup overhead of
spawning new CLI processes for each Nushell script operation. Instead of:

```text
Script calls provisioning → Spawn new process (500ms) → Execute → Exit
Script calls provisioning → Spawn new process (500ms) → Execute → Exit
```

It provides:

```text
Script connects to daemon → Execute in-process (10ms) → Return result
Script uses cached connection → Execute in-process (10ms) → Return result
```

**Performance Improvement**: 50x faster for script-heavy workloads (500ms → 10ms per operation)

<p align="center">
    <img src="../resources/diagrams/features/daemon-architecture.svg"
         alt="Provisioning Daemon TCP Server Connection Pooling LRU Cache Performance" width="700"/>
</p>

## Architecture

```text
Nushell Script
    ↓
Daemon Client Library
    ├─ TCP connection to daemon (port 9090, configurable)
    ├─ Connection pooling
    ├─ Retry logic with backoff
    └─ Automatic fallback to CLI
    ↓
Provisioning Daemon (Rust)
    ├─ Accept requests
    ├─ Route to handlers
    ├─ Maintain state cache
    ├─ Log operations
    └─ Return results
    ↓
Client receives response
```

## Installation and Setup

### Start the Daemon

```bash
# Start daemon with default settings
provisioning daemon start

# Start with custom port
provisioning daemon start --port 9091

# Start with logging
provisioning daemon start --log-level debug

# Start in background
provisioning daemon start --background

# Output:
# ✓ Daemon started on 127.0.0.1:9090
# PID: 12345
# Log file: /var/log/provisioning-daemon.log
```

### Configuration

```nushell
# Enable daemon in config
provisioning config set --key daemon.enabled --value true

# Set daemon port
provisioning config set --key daemon.port --value 9090

# Set connection timeout
provisioning config set --key daemon.connection_timeout_seconds --value 10

# Enable auto-restart on crash
provisioning config set --key daemon.auto_restart --value true

# Set log level
provisioning config set --key daemon.log_level --value info
```

### Configuration File

```yaml
# ~/.config/provisioning/daemon.yaml
daemon:
  enabled: true
  port: 9090
  bind_address: 127.0.0.1

  connection:
    max_connections: 100
    timeout_seconds: 10
    keep_alive: true
    keep_alive_interval_seconds: 30

  performance:
    cache_enabled: true
    cache_ttl_seconds: 300
    request_queue_size: 1000

  logging:
    level: info
    file: /var/log/provisioning-daemon.log
    max_size_mb: 100
    max_backups: 5

  security:
    require_auth: false
    allowed_clients: ["127.0.0.1", "::1"]
```

## Usage from Nushell

### Direct API

```nushell
# Use daemon automatically (if running)
provisioning workspace list

# This automatically:
# 1. Checks if daemon is running
# 2. Connects and sends request
# 3. Returns result
# 4. Falls back to CLI if daemon unavailable
```

### Explicit Daemon Connection

```nushell
# Connect to daemon explicitly
let daemon_client = (provisioning daemon connect)

# Execute operations through daemon
let workspaces = ($daemon_client | send "workspace list")
let resources = ($daemon_client | send "resource list")
let status = ($daemon_client | send "status")

# Operations are very fast (10-50ms each)
```

### Daemon-Optimized Script

```nushell
def bulk-operations-with-daemon [] {
    print "Using daemon for bulk operations..."

    # Connect once, use for multiple operations
    let daemon = (provisioning daemon connect)

    # Get all workspaces
    let workspaces = ($daemon | send "workspace list" | from json)

    # For each workspace, get resources (parallel)
    let all_resources = (
        $workspaces
        | par-each -n 4 { | workspace |
            let resources = ($daemon | send $"resource list --workspace ($workspace.id)" | from json)
            {
                workspace: $workspace.id,
                resources: $resources
            }
        }
    )

    print $"Processed ($workspaces | length) workspaces in {elapsed}ms"
}
```

## Performance Benchmarks

### Single Operation

```text
CLI (no daemon):        500ms (process spawn + execution)
Daemon:                  10ms (network + execution)
Improvement:            50x faster
```

### Script with 100 Operations

```text
Without daemon:  100 × 500ms = 50,000ms (50 seconds)
With daemon:     100 × 10ms  =  1,000ms (1 second)
Improvement:     50x faster
```

### Real-World Example: Bulk Deployment

```nushell
# Deploy 50 services without daemon: ~42 minutes
for i in (1..50) {
    provisioning deploy --service $"service-($i)"  # 50 seconds each
}

# Deploy 50 services with daemon: ~50 seconds
let daemon = (provisioning daemon connect)
(1..50) | par-each -n 10 { | i |
    $daemon | send $"deploy --service service-($i)"  # 1 second each, 10 parallel
}
```

## Health Monitoring

```nushell
def check-daemon-health [] {
    # Check if daemon is running
    let status = (provisioning daemon status)

    if $status.running {
        print "✓ Daemon is running"
        print $"  PID: ($status.pid)"
        print $"  Uptime: ($status.uptime)"
        print $"  Connections: ($status.active_connections)"
        print $"  Requests processed: ($status.total_requests)"
        print $"  Memory usage: ($status.memory_mb)MB"
    } else {
        print "✗ Daemon is not running"
        print "  To start: provisioning daemon start"
    }
}
```

## Request Protocol

### Request Format

```json
{
  "id": "req-12345",
  "command": "workspace",
  "subcommand": "list",
  "args": {
    "format": "json",
    "limit": 100
  }
}
```

### Response Format

```json
{
  "id": "req-12345",
  "status": "success",
  "code": 0,
  "result": [
    {
      "id": "ws-001",
      "name": "production",
      "status": "active"
    }
  ],
  "execution_time_ms": 12
}
```

## Error Handling

### Automatic Fallback

```nushell
def operation-with-fallback [] {
    try {
        # Try to use daemon
        let daemon = (provisioning daemon connect --timeout 5s)
        let result = ($daemon | send "workspace list")
        print "✓ Used daemon connection"
        return $result
    } catch { | err |
        # Fallback to CLI
        print "⚠️  Daemon unavailable, using CLI fallback"
        let result = (provisioning workspace list)
        return $result
    }
}
```

### Retry Logic

```rust
// provisioning-daemon client code
async fn execute_with_retry(
    request: Request,
    max_retries: usize,
) -> Result<Response> {
    let mut retries = 0;
    let mut backoff = Duration::from_millis(10);

    loop {
        match execute(request).await {
            Ok(response) => return Ok(response),
            Err(e) if retries < max_retries => {
                retries += 1;
                tokio::time::sleep(backoff).await;
                backoff = Duration::from_millis((backoff.as_millis() * 2).min(5000));
            },
            Err(e) => return Err(e),
        }
    }
}
```

## State Caching

The daemon maintains caches to speed up repeated queries:

```rust
pub struct DaemonCache {
    workspace_cache: Arc<RwLock<LruCache<String, Workspace>>>,
    resource_cache: Arc<RwLock<LruCache<String, Resource>>>,
    config_cache: Arc<RwLock<LruCache<String, Config>>>,
    cache_ttl: Duration,
}

impl DaemonCache {
    pub async fn get_workspace(&self, id: &str) -> Option<Workspace> {
        self.workspace_cache.read().await.get(id).cloned()
    }

    pub async fn invalidate_workspace(&self, id: &str) {
        self.workspace_cache.write().await.pop(id);
    }
}
```

### Cache Statistics

```nushell
def show-daemon-cache-stats [] {
    let stats = (provisioning daemon cache-stats)

    print "Cache Statistics:"
    print $"Workspace cache: ($stats.workspace_cache.hits) hits, ($stats.workspace_cache.misses) misses"
    print $"  Hit rate: ($stats.workspace_cache.hit_rate)%"
    print $"Resource cache: ($stats.resource_cache.hits) hits, ($stats.resource_cache.misses) misses"
    print $"  Hit rate: ($stats.resource_cache.hit_rate)%"
    print $"Config cache: ($stats.config_cache.hits) hits, ($stats.config_cache.misses) misses"
    print $"  Hit rate: ($stats.config_cache.hit_rate)%"
}
```

## Troubleshooting

### Daemon Won't Start

```bash
# Check if port is already in use
lsof -i :9090

# Try different port
provisioning daemon start --port 9091

# Check logs
tail -f /var/log/provisioning-daemon.log
```

### Connection Timeout

```nushell
# Increase connection timeout
provisioning config set --key daemon.connection_timeout_seconds --value 30

# Restart daemon
provisioning daemon restart
```

### High Memory Usage

```nushell
# Reduce cache size
provisioning config set --key daemon.cache_max_items --value 100

# Clear cache
provisioning daemon cache-clear

# Restart daemon
provisioning daemon restart
```

## Advanced Configuration

### Load Balancing Multiple Daemons

```nushell
def multi-daemon-setup [] {
    # Start multiple daemon instances
    provisioning daemon start --port 9090
    provisioning daemon start --port 9091
    provisioning daemon start --port 9092

    # Configure client-side load balancing
    provisioning config set --key daemon.endpoints --value [
        "127.0.0.1:9090",
        "127.0.0.1:9091",
        "127.0.0.1:9092"
    ]

    # Configure load balancing strategy
    provisioning config set --key daemon.load_balancing --value "round-robin"
}
```

### Custom Request Middleware

```rust
pub struct RequestLogger {
    inner: Box<dyn Handler>,
}

impl Handler for RequestLogger {
    async fn handle(&self, request: Request) -> Result<Response> {
        let start = Instant::now();
        let result = self.inner.handle(request).await;
        let duration = start.elapsed();

        println!("[{}] {:?}", request.id, duration);

        result
    }
}
```

## Monitoring and Metrics

```bash
# Get daemon metrics
provisioning daemon metrics

# Output:
# Uptime: 2d 15h 34m
# Total requests: 45,234
# Success rate: 99.8%
# Avg response time: 8.2ms
# P99 response time: 45ms
# Memory: 234MB
# Connections: 12/100
# Cache hit rate: 87%
```

## Best Practices

1. **Always Enable for Scripts**: 50x performance improvement for Nushell scripts
2. **Monitor Memory**: Set appropriate cache sizes for your workload
3. **Use Connection Pooling**: Reuse connections across operations
4. **Handle Fallback**: Always have CLI fallback for reliability
5. **Graceful Degradation**: Scripts work with or without daemon
6. **Regular Restarts**: Restart daemon weekly to clear state
7. **Log Analysis**: Monitor logs for errors and patterns

## Security Considerations

- Daemon only listens on localhost (127.0.0.1) by default
- No authentication required for local connections
- All operations logged with timestamps
- Consider firewall rules in multi-tenant environments

## See Also

- [Nushell Script Best Practices](./nushell-script-best-practices.md)
- [Batch Workflow System](../infrastructure/batch-workflows.md)
- [Orchestrator System](../features/orchestrator-system.md)
