# Provisioning Daemon Internals

Deep dive into the architecture, implementation, and design of the `provisioning-daemon` crate.

## Crate Structure

```text
provisioning-daemon/
├── Cargo.toml
├── src/
│   ├── lib.rs
│   ├── server.rs          # TCP server implementation
│   ├── client.rs          # Client connection handler
│   ├── request.rs         # Request parsing/serialization
│   ├── handler.rs         # Operation handlers
│   ├── cache/
│   │   ├── mod.rs
│   │   ├── workspace.rs
│   │   ├── resource.rs
│   │   └── config.rs
│   ├── metrics.rs         # Prometheus metrics
│   ├── logging.rs         # Structured logging
│   └── bin/
│       └── daemon.rs      # Binary entrypoint
└── tests/
    ├── integration_tests.rs
    └── performance_tests.rs
```

## Server Architecture

### TCP Server Implementation

```rust
// src/server.rs
use tokio::net::{TcpListener, TcpStream};
use std::sync::Arc;

pub struct DaemonServer {
    listener: TcpListener,
    handler: Arc<RequestHandler>,
    cache: Arc<DaemonCache>,
    config: Arc<DaemonConfig>,
    metrics: Arc<MetricsCollector>,
}

impl DaemonServer {
    pub async fn new(config: DaemonConfig) -> Result<Self> {
        let addr = format!("{}:{}", config.bind_address, config.port);
        let listener = TcpListener::bind(&addr).await?;

        Ok(Self {
            listener,
            handler: Arc::new(RequestHandler::new()),
            cache: Arc::new(DaemonCache::new(config.cache_ttl)),
            config: Arc::new(config),
            metrics: Arc::new(MetricsCollector::new()),
        })
    }

    pub async fn run(&self) -> Result<()> {
        loop {
            let (socket, peer_addr) = self.listener.accept().await?;

            let handler = self.handler.clone();
            let cache = self.cache.clone();
            let config = self.config.clone();
            let metrics = self.metrics.clone();

            tokio::spawn(async move {
                if let Err(e) = Self::handle_connection(
                    socket,
                    handler,
                    cache,
                    config,
                    metrics,
                ).await {
                    eprintln!("Connection error: {}", e);
                }
            });
        }
    }

    async fn handle_connection(
        socket: TcpStream,
        handler: Arc<RequestHandler>,
        cache: Arc<DaemonCache>,
        config: Arc<DaemonConfig>,
        metrics: Arc<MetricsCollector>,
    ) -> Result<()> {
        let mut client = DaemonClient::new(socket, handler, cache, config, metrics);
        client.handle_requests().await
    }
}
```

### Connection Handling

```rust
// src/client.rs
pub struct DaemonClient {
    stream: BufReader<BufWriter<TcpStream>>,
    handler: Arc<RequestHandler>,
    cache: Arc<DaemonCache>,
    config: Arc<DaemonConfig>,
    metrics: Arc<MetricsCollector>,
}

impl DaemonClient {
    pub async fn handle_requests(&mut self) -> Result<()> {
        loop {
            // Read request line (JSON)
            let mut line = String::new();
            let bytes_read = self.stream.read_line(&mut line).await?;

            if bytes_read == 0 {
                // Connection closed
                break;
            }

            let request: Request = serde_json::from_str(&line)?;
            let response = self.process_request(request).await;

            // Write response
            self.stream.write_all(serde_json::to_string(&response)?.as_bytes()).await?;
            self.stream.write_all(b"
").await?;
            self.stream.flush().await?;
        }

        Ok(())
    }

    async fn process_request(&mut self, request: Request) -> Response {
        let start = Instant::now();

        // Check cache first
        if let Some(cached_response) = self.cache.get(&request).await {
            self.metrics.record_cache_hit(&request.command);
            return Response {
                id: request.id,
                cached: true,
                ..cached_response
            };
        }

        // Execute request
        let result = match self.handler.handle(&request).await {
            Ok(result) => {
                self.metrics.record_success(&request.command);
                result
            },
            Err(e) => {
                self.metrics.record_error(&request.command);
                return Response {
                    id: request.id,
                    status: "error".to_string(),
                    error: Some(e.to_string()),
                    execution_time_ms: start.elapsed().as_millis() as u64,
                    ..Default::default()
                };
            }
        };

        // Cache result if cacheable
        if request.cacheable {
            self.cache.set(&request, &result).await;
        }

        Response {
            id: request.id,
            status: "success".to_string(),
            result,
            execution_time_ms: start.elapsed().as_millis() as u64,
            ..Default::default()
        }
    }
}
```

## Request Processing

### Request Types and Routing

```rust
// src/request.rs
#[derive(Serialize, Deserialize)]
pub struct Request {
    pub id: String,
    pub command: String,
    pub subcommand: Option<String>,
    pub args: serde_json::Value,
    pub cacheable: bool,
    pub timeout_seconds: Option<u64>,
}

// src/handler.rs
pub struct RequestHandler {
    workspace_handler: Arc<WorkspaceHandler>,
    resource_handler: Arc<ResourceHandler>,
    config_handler: Arc<ConfigHandler>,
    deployment_handler: Arc<DeploymentHandler>,
}

impl RequestHandler {
    pub async fn handle(&self, request: &Request) -> Result<serde_json::Value> {
        match request.command.as_str() {
            "workspace" => self.workspace_handler.handle(request).await,
            "resource" => self.resource_handler.handle(request).await,
            "config" => self.config_handler.handle(request).await,
            "deployment" => self.deployment_handler.handle(request).await,
            _ => Err("Unknown command".into()),
        }
    }
}
```

## Caching Layer

### LRU Cache Implementation

```rust
// src/cache/mod.rs
use lru::LruCache;
use std::sync::Arc;
use tokio::sync::RwLock;

pub struct DaemonCache {
    workspace_cache: Arc<RwLock<LruCache<String, CacheEntry<Workspace>>>>,
    resource_cache: Arc<RwLock<LruCache<String, CacheEntry<Resource>>>>,
    config_cache: Arc<RwLock<LruCache<String, CacheEntry<Config>>>>,
    ttl: Duration,
}

#[derive(Clone)]
struct CacheEntry<T> {
    data: T,
    inserted_at: Instant,
}

impl<T> CacheEntry<T> {
    fn is_expired(&self, ttl: Duration) -> bool {
        self.inserted_at.elapsed() > ttl
    }
}

impl DaemonCache {
    pub async fn get<T: Clone>(&self, key: &str) -> Option<T> {
        // Cache logic for different types based on key prefix
        match key.split(':').next() {
            Some("workspace") => {
                let cache = self.workspace_cache.read().await;
                cache.peek(key).and_then| ( entr| y {
                    if !entry.is_expired(self.ttl) {
                        Some(entry.data.clone())
                    } else {
                        None
                    }
                })
            },
            _ => None,
        }
    }

    pub async fn set<T>(&self, key: String, value: T) {
        // Set cache entry
    }

    pub async fn invalidate(&self, pattern: &str) {
        // Invalidate cache entries matching pattern
        // e.g., "workspace:*" invalidates all workspace entries
    }
}
```

## Metrics and Monitoring

### Prometheus Metrics

```rust
// src/metrics.rs
use prometheus::{Counter, Histogram, Registry};

pub struct MetricsCollector {
    requests_total: Counter,
    request_duration: Histogram,
    cache_hits: Counter,
    cache_misses: Counter,
    errors: Counter,
    active_connections: Gauge,
}

impl MetricsCollector {
    pub fn record_request(&self, command: &str, duration: Duration) {
        self.requests_total.inc();
        self.request_duration.observe(duration.as_secs_f64());
    }

    pub fn record_cache_hit(&self, command: &str) {
        self.cache_hits.inc();
    }

    pub fn record_cache_miss(&self, command: &str) {
        self.cache_misses.inc();
    }

    pub fn record_error(&self, command: &str) {
        self.errors.inc();
    }

    pub fn set_active_connections(&self, count: usize) {
        self.active_connections.set(count as i64);
    }

    pub fn get_metrics(&self) -> MetricsSnapshot {
        MetricsSnapshot {
            total_requests: self.requests_total.get(),
            cache_hit_rate: self.cache_hits.get() / (self.cache_hits.get() + self.cache_misses.get()),
            error_rate: self.errors.get() / self.requests_total.get(),
            active_connections: self.active_connections.get() as usize,
        }
    }
}
```

### Health Check Endpoint

```rust
// Health check for daemon
#[get("/health")]
pub async fn health_check(
    metrics: web::Data<MetricsCollector>,
) -> HttpResponse {
    let metrics = metrics.get_metrics();

    let status = if metrics.error_rate < 0.01 {
        "healthy"
    } else {
        "degraded"
    };

    HttpResponse::Ok().json(json!({
        "status": status,
        "uptime": format!("{:?}", metrics.uptime),
        "total_requests": metrics.total_requests,
        "cache_hit_rate": metrics.cache_hit_rate,
        "active_connections": metrics.active_connections,
    }))
}
```

## Graceful Shutdown

```rust
// src/server.rs
impl DaemonServer {
    pub async fn shutdown(&self) {
        // Stop accepting new connections
        drop(self.listener);

        // Give existing connections time to complete
        let shutdown_timeout = Duration::from_secs(30);
        let start = Instant::now();

        while self.metrics.active_connections() > 0 && start.elapsed() < shutdown_timeout {
            tokio::time::sleep(Duration::from_millis(100)).await;
        }

        // Force close remaining connections if timeout exceeded
        if self.metrics.active_connections() > 0 {
            eprintln!("Forcing shutdown of {} remaining connections",
                self.metrics.active_connections());
        }
    }
}
```

## Performance Optimizations

### Connection Pool (Client-Side)

```rust
// provisioning-cli client integration
pub struct DaemonConnectionPool {
    connections: Arc<RwLock<VecDeque<TcpStream>>>,
    pool_size: usize,
    address: String,
}

impl DaemonConnectionPool {
    pub async fn get_connection(&self) -> Result<PooledConnection> {
        let mut conns = self.connections.write().await;

        // Try to reuse existing connection
        if let Some(conn) = conns.pop_front() {
            return Ok(PooledConnection::new(conn, self.address.clone()));
        }

        // Create new connection if pool not full
        if conns.len() < self.pool_size {
            let conn = TcpStream::connect(&self.address).await?;
            return Ok(PooledConnection::new(conn, self.address.clone()));
        }

        // Wait for connection to be available
        Err("Connection pool exhausted".into())
    }
}
```

### Async I/O with Tokio

```rust
// Non-blocking I/O for all operations
pub async fn handle_workspace_list(cache: Arc<DaemonCache>) -> Result<Vec<Workspace>> {
    // Check cache first (non-blocking)
    if let Some(cached) = cache.get("workspace:list").await {
        return Ok(cached);
    }

    // Fetch from database asynchronously
    let workspaces = provisioning::workspace::list().await?;

    // Update cache asynchronously
    cache.set("workspace:list".to_string(), workspaces.clone()).await;

    Ok(workspaces)
}
```

## Testing

### Integration Tests

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_workspace_list_request() {
        let server = DaemonServer::new(test_config()).await.unwrap();
        let server_handle = tokio::spawn(async move {
            server.run().await
        });

        // Give server time to start
        tokio::time::sleep(Duration::from_millis(100)).await;

        // Connect as client
        let mut client = TcpStream::connect("127.0.0.1:9090").await.unwrap();

        // Send request
        let request = Request {
            id: "test-1".to_string(),
            command: "workspace".to_string(),
            subcommand: Some("list".to_string()),
            args: json!({}),
            cacheable: true,
            timeout_seconds: Some(10),
        };

        client.write_all(serde_json::to_string(&request).unwrap().as_bytes()).await.unwrap();
        client.write_all(b"
").await.unwrap();

        // Read response
        let mut buf = [0; 4096];
        let n = client.read(&mut buf).await.unwrap();
        let response: Response = serde_json::from_slice(&buf[..n]).unwrap();

        assert_eq!(response.status, "success");
        assert!(!response.result.is_empty());
    }

    #[tokio::test]
    async fn test_cache_effectiveness() {
        let cache = DaemonCache::new(Duration::from_secs(60));

        // First access - cache miss
        let start = Instant::now();
        let _ = cache.get("workspace:list").await;
        let cache_miss_time = start.elapsed();

        // Set cache
        cache.set("workspace:list".to_string(), vec![]).await;

        // Second access - cache hit
        let start = Instant::now();
        let _ = cache.get("workspace:list").await;
        let cache_hit_time = start.elapsed();

        assert!(cache_hit_time < cache_miss_time);
    }
}
```

## Binary Entrypoint

```rust
// src/bin/daemon.rs
#[tokio::main]
async fn main() -> Result<()> {
    // Parse CLI arguments
    let args = DaemonArgs::parse();

    // Initialize logging
    tracing_subscriber::fmt()
        .with_max_level(args.log_level.parse()?)
        .init();

    // Create configuration
    let config = DaemonConfig {
        port: args.port.unwrap_or(9090),
        bind_address: args.bind_address.unwrap_or_else| ( | "127.0.0.1".to_string()),
        cache_ttl: Duration::from_secs(args.cache_ttl_seconds.unwrap_or(300)),
        ..Default::default()
    };

    // Start server
    let server = DaemonServer::new(config).await?;

    // Handle signals
    let shutdown = tokio::signal::ctrl_c();
    tokio::select! {
        _ = shutdown => {
            println!("Shutting down daemon...");
            server.shutdown().await;
        }
        result = server.run() => {
            result?;
        }
    }

    Ok(())
}
```

## Dependencies

```toml
[dependencies]
tokio = { version = "1.0", features = ["full"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tracing = "0.1"
tracing-subscriber = "0.3"
prometheus = "0.13"
lru = "0.12"
```

## See Also

- [Provisioning Daemon Feature Documentation](../features/provisioning-daemon.md)
- [Nushell Script Best Practices](./nushell-script-best-practices.md)
- [Performance Optimization Guide](../operations/performance-optimization.md)
