prvng_extensions/taskservs/container_runtime/containerd/README.md

# Containerd Task Service

## Overview

The Containerd task service provides a complete installation and configuration of [containerd](https://containerd.io/), an industry-standard container runtime that manages the complete container lifecycle on Linux and Windows systems. Containerd is designed to be embedded into larger systems and provides a consistent, reliable foundation for running containers at scale.

## Features

### Core Runtime Capabilities
- **Container Lifecycle Management** - Complete container creation, execution, and deletion
- **Image Management** - Pull, push, and manage container images
- **Snapshot Management** - Efficient filesystem snapshots for container layers
- **Network Integration** - CNI (Container Network Interface) plugin support
- **Storage Management** - Pluggable storage drivers and volume management

### Runtime Support
- **Multiple Runtime Backends** - Support for runc, crun, youki, and custom runtimes
- **OCI Compliance** - Full Open Container Initiative specification compliance
- **Systemd Integration** - Complete systemd service management
- **CRI Support** - Container Runtime Interface for Kubernetes integration
- **Security Features** - User namespaces, seccomp, and AppArmor support

### Management Features
- **Health Monitoring** - Built-in health checks and monitoring endpoints
- **Logging Integration** - Structured logging with configurable levels
- **Metrics Export** - Prometheus metrics for monitoring and alerting
- **Plugin Architecture** - Extensible plugin system for custom functionality
- **Multi-Platform** - Support for Linux and Windows containers

### Development Tools
- **crictl Integration** - Kubernetes CRI debugging and inspection tool
- **Content Store** - Efficient content-addressable storage
- **Event System** - Real-time container lifecycle events
- **API Access** - gRPC API for programmatic access
- **Debugging Tools** - Built-in debugging and introspection capabilities

## Configuration

### Basic Configuration
```kcl
containerd: Containerd = {
    name: "containerd"
    version: "1.7.18"
    runtime_default: "runc"
    runtimes: "runc"
}
```

### Production Configuration
```kcl
containerd: Containerd = {
    name: "containerd"
    version: "1.7.18"
    runtime_default: "runc"
    runtimes: "runc,crun,youki"
    config: {
        version: 2
        root: "/var/lib/containerd"
        state: "/run/containerd"
        plugin_dir: ""
        disabled_plugins: []
        required_plugins: []
        oom_score: 0
        grpc: {
            address: "/run/containerd/containerd.sock"
            tcp_address: ""
            tcp_tls_cert: ""
            tcp_tls_key: ""
            uid: 0
            gid: 0
            max_recv_message_size: 16777216
            max_send_message_size: 16777216
        }
        debug: {
            address: ""
            uid: 0
            gid: 0
            level: ""
        }
    }
    plugins: {
        cri: {
            disable_tcp_service: true
            stream_server_address: "127.0.0.1"
            stream_server_port: "0"
            stream_idle_timeout: "4h"
            enable_selinux: false
            selinux_category_range: 1024
            sandbox_image: "registry.k8s.io/pause:3.9"
            stats_collect_period: 10
            systemd_cgroup: true
            enable_tls_streaming: false
            max_container_log_line_size: 16384
            disable_cgroup: false
            disable_apparmor: false
            restrict_oom_score_adj: false
            max_concurrent_downloads: 3
            disable_proc_mount: false
            unset_seccomp_profile: ""
            tolerate_missing_hugetlb_controller: true
            disable_hugetlb_controller: true
            ignore_image_defined_volumes: false
        }
    }
    crictl: {
        runtime_endpoint: "/run/containerd/containerd.sock"
        image_endpoint: "/run/containerd/containerd.sock"
        timeout: 2
        debug: false
        pull_image_on_create: false
        disable_pull_on_run: false
    }
}
```

### Multi-Runtime Configuration
```kcl
containerd: Containerd = {
    name: "containerd"
    version: "1.7.18"
    runtime_default: "runc"
    runtimes: "runc,crun,youki"
    runtime_configs: {
        runc: {
            runtime_type: "io.containerd.runc.v2"
            options: {
                NoPivotRoot: false
                NoNewKeyring: false
                ShimCgroup: ""
                IoUid: 0
                IoGid: 0
                BinaryName: "/usr/local/bin/runc"
                Root: ""
                CriuPath: ""
                SystemdCgroup: true
            }
        }
        crun: {
            runtime_type: "io.containerd.runc.v2"
            options: {
                BinaryName: "/usr/local/bin/crun"
                SystemdCgroup: true
            }
        }
        youki: {
            runtime_type: "io.containerd.runc.v2"
            options: {
                BinaryName: "/usr/local/bin/youki"
                SystemdCgroup: false
            }
        }
    }
}
```

### Security-Hardened Configuration
```kcl
containerd: Containerd = {
    name: "containerd"
    version: "1.7.18"
    # ... base configuration
    security: {
        apparmor_profile: "containerd-default"
        seccomp_profile: "/etc/containerd/seccomp.json"
        selinux_enabled: true
        no_new_privileges: true
        user_namespaces: true
    }
    plugins: {
        cri: {
            # ... other CRI config
            enable_selinux: true
            disable_apparmor: false
            restrict_oom_score_adj: true
            unset_seccomp_profile: ""
            seccomp_profile_root: "/var/lib/containerd/seccomp"
        }
    }
    limits: {
        max_containers: 1000
        max_images: 500
        max_snapshots: 1000
    }
}
```

### High-Performance Configuration
```kcl
containerd: Containerd = {
    name: "containerd"
    version: "1.7.18"
    # ... base configuration
    performance: {
        max_concurrent_downloads: 10
        max_concurrent_uploads: 5
        disable_snapshot_annotations: true
        snapshotter: "overlayfs"
    }
    plugins: {
        snapshots: {
            overlayfs: {
                root_path: "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
                upperdir_label: false
            }
        }
        content: {
            gc: {
                schedule: "*/5 * * * *"
                threshold: 75
            }
        }
    }
    resources: {
        cpu_quota: 0
        memory_limit: 0
        pid_limit: 0
    }
}
```

## Usage

### Deploy Containerd
```bash
./core/nulib/provisioning taskserv create containerd --infra <infrastructure-name>
```

### List Available Task Services
```bash
./core/nulib/provisioning taskserv list
```

### SSH to Containerd Server
```bash
./core/nulib/provisioning server ssh <containerd-server>
```

### Service Management
```bash
# Check containerd status
systemctl status containerd

# Start/stop containerd
systemctl start containerd
systemctl stop containerd
systemctl restart containerd

# View containerd logs
journalctl -u containerd -f

# Check containerd version
containerd --version
```

### Container Management with crictl
```bash
# List running containers
crictl ps

# List all containers
crictl ps -a

# Create a container
crictl create <container-config.json> <pod-config.json>

# Start a container
crictl start <container-id>

# Execute command in container
crictl exec <container-id> <command>

# Get container logs
crictl logs <container-id>

# Stop a container
crictl stop <container-id>

# Remove a container
crictl rm <container-id>
```

### Image Management
```bash
# List images
crictl images

# Pull an image
crictl pull <image-name>

# Remove an image
crictl rmi <image-id>

# Inspect an image
crictl inspecti <image-id>
```

### Pod Management
```bash
# List pods
crictl pods

# Create a pod
crictl runp <pod-config.json>

# Stop a pod
crictl stopp <pod-id>

# Remove a pod
crictl rmp <pod-id>

# Inspect a pod
crictl inspectp <pod-id>
```

### Runtime Management
```bash
# Check runtime status
crictl info

# Get runtime version
crictl version

# Check node configuration
crictl config --help

# Debug runtime issues
crictl config --debug
```

## Architecture

### System Architecture
```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Kubernetes    │────│   Containerd     │────│   Container     │
│   kubelet       │    │                  │    │   Runtimes      │
│                 │    │ • CRI Server     │    │                 │
│ • Pod Mgmt      │────│ • Image Service  │────│ • runc          │
│ • Node Status   │    │ • Runtime API    │    │ • crun          │
│ • Volume Mgmt   │    │ • Event System   │    │ • youki         │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

### Component Architecture
```
┌─────────────────────────────────────────────────────────────┐
│                      Containerd Core                       │
├─────────────────────────────────────────────────────────────┤
│  gRPC API Server   │  Event System     │  Metadata Store   │
│                    │                   │                   │
│ • CRI Service      │ • Lifecycle Events│ • Content Store   │
│ • Content API      │ • Health Events   │ • Snapshot Store  │
│ • Image API        │ • Custom Events   │ • Image Store     │
│ • Task API         │ • Event Routing   │ • Lease Store     │
├─────────────────────────────────────────────────────────────┤
│                        Plugin System                       │
├─────────────────────────────────────────────────────────────┤
│  Runtime Plugins   │  Snapshot Plugins │  Content Plugins  │
│                    │                   │                   │
│ • OCI Runtime      │ • OverlayFS       │ • Registry        │
│ • Custom Runtimes  │ • BTRFS           │ • Local Content   │
│ • Runtime Shims    │ • ZFS             │ • Remote Content  │
└─────────────────────────────────────────────────────────────┘
```

### File System Layout
```
/var/lib/containerd/          # Root directory
├── io.containerd.content.v1.content/  # Content store
├── io.containerd.snapshotter.v1.overlayfs/  # Snapshots
├── io.containerd.metadata.v1.bolt/    # Metadata database
└── io.containerd.runtime.v2.task/     # Runtime tasks

/run/containerd/              # Runtime directory
├── containerd.sock          # Main API socket
├── debug.sock              # Debug API socket
└── runc/                   # Runtime state
```

## Supported Operating Systems

- Ubuntu 20.04+ / Debian 11+
- CentOS 8+ / RHEL 8+ / Fedora 35+
- Amazon Linux 2+
- SUSE Linux Enterprise 15+

## System Requirements

### Minimum Requirements
- **RAM**: 1GB (2GB+ recommended)
- **Storage**: 10GB (50GB+ for production)
- **CPU**: 1 core (2+ cores recommended)
- **Kernel**: Linux 4.x+ with namespace support

### Production Requirements
- **RAM**: 4GB+ (depends on workload)
- **Storage**: 100GB+ NVMe SSD
- **CPU**: 4+ cores
- **Network**: High bandwidth for image operations

### Kernel Features
- **Namespaces** - PID, network, mount, UTS, IPC, user
- **Control Groups** - v1 or v2 (systemd support recommended)
- **Overlay Filesystem** - For efficient layer storage
- **Seccomp** - Security computing mode support

## Troubleshooting

### Service Issues
```bash
# Check containerd status
systemctl status containerd

# View detailed logs
journalctl -u containerd --no-pager -l

# Check socket permissions
ls -la /run/containerd/containerd.sock

# Test API connectivity
crictl info
```

### Runtime Issues
```bash
# Check available runtimes
crictl info | grep -A 10 runtimes

# Test specific runtime
crictl --runtime <runtime-name> ps

# Check runtime binaries
which runc crun youki

# Verify runtime installation
runc --version
crun --version
youki --version
```

### Image Issues
```bash
# Check image pull logs
journalctl -u containerd | grep -i pull

# Test registry connectivity
crictl pull hello-world

# Check storage usage
du -sh /var/lib/containerd

# Clear unused images
crictl rmi --prune
```

### Performance Issues
```bash
# Check resource usage
systemctl status containerd
ps aux | grep containerd

# Monitor system calls
strace -p $(pgrep containerd)

# Check filesystem performance
iostat -x 1

# Monitor network I/O
iftop -i eth0
```

### Configuration Issues
```bash
# Validate configuration
containerd config dump

# Check plugin status
crictl info | grep -A 20 plugins

# Test configuration reload
systemctl reload containerd

# Debug configuration parsing
containerd --log-level debug
```

## Security Considerations

### Runtime Security
- **Least Privilege** - Run containers with minimal privileges
- **User Namespaces** - Isolate container users from host users
- **Seccomp Profiles** - Restrict system call access
- **AppArmor/SELinux** - Mandatory access control integration

### Image Security
- **Image Signing** - Verify image signatures before execution
- **Vulnerability Scanning** - Regular security scans of container images
- **Registry Security** - Use secure, authenticated registries
- **Content Trust** - Enable Docker Content Trust for image verification

### Network Security
- **Network Policies** - Implement pod-to-pod communication policies
- **TLS Configuration** - Secure API communication with TLS
- **Firewall Rules** - Restrict network access to containerd APIs
- **Registry Access** - Control access to container registries

### Operational Security
- **Regular Updates** - Keep containerd and runtimes updated
- **Audit Logging** - Enable comprehensive audit logging
- **Resource Limits** - Implement container resource constraints
- **Monitoring** - Continuous monitoring of container activities

## Performance Optimization

### Storage Performance
- **SSD Storage** - Use NVMe SSDs for container storage
- **Snapshotter Selection** - Choose optimal snapshotter for workload
- **Garbage Collection** - Configure efficient garbage collection
- **Content Deduplication** - Leverage layer sharing and deduplication

### Runtime Performance
- **Runtime Selection** - Choose optimal runtime for workload (runc/crun/youki)
- **Cgroup Configuration** - Optimize cgroup settings
- **Memory Management** - Configure memory limits and swapping
- **CPU Affinity** - Pin containers to specific CPU cores

### Network Performance
- **CNI Plugin** - Select high-performance CNI plugins
- **Network Mode** - Use host networking for high-throughput applications
- **Buffer Tuning** - Optimize network buffer sizes
- **Bandwidth Limits** - Configure traffic shaping when needed

### Image Performance
- **Registry Mirrors** - Use local registry mirrors
- **Parallel Downloads** - Configure concurrent download limits
- **Image Caching** - Implement local image caching strategies
- **Layer Optimization** - Optimize container images for size and layers

## Integration Examples

### Kubernetes Integration
```yaml
# kubelet configuration
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerRuntime: remote
containerRuntimeEndpoint: /run/containerd/containerd.sock
imageServiceEndpoint: /run/containerd/containerd.sock
```

### Prometheus Monitoring
```yaml
# containerd metrics configuration
[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false
```

### Log Forwarding
```yaml
# fluentd configuration for containerd logs
<source>
  @type systemd
  tag containerd
  path /var/log/journal
  matches [{ "_SYSTEMD_UNIT": "containerd.service" }]
</source>
```

## Resources

- **Official Documentation**: [containerd.io](https://containerd.io/)
- **GitHub Repository**: [containerd/containerd](https://github.com/containerd/containerd)
- **CRI Tools**: [kubernetes-sigs/cri-tools](https://github.com/kubernetes-sigs/cri-tools)
- **Community**: [containerd.slack.com](https://containerd.slack.com)
- **CNCF Project**: [cncf.io/projects/containerd](https://www.cncf.io/projects/containerd/)