Vapora/provisioning/.woodpecker/SETUP.md

# Woodpecker CI Setup Guide for VAPORA

Comprehensive guide for setting up and using Woodpecker CI/CD pipelines for VAPORA provisioning.

## Overview

Woodpecker is a self-hosted, container-based CI/CD platform compatible with Docker Compose and Kubernetes. This guide covers VAPORA's 5 production-ready Woodpecker pipelines as an alternative to GitHub Actions.

### Key Features

- **Self-Hosted**: Deploy on your own infrastructure (Docker, Kubernetes, VMs)
- **Container-Based**: Runs pipeline steps in isolated Docker containers
- **YAML Pipelines**: Simple YAML syntax for defining workflows
- **Flexible Triggers**: Git webhooks, cron schedules, manual promotions
- **Secret Management**: Built-in secret storage with environment variable injection
- **Artifact Handling**: Workspace persistence across stages
- **Multi-Pipeline Support**: Run multiple pipelines in parallel

### VAPORA Woodpecker Pipelines

| Pipeline | Purpose | Trigger | Duration |
|----------|---------|---------|----------|
| **validate-and-build.yml** | Validate configs, generate artifacts | Push, PR, manual | ~5 min |
| **deploy-docker.yml** | Deploy to Docker Compose | Manual, after validation | ~3 min |
| **deploy-kubernetes.yml** | Deploy to Kubernetes | Manual with dry-run | ~5-10 min |
| **health-check.yml** | Continuous monitoring | Cron (15min, 6hr), manual | ~5 min |
| **rollback.yml** | Safe rollback with verification | Manual only | ~3-5 min |

---

## Prerequisites

### Infrastructure Requirements

**Minimum**:
- Linux server (Ubuntu 20.04+, Debian 11+, CentOS 8+)
- Docker 20.10+ installed and running
- 2 CPU cores, 4GB RAM, 20GB disk

**Recommended for Production**:
- Kubernetes cluster (v1.24+)
- 4+ CPU cores, 8GB+ RAM, 50GB+ disk
- Separate storage for workspace/artifacts
- SSL/TLS for Woodpecker UI

### Prerequisites to Install

```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y docker.io docker-compose git curl wget jq

# Start Docker daemon
sudo systemctl start docker
sudo systemctl enable docker

# Add current user to docker group (after logout/login required)
sudo usermod -aG docker $USER
```

### Git Repository

- GitLab, GitHub, Gitea, or Gogs repository
- Repository webhook URL accessible from Woodpecker server
- OAuth token for repository access (for most Git services)

---

## Installation

### Option 1: Docker Compose Installation (Recommended for Testing)

```bash
# Create Woodpecker directory
mkdir -p ~/woodpecker && cd ~/woodpecker

# Create docker-compose.yml
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:latest
    ports:
      - "80:8000"
      - "443:443"
    environment:
      - WOODPECKER_ADMIN_USER=admin
      - WOODPECKER_ADMIN_PASSWORD=admin123
      - WOODPECKER_GITHUB_SERVER=https://github.com
      - WOODPECKER_GITHUB_CLIENT_ID=<YOUR_GITHUB_CLIENT_ID>
      - WOODPECKER_GITHUB_CLIENT_SECRET=<YOUR_GITHUB_CLIENT_SECRET>
      - WOODPECKER_RPC_SECRET=<GENERATE_WITH: head -c 32 /dev/urandom | base64>
      - WOODPECKER_LOG_LEVEL=info
    volumes:
      - woodpecker-data:/var/lib/woodpecker
    restart: always

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:latest
    environment:
      - WOODPECKER_SERVER=http://woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=<SAME_AS_RPC_SECRET>
      - WOODPECKER_LOG_LEVEL=info
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
    depends_on:
      - woodpecker-server

volumes:
  woodpecker-data:
EOF

# Generate RPC secret
RPC_SECRET=$(head -c 32 /dev/urandom | base64)
echo "RPC_SECRET=$RPC_SECRET"

# Start services
docker-compose up -d

# Logs
docker-compose logs -f
```

### Option 2: Kubernetes Deployment

Create `woodpecker-deployment.yaml`:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: woodpecker

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: woodpecker-server
  namespace: woodpecker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: woodpecker-server
  template:
    metadata:
      labels:
        app: woodpecker-server
    spec:
      containers:
      - name: server
        image: woodpeckerci/woodpecker-server:latest
        ports:
        - containerPort: 8000
        - containerPort: 9000
        env:
        - name: WOODPECKER_ADMIN_USER
          value: "admin"
        - name: WOODPECKER_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: admin-password
        - name: WOODPECKER_RPC_SECRET
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: rpc-secret
        - name: WOODPECKER_GITHUB_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: github-client-id
        - name: WOODPECKER_GITHUB_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: github-client-secret
        volumeMounts:
        - name: woodpecker-data
          mountPath: /var/lib/woodpecker
      volumes:
      - name: woodpecker-data
        persistentVolumeClaim:
          claimName: woodpecker-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: woodpecker-server
  namespace: woodpecker
spec:
  selector:
    app: woodpecker-server
  ports:
  - name: ui
    port: 8000
    targetPort: 8000
  - name: rpc
    port: 9000
    targetPort: 9000
  type: LoadBalancer
```

Deploy:

```bash
# Create secrets
kubectl create secret generic woodpecker-secrets \
  -n woodpecker \
  --from-literal=admin-password=YOUR_PASSWORD \
  --from-literal=rpc-secret=$(head -c 32 /dev/urandom | base64) \
  --from-literal=github-client-id=YOUR_CLIENT_ID \
  --from-literal=github-client-secret=YOUR_CLIENT_SECRET

# Apply deployment
kubectl apply -f woodpecker-deployment.yaml

# Check status
kubectl get pods -n woodpecker
kubectl port-forward -n woodpecker svc/woodpecker-server 8000:8000
```

---

## GitHub Integration Setup

### Step 1: Create GitHub OAuth App

1. Go to GitHub → Settings → Developer settings → OAuth Apps
2. Click "New OAuth App"
3. Fill in:
   - **Application name**: `VAPORA Woodpecker`
   - **Homepage URL**: `https://woodpecker.your-domain.com`
   - **Authorization callback URL**: `https://woodpecker.your-domain.com/authorize`
4. Copy `Client ID` and `Client Secret`

### Step 2: Configure Woodpecker

For Docker Compose:

```bash
# Update docker-compose.yml environment variables
WOODPECKER_GITHUB_CLIENT_ID=your_client_id_here
WOODPECKER_GITHUB_CLIENT_SECRET=your_client_secret_here
WOODPECKER_RPC_SECRET=$(head -c 32 /dev/urandom | base64)
```

For Kubernetes:

```bash
kubectl patch secret woodpecker-secrets -n woodpecker \
  --type=merge \
  -p '{"data":{"github-client-id":"'$(echo -n YOUR_CLIENT_ID | base64)'","github-client-secret":"'$(echo -n YOUR_CLIENT_SECRET | base64)'"}}'
```

### Step 3: Repository Setup

1. Access Woodpecker UI: `http://localhost:8000` (or your domain)
2. Login with admin credentials
3. Go to Admin → Repositories
4. Authorize your VAPORA repository
5. Enable webhooks by visiting `http://your-github.com/settings/hooks`

---

## Secret Management

### Adding Secrets

#### Via Woodpecker UI

1. Go to repository → Settings → Secrets
2. Click "Add secret"
3. Name: `SECRET_NAME`
4. Value: Your secret value
5. Save

#### Via CLI

```bash
# Install woodpecker-cli
go install github.com/woodpeckerci/woodpecker/cmd/woodpecker-cli@latest

# Login
woodpecker-cli login -s http://woodpecker-server:8000 \
  -u admin \
  -p admin_password

# Add secret
woodpecker-cli secret add \
  -r owner/repo \
  -n KUBE_CONFIG_STAGING \
  -v "$(cat ~/.kube/config | base64)"
```

### Required VAPORA Secrets

```bash
# Kubernetes kubeconfigs (base64 encoded)
KUBE_CONFIG_STAGING         # Staging cluster kubeconfig
KUBE_CONFIG_PRODUCTION      # Production cluster kubeconfig

# Optional: Slack notifications
SLACK_WEBHOOK               # General notifications webhook
SLACK_WEBHOOK_ALERTS        # Critical alerts webhook

# Optional: Docker registry
DOCKER_USERNAME             # Docker Hub username
DOCKER_PASSWORD             # Docker Hub access token
```

### Encoding Kubeconfig

```bash
# Get kubeconfig and encode as base64
cat ~/.kube/config | base64 > kube_config_base64.txt

# Use the output in Woodpecker secret UI
cat kube_config_base64.txt

# Verify locally before adding to Woodpecker
echo "$(cat kube_config_base64.txt)" | base64 -d | kubectl cluster-info
```

---

## Pipeline Triggers

### Automatic Triggers

Pipelines trigger automatically when:

```yaml
# On push to main/develop branches (if provisioning files change)
trigger:
  event: [push]
  branch: [main, develop]
  paths:
    include:
      - provisioning/schemas/**
      - provisioning/scripts/**

# On pull requests
trigger:
  event: [pull_request]
  branch: [main, develop]
```

### Manual Triggers (Promotions)

Manually trigger from UI:

1. Go to repository → Active builds
2. Find a completed build
3. Click "Promote"
4. Select pipeline: `deploy-docker`, `deploy-kubernetes`, etc.
5. Set deployment parameters:
   - Mode: `solo`, `multiuser`, `enterprise`
   - Environment: `staging`, `production`
   - Dry-run: `true`/`false`
6. Click "Promote"

### Scheduled Triggers (Cron)

Health check pipeline runs on schedule:

```yaml
trigger:
  cron:
    - "*/15 * * * *"    # Every 15 minutes
    - "0 */6 * * *"     # Every 6 hours
```

---

## Deployment Workflows

### Workflow 1: Local Development

```
Developer pushes to feature branch
    ↓
[Validate & Build] runs automatically
    ↓
Review artifacts in workspace
    ↓
[Deploy to Docker] manually for local testing
    ↓
Test with docker compose
    ↓
Create PR
```

### Workflow 2: Staging Deployment

```
Merge PR to develop
    ↓
[Validate & Build] runs automatically
    ↓
Download artifacts from workspace
    ↓
Manually run [Deploy to Kubernetes]
    - Mode: multiuser
    - Environment: staging
    - Dry-run: true
    ↓
Review dry-run output
    ↓
Run again with dry-run: false
    ↓
[Health Check] verifies deployment
    ↓
Staging live
```

### Workflow 3: Production Deployment

```
Code review approved
    ↓
Merge to main
    ↓
[Validate & Build] runs automatically
    ↓
Manually run [Deploy to Kubernetes]
    - Mode: enterprise
    - Environment: production
    - Dry-run: true
    ↓
Carefully review changes
    ↓
Run with dry-run: false
    ↓
[Health Check] monitoring (auto every 6 hours)
    ↓
Production deployment complete
```

### Workflow 4: Emergency Rollback

```
Production issue detected
    ↓
[Health Check] alerts in Slack
    ↓
Manually run [Rollback Deployment]
    - Environment: production
    ↓
Services restored
    ↓
Investigate root cause
```

---

## Configuration Environment Variables

### For validate-and-build.yml

```bash
ARTIFACTS_DIR=provisioning/artifacts    # Output directory for configs
LOG_DIR=provisioning/logs               # Output directory for logs
```

### For deploy-docker.yml

```bash
ARTIFACTS_DIR=provisioning/artifacts
LOGS_DIR=provisioning/logs
```

### For deploy-kubernetes.yml

```bash
ARTIFACTS_DIR=provisioning/artifacts
LOGS_DIR=provisioning/logs
VAPORA_NAMESPACE=vapora                 # Kubernetes namespace
```

### For health-check.yml

```bash
LOGS_DIR=provisioning/logs
VAPORA_NAMESPACE=vapora
```

---

## Monitoring & Logs

### Via Woodpecker UI

1. Go to repository → Active/Previous builds
2. Click a build to see full pipeline execution
3. Click a stage to see detailed logs
4. Download logs or artifacts

### Via CLI

```bash
# List recent builds
woodpecker-cli build list -r owner/repo

# View build details
woodpecker-cli build view -r owner/repo -b <BUILD_NUMBER>

# Watch build in real-time
woodpecker-cli build watch -r owner/repo -b <BUILD_NUMBER>

# Get build logs
woodpecker-cli build logs -r owner/repo -b <BUILD_NUMBER>
```

### Logs Location

All logs stored in workspace:

```bash
provisioning/logs/
├── validate-solo.log
├── validate-multiuser.log
├── validate-enterprise.log
├── build.log
├── docker/
│   ├── backend.log
│   ├── frontend.log
│   └── all-services.log
├── kubernetes/
│   ├── backend.log
│   ├── agents.log
│   ├── llm-router.log
│   └── events.log
└── health-checks/
    ├── docker-endpoints.log
    ├── k8s-deployments.log
    └── HEALTH_REPORT.md
```

---

## Slack Integration

### Setup Webhook

1. Go to Slack workspace → Apps → Custom Integrations
2. Create Incoming Webhook
3. Select channel: `#deployments`
4. Copy Webhook URL
5. Add to Woodpecker secret: `SLACK_WEBHOOK`

### Slack Messages

**Build Success**:
```
✅ VAPORA Artifact Build Complete
Artifacts ready for deployment
```

**Docker Deploy Success**:
```
✅ VAPORA Docker deployment successful!
Mode: multiuser | Environment: staging
```

**Kubernetes Deploy Success**:
```
✅ VAPORA Kubernetes deployment successful!
Mode: enterprise | Environment: production
```

**Health Check Alert**:
```
❌ VAPORA Health Check Failed
Target: kubernetes
```

**Rollback Alert**:
```
🔙 VAPORA Rollback Executed
Environment: production
Verify service health immediately
```

---

## Troubleshooting

### Pipeline Not Triggering

**Problem**: Push doesn't trigger validate-and-build

**Solution**:
1. Check repository is authorized in Woodpecker
2. Verify webhook exists in GitHub settings
3. Check file paths in `trigger.paths.include` match your changes
4. Enable debug logging: `WOODPECKER_LOG_LEVEL=debug`

### Secret Not Found

**Problem**: `Secret not found` error in logs

**Solution**:
1. Verify secret exists in repository settings
2. Check exact spelling (case-sensitive)
3. Ensure secret value is not empty
4. Test secret value locally before adding

### Kubeconfig Decode Error

**Problem**: `base64: invalid input` during kubectl setup

**Solution**:
```bash
# Test locally first
echo "$(cat kube_config_base64.txt)" | base64 -d | kubectl cluster-info

# If it fails, re-encode
cat ~/.kube/config | base64 | pbcopy  # macOS

# Update secret in Woodpecker UI
```

### Docker Connection Failed

**Problem**: `Cannot connect to Docker daemon` in deploy-docker stage

**Solution**:
1. Ensure Docker socket mounted in agent: `-v /var/run/docker.sock:/var/run/docker.sock`
2. Verify Docker daemon running: `docker ps`
3. Check volume permissions: `sudo chmod 666 /var/run/docker.sock`

### Deployment Hangs

**Problem**: Pipeline stage times out waiting for rollout

**Solution**:
1. Check pod logs: `kubectl logs -n vapora <pod>`
2. Describe pod: `kubectl describe pod -n vapora <pod>`
3. Increase timeout in pipeline stage
4. Check resource requests/limits
5. Verify cluster has sufficient resources

### Workspace Persistence Issues

**Problem**: Files from one stage not available in next stage

**Solution**:
1. Create file in correct location (workspace root or subdirectory)
2. Use absolute paths: `${LOGS_DIR}/output.log`
3. Check artifact uploads in "publish" stages
4. Verify docker volumes: `docker volume ls`

---

## Advanced Configuration

### Multi-Agent Setup

For distributed build execution:

```yaml
# Agent 1 (Docker builds)
environment:
  - WOODPECKER_FILTER_LABELS=type:docker

# Agent 2 (Kubernetes operations)
environment:
  - WOODPECKER_FILTER_LABELS=type:kubernetes

# Agent 3 (Health checks)
environment:
  - WOODPECKER_FILTER_LABELS=type:monitoring
```

### Pipeline Concurrency Control

Limit concurrent executions:

```yaml
concurrency:
  limit: 2              # Max 2 concurrent builds
  timeout_minutes: 60   # Timeout after 60 minutes
```

### Conditional Stage Execution

Run stage only if conditions met:

```yaml
when:
  evaluate: 'return build.Deploy_Environment == "production"'
```

---

## Comparison: Woodpecker vs GitHub Actions

| Feature | Woodpecker | GitHub Actions |
|---------|-----------|---|
| **Hosting** | Self-hosted | GitHub-hosted |
| **YAML Format** | Similar | Familiar |
| **Manual Dispatch** | Promotion UI | workflow_dispatch |
| **Scheduled Workflows** | Cron syntax | schedule syntax |
| **Artifact Storage** | Workspace persistence | upload-artifact action |
| **PR Comments** | Limited | ✓ Native |
| **Slack Integration** | Via webhooks | Actions |
| **Secret Management** | Built-in UI | Built-in |
| **Free for Public** | Self-hosted cost | ✓ Free |
| **Concurrency Control** | ✓ Advanced | ✓ Concurrency groups |
| **Deployment Safety** | Dry-run support | Deployment protection |

### When to Use Woodpecker

- ✓ You want full control over CI/CD infrastructure
- ✓ You need to run on-premise for compliance
- ✓ You prefer self-hosted solutions
- ✓ You have multiple repositories needing unified CI/CD
- ✓ You want to avoid vendor lock-in

### When to Use GitHub Actions

- ✓ You want GitHub-hosted runners (no infrastructure)
- ✓ You prefer tight GitHub integration
- ✓ You want PR comments and GitHub UI integration
- ✓ You're already using GitHub workflow syntax

---

## First Deployment with Woodpecker

### Step 1: Enable Woodpecker for Repository

1. Access Woodpecker UI
2. Click "Administration" → "Repositories"
3. Find VAPORA repository
4. Click to enable
5. Grant webhook access

### Step 2: Create Test Branch

```bash
git checkout -b test/woodpecker-setup
echo "# Woodpecker Test" >> README.md
git add README.md
git commit -m "test: trigger Woodpecker"
git push origin test/woodpecker-setup
```

### Step 3: Monitor Pipeline

1. Go to Woodpecker → repository
2. See "Validate & Build" trigger automatically
3. Monitor pipeline execution
4. Check logs for each stage

### Step 4: Download Artifacts

1. In completed build, find "Files" section
2. Access workspace artifacts:
   - `provisioning/artifacts/` - Generated configs
   - `provisioning/logs/` - Pipeline logs

### Step 5: Test Docker Deployment

1. Download artifacts
2. Go to Woodpecker → repository
3. Click "Promote" on validated build
4. Select "deploy-docker"
5. Set:
   - Mode: `multiuser`
   - Environment: `staging`
   - Dry-run: `true`
6. Monitor deployment

### Step 6: Create Pull Request

```bash
git push origin test/woodpecker-setup
# Create PR on GitHub
```

---

## Security Best Practices

✅ **Do**:
- Use environment-specific kubeconfigs
- Rotate secrets regularly
- Run health checks after deployments
- Enable dry-run by default
- Keep logs for audit trail
- Use RBAC in Kubernetes
- Monitor Slack alerts
- Test on staging first

❌ **Don't**:
- Commit secrets to repository
- Deploy directly to production without testing
- Disable dry-run validation
- Skip health checks
- Use same credentials for all environments
- Share Woodpecker admin credentials
- Keep old pipelines around
- Ignore Slack alerts

---

## Support & Resources

- **Woodpecker Docs**: https://woodpecker-ci.org/docs/intro
- **VAPORA Docs**: See `./../docs/` directory
- **GitHub Actions Guide**: `./../.github/GITHUB_ACTIONS_GUIDE.md`
- **Nushell Scripts**: `provisioning/scripts/*.nu`

---

## Files Created

```
.woodpecker/
├── validate-and-build.yml       (410 lines)
├── deploy-docker.yml            (340 lines)
├── deploy-kubernetes.yml        (380 lines)
├── health-check.yml             (290 lines)
├── rollback.yml                 (330 lines)
└── SETUP.md                     (This file)

Total: 5 pipelines + comprehensive documentation
```

---

## Next Steps

1. ✅ Install and configure Woodpecker server
2. → Integrate with GitHub repository
3. → Add secrets for Kubernetes kubeconfigs
4. → Configure Slack webhooks (optional)
5. → Run first validation pipeline
6. → Test Docker deployment
7. → Test Kubernetes deployment
8. → Configure health checks
9. → Document team runbooks
10. → Deploy to production

---

**Generated**: 2026-01-12
**Status**: Production-ready
**Pipelines**: 5 (validate-and-build, deploy-docker, deploy-kubernetes, health-check, rollback)
**Documentation**: Complete