Vapora/provisioning/.woodpecker/SETUP.md

Woodpecker CI Setup Guide for VAPORA

Comprehensive guide for setting up and using Woodpecker CI/CD pipelines for VAPORA provisioning.

Overview

Woodpecker is a self-hosted, container-based CI/CD platform compatible with Docker Compose and Kubernetes. This guide covers VAPORA's 5 production-ready Woodpecker pipelines as an alternative to GitHub Actions.

Key Features

• Self-Hosted: Deploy on your own infrastructure (Docker, Kubernetes, VMs)
• Container-Based: Runs pipeline steps in isolated Docker containers
• YAML Pipelines: Simple YAML syntax for defining workflows
• Flexible Triggers: Git webhooks, cron schedules, manual promotions
• Secret Management: Built-in secret storage with environment variable injection
• Artifact Handling: Workspace persistence across stages
• Multi-Pipeline Support: Run multiple pipelines in parallel

VAPORA Woodpecker Pipelines

Pipeline	Purpose	Trigger	Duration
validate-and-build.yml	Validate configs, generate artifacts	Push, PR, manual	~5 min
deploy-docker.yml	Deploy to Docker Compose	Manual, after validation	~3 min
deploy-kubernetes.yml	Deploy to Kubernetes	Manual with dry-run	~5-10 min
health-check.yml	Continuous monitoring	Cron (15min, 6hr), manual	~5 min
rollback.yml	Safe rollback with verification	Manual only	~3-5 min

---

Prerequisites

Infrastructure Requirements

Minimum:

• Linux server (Ubuntu 20.04+, Debian 11+, CentOS 8+)
• Docker 20.10+ installed and running
• 2 CPU cores, 4GB RAM, 20GB disk

Recommended for Production:

• Kubernetes cluster (v1.24+)
• 4+ CPU cores, 8GB+ RAM, 50GB+ disk
• Separate storage for workspace/artifacts
• SSL/TLS for Woodpecker UI

Prerequisites to Install

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y docker.io docker-compose git curl wget jq

# Start Docker daemon
sudo systemctl start docker
sudo systemctl enable docker

# Add current user to docker group (after logout/login required)
sudo usermod -aG docker $USER

Git Repository

• GitLab, GitHub, Gitea, or Gogs repository
• Repository webhook URL accessible from Woodpecker server
• OAuth token for repository access (for most Git services)

---

Installation

Option 1: Docker Compose Installation (Recommended for Testing)

# Create Woodpecker directory
mkdir -p ~/woodpecker && cd ~/woodpecker

# Create docker-compose.yml
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:latest
    ports:
      - "80:8000"
      - "443:443"
    environment:
      - WOODPECKER_ADMIN_USER=admin
      - WOODPECKER_ADMIN_PASSWORD=admin123
      - WOODPECKER_GITHUB_SERVER=https://github.com
      - WOODPECKER_GITHUB_CLIENT_ID=<YOUR_GITHUB_CLIENT_ID>
      - WOODPECKER_GITHUB_CLIENT_SECRET=<YOUR_GITHUB_CLIENT_SECRET>
      - WOODPECKER_RPC_SECRET=<GENERATE_WITH: head -c 32 /dev/urandom | base64>
      - WOODPECKER_LOG_LEVEL=info
    volumes:
      - woodpecker-data:/var/lib/woodpecker
    restart: always

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:latest
    environment:
      - WOODPECKER_SERVER=http://woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=<SAME_AS_RPC_SECRET>
      - WOODPECKER_LOG_LEVEL=info
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
    depends_on:
      - woodpecker-server

volumes:
  woodpecker-data:
EOF

# Generate RPC secret
RPC_SECRET=$(head -c 32 /dev/urandom | base64)
echo "RPC_SECRET=$RPC_SECRET"

# Start services
docker-compose up -d

# Logs
docker-compose logs -f

Option 2: Kubernetes Deployment

Create woodpecker-deployment.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: woodpecker

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: woodpecker-server
  namespace: woodpecker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: woodpecker-server
  template:
    metadata:
      labels:
        app: woodpecker-server
    spec:
      containers:
      - name: server
        image: woodpeckerci/woodpecker-server:latest
        ports:
        - containerPort: 8000
        - containerPort: 9000
        env:
        - name: WOODPECKER_ADMIN_USER
          value: "admin"
        - name: WOODPECKER_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: admin-password
        - name: WOODPECKER_RPC_SECRET
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: rpc-secret
        - name: WOODPECKER_GITHUB_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: github-client-id
        - name: WOODPECKER_GITHUB_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: woodpecker-secrets
              key: github-client-secret
        volumeMounts:
        - name: woodpecker-data
          mountPath: /var/lib/woodpecker
      volumes:
      - name: woodpecker-data
        persistentVolumeClaim:
          claimName: woodpecker-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: woodpecker-server
  namespace: woodpecker
spec:
  selector:
    app: woodpecker-server
  ports:
  - name: ui
    port: 8000
    targetPort: 8000
  - name: rpc
    port: 9000
    targetPort: 9000
  type: LoadBalancer

Deploy:

# Create secrets
kubectl create secret generic woodpecker-secrets \
  -n woodpecker \
  --from-literal=admin-password=YOUR_PASSWORD \
  --from-literal=rpc-secret=$(head -c 32 /dev/urandom | base64) \
  --from-literal=github-client-id=YOUR_CLIENT_ID \
  --from-literal=github-client-secret=YOUR_CLIENT_SECRET

# Apply deployment
kubectl apply -f woodpecker-deployment.yaml

# Check status
kubectl get pods -n woodpecker
kubectl port-forward -n woodpecker svc/woodpecker-server 8000:8000

---

GitHub Integration Setup

Step 1: Create GitHub OAuth App

1. Go to GitHub → Settings → Developer settings → OAuth Apps
2. Click "New OAuth App"
3. Fill in:
   • Application name: VAPORA Woodpecker
   • Homepage URL: https://woodpecker.your-domain.com
   • Authorization callback URL: https://woodpecker.your-domain.com/authorize
4. Copy Client ID and Client Secret

Step 2: Configure Woodpecker

For Docker Compose:

# Update docker-compose.yml environment variables
WOODPECKER_GITHUB_CLIENT_ID=your_client_id_here
WOODPECKER_GITHUB_CLIENT_SECRET=your_client_secret_here
WOODPECKER_RPC_SECRET=$(head -c 32 /dev/urandom | base64)

For Kubernetes:

kubectl patch secret woodpecker-secrets -n woodpecker \
  --type=merge \
  -p '{"data":{"github-client-id":"'$(echo -n YOUR_CLIENT_ID | base64)'","github-client-secret":"'$(echo -n YOUR_CLIENT_SECRET | base64)'"}}'

Step 3: Repository Setup

1. Access Woodpecker UI: http://localhost:8000 (or your domain)
2. Login with admin credentials
3. Go to Admin → Repositories
4. Authorize your VAPORA repository
5. Enable webhooks by visiting http://your-github.com/settings/hooks

---

Secret Management

Adding Secrets

Via Woodpecker UI

1. Go to repository → Settings → Secrets
2. Click "Add secret"
3. Name: SECRET_NAME
4. Value: Your secret value
5. Save

Via CLI

# Install woodpecker-cli
go install github.com/woodpeckerci/woodpecker/cmd/woodpecker-cli@latest

# Login
woodpecker-cli login -s http://woodpecker-server:8000 \
  -u admin \
  -p admin_password

# Add secret
woodpecker-cli secret add \
  -r owner/repo \
  -n KUBE_CONFIG_STAGING \
  -v "$(cat ~/.kube/config | base64)"

Required VAPORA Secrets

# Kubernetes kubeconfigs (base64 encoded)
KUBE_CONFIG_STAGING         # Staging cluster kubeconfig
KUBE_CONFIG_PRODUCTION      # Production cluster kubeconfig

# Optional: Slack notifications
SLACK_WEBHOOK               # General notifications webhook
SLACK_WEBHOOK_ALERTS        # Critical alerts webhook

# Optional: Docker registry
DOCKER_USERNAME             # Docker Hub username
DOCKER_PASSWORD             # Docker Hub access token

Encoding Kubeconfig

# Get kubeconfig and encode as base64
cat ~/.kube/config | base64 > kube_config_base64.txt

# Use the output in Woodpecker secret UI
cat kube_config_base64.txt

# Verify locally before adding to Woodpecker
echo "$(cat kube_config_base64.txt)" | base64 -d | kubectl cluster-info

---

Pipeline Triggers

Automatic Triggers

Pipelines trigger automatically when:

# On push to main/develop branches (if provisioning files change)
trigger:
  event: [push]
  branch: [main, develop]
  paths:
    include:
      - provisioning/schemas/**
      - provisioning/scripts/**

# On pull requests
trigger:
  event: [pull_request]
  branch: [main, develop]

Manual Triggers (Promotions)

Manually trigger from UI:

1. Go to repository → Active builds
2. Find a completed build
3. Click "Promote"
4. Select pipeline: deploy-docker, deploy-kubernetes, etc.
5. Set deployment parameters:
   • Mode: solo, multiuser, enterprise
   • Environment: staging, production
   • Dry-run: true/false
6. Click "Promote"

Scheduled Triggers (Cron)

Health check pipeline runs on schedule:

trigger:
  cron:
    - "*/15 * * * *"    # Every 15 minutes
    - "0 */6 * * *"     # Every 6 hours

---

Deployment Workflows

Workflow 1: Local Development

Developer pushes to feature branch
    ↓
[Validate & Build] runs automatically
    ↓
Review artifacts in workspace
    ↓
[Deploy to Docker] manually for local testing
    ↓
Test with docker compose
    ↓
Create PR

Workflow 2: Staging Deployment

Merge PR to develop
    ↓
[Validate & Build] runs automatically
    ↓
Download artifacts from workspace
    ↓
Manually run [Deploy to Kubernetes]
    - Mode: multiuser
    - Environment: staging
    - Dry-run: true
    ↓
Review dry-run output
    ↓
Run again with dry-run: false
    ↓
[Health Check] verifies deployment
    ↓
Staging live

Workflow 3: Production Deployment

Code review approved
    ↓
Merge to main
    ↓
[Validate & Build] runs automatically
    ↓
Manually run [Deploy to Kubernetes]
    - Mode: enterprise
    - Environment: production
    - Dry-run: true
    ↓
Carefully review changes
    ↓
Run with dry-run: false
    ↓
[Health Check] monitoring (auto every 6 hours)
    ↓
Production deployment complete

Workflow 4: Emergency Rollback

Production issue detected
    ↓
[Health Check] alerts in Slack
    ↓
Manually run [Rollback Deployment]
    - Environment: production
    ↓
Services restored
    ↓
Investigate root cause

---

Configuration Environment Variables

For validate-and-build.yml

ARTIFACTS_DIR=provisioning/artifacts    # Output directory for configs
LOG_DIR=provisioning/logs               # Output directory for logs

For deploy-docker.yml

ARTIFACTS_DIR=provisioning/artifacts
LOGS_DIR=provisioning/logs

For deploy-kubernetes.yml

ARTIFACTS_DIR=provisioning/artifacts
LOGS_DIR=provisioning/logs
VAPORA_NAMESPACE=vapora                 # Kubernetes namespace

For health-check.yml

LOGS_DIR=provisioning/logs
VAPORA_NAMESPACE=vapora

---

Monitoring & Logs

Via Woodpecker UI

1. Go to repository → Active/Previous builds
2. Click a build to see full pipeline execution
3. Click a stage to see detailed logs
4. Download logs or artifacts

Via CLI

# List recent builds
woodpecker-cli build list -r owner/repo

# View build details
woodpecker-cli build view -r owner/repo -b <BUILD_NUMBER>

# Watch build in real-time
woodpecker-cli build watch -r owner/repo -b <BUILD_NUMBER>

# Get build logs
woodpecker-cli build logs -r owner/repo -b <BUILD_NUMBER>

Logs Location

All logs stored in workspace:

provisioning/logs/
├── validate-solo.log
├── validate-multiuser.log
├── validate-enterprise.log
├── build.log
├── docker/
│   ├── backend.log
│   ├── frontend.log
│   └── all-services.log
├── kubernetes/
│   ├── backend.log
│   ├── agents.log
│   ├── llm-router.log
│   └── events.log
└── health-checks/
    ├── docker-endpoints.log
    ├── k8s-deployments.log
    └── HEALTH_REPORT.md

---

Slack Integration

Setup Webhook

1. Go to Slack workspace → Apps → Custom Integrations
2. Create Incoming Webhook
3. Select channel: #deployments
4. Copy Webhook URL
5. Add to Woodpecker secret: SLACK_WEBHOOK

Slack Messages

Build Success:

✅ VAPORA Artifact Build Complete
Artifacts ready for deployment

Docker Deploy Success:

✅ VAPORA Docker deployment successful!
Mode: multiuser | Environment: staging

Kubernetes Deploy Success:

✅ VAPORA Kubernetes deployment successful!
Mode: enterprise | Environment: production

Health Check Alert:

❌ VAPORA Health Check Failed
Target: kubernetes

Rollback Alert:

🔙 VAPORA Rollback Executed
Environment: production
Verify service health immediately

---

Troubleshooting

Pipeline Not Triggering

Problem: Push doesn't trigger validate-and-build

Solution:

1. Check repository is authorized in Woodpecker
2. Verify webhook exists in GitHub settings
3. Check file paths in trigger.paths.include match your changes
4. Enable debug logging: WOODPECKER_LOG_LEVEL=debug

Secret Not Found

Problem: Secret not found error in logs

Solution:

1. Verify secret exists in repository settings
2. Check exact spelling (case-sensitive)
3. Ensure secret value is not empty
4. Test secret value locally before adding

Kubeconfig Decode Error

Problem: base64: invalid input during kubectl setup

Solution:

# Test locally first
echo "$(cat kube_config_base64.txt)" | base64 -d | kubectl cluster-info

# If it fails, re-encode
cat ~/.kube/config | base64 | pbcopy  # macOS

# Update secret in Woodpecker UI

Docker Connection Failed

Problem: Cannot connect to Docker daemon in deploy-docker stage

Solution:

1. Ensure Docker socket mounted in agent: -v /var/run/docker.sock:/var/run/docker.sock
2. Verify Docker daemon running: docker ps
3. Check volume permissions: sudo chmod 666 /var/run/docker.sock

Deployment Hangs

Problem: Pipeline stage times out waiting for rollout

Solution:

1. Check pod logs: kubectl logs -n vapora <pod>
2. Describe pod: kubectl describe pod -n vapora <pod>
3. Increase timeout in pipeline stage
4. Check resource requests/limits
5. Verify cluster has sufficient resources

Workspace Persistence Issues

Problem: Files from one stage not available in next stage

Solution:

1. Create file in correct location (workspace root or subdirectory)
2. Use absolute paths: ${LOGS_DIR}/output.log
3. Check artifact uploads in "publish" stages
4. Verify docker volumes: docker volume ls

---

Advanced Configuration

Multi-Agent Setup

For distributed build execution:

# Agent 1 (Docker builds)
environment:
  - WOODPECKER_FILTER_LABELS=type:docker

# Agent 2 (Kubernetes operations)
environment:
  - WOODPECKER_FILTER_LABELS=type:kubernetes

# Agent 3 (Health checks)
environment:
  - WOODPECKER_FILTER_LABELS=type:monitoring

Pipeline Concurrency Control

Limit concurrent executions:

concurrency:
  limit: 2              # Max 2 concurrent builds
  timeout_minutes: 60   # Timeout after 60 minutes

Conditional Stage Execution

Run stage only if conditions met:

when:
  evaluate: 'return build.Deploy_Environment == "production"'

---

Comparison: Woodpecker vs GitHub Actions

Feature	Woodpecker	GitHub Actions
Hosting	Self-hosted	GitHub-hosted
YAML Format	Similar	Familiar
Manual Dispatch	Promotion UI	workflow_dispatch
Scheduled Workflows	Cron syntax	schedule syntax
Artifact Storage	Workspace persistence	upload-artifact action
PR Comments	Limited	✓ Native
Slack Integration	Via webhooks	Actions
Secret Management	Built-in UI	Built-in
Free for Public	Self-hosted cost	✓ Free
Concurrency Control	✓ Advanced	✓ Concurrency groups
Deployment Safety	Dry-run support	Deployment protection

When to Use Woodpecker

• ✓ You want full control over CI/CD infrastructure
• ✓ You need to run on-premise for compliance
• ✓ You prefer self-hosted solutions
• ✓ You have multiple repositories needing unified CI/CD
• ✓ You want to avoid vendor lock-in

When to Use GitHub Actions

• ✓ You want GitHub-hosted runners (no infrastructure)
• ✓ You prefer tight GitHub integration
• ✓ You want PR comments and GitHub UI integration
• ✓ You're already using GitHub workflow syntax

---

First Deployment with Woodpecker

Step 1: Enable Woodpecker for Repository

1. Access Woodpecker UI
2. Click "Administration" → "Repositories"
3. Find VAPORA repository
4. Click to enable
5. Grant webhook access

Step 2: Create Test Branch

git checkout -b test/woodpecker-setup
echo "# Woodpecker Test" >> README.md
git add README.md
git commit -m "test: trigger Woodpecker"
git push origin test/woodpecker-setup

Step 3: Monitor Pipeline

1. Go to Woodpecker → repository
2. See "Validate & Build" trigger automatically
3. Monitor pipeline execution
4. Check logs for each stage

Step 4: Download Artifacts

1. In completed build, find "Files" section
2. Access workspace artifacts:
   • provisioning/artifacts/ - Generated configs
   • provisioning/logs/ - Pipeline logs

Step 5: Test Docker Deployment

1. Download artifacts
2. Go to Woodpecker → repository
3. Click "Promote" on validated build
4. Select "deploy-docker"
5. Set:
   • Mode: multiuser
   • Environment: staging
   • Dry-run: true
6. Monitor deployment

Step 6: Create Pull Request

git push origin test/woodpecker-setup
# Create PR on GitHub

---

Security Best Practices

✅ Do:

• Use environment-specific kubeconfigs
• Rotate secrets regularly
• Run health checks after deployments
• Enable dry-run by default
• Keep logs for audit trail
• Use RBAC in Kubernetes
• Monitor Slack alerts
• Test on staging first

❌ Don't:

• Commit secrets to repository
• Deploy directly to production without testing
• Disable dry-run validation
• Skip health checks
• Use same credentials for all environments
• Share Woodpecker admin credentials
• Keep old pipelines around
• Ignore Slack alerts

---

Support & Resources

• Woodpecker Docs: https://woodpecker-ci.org/docs/intro
• VAPORA Docs: See ./../docs/ directory
• GitHub Actions Guide: ./../.github/GITHUB_ACTIONS_GUIDE.md
• Nushell Scripts: provisioning/scripts/*.nu

---

Files Created

.woodpecker/
├── validate-and-build.yml       (410 lines)
├── deploy-docker.yml            (340 lines)
├── deploy-kubernetes.yml        (380 lines)
├── health-check.yml             (290 lines)
├── rollback.yml                 (330 lines)
└── SETUP.md                     (This file)

Total: 5 pipelines + comprehensive documentation

---

Next Steps

1. ✅ Install and configure Woodpecker server
2. → Integrate with GitHub repository
3. → Add secrets for Kubernetes kubeconfigs
4. → Configure Slack webhooks (optional)
5. → Run first validation pipeline
6. → Test Docker deployment
7. → Test Kubernetes deployment
8. → Configure health checks
9. → Document team runbooks
10. → Deploy to production

---

Generated: 2026-01-12 Status: Production-ready Pipelines: 5 (validate-and-build, deploy-docker, deploy-kubernetes, health-check, rollback) Documentation: Complete