Vapora/provisioning/.woodpecker/WOODPECKER_GUIDE.md

Woodpecker CI/CD Guide for VAPORA Provisioning

Complete reference for understanding, running, and troubleshooting VAPORA's Woodpecker CI/CD pipelines.

---

Overview

VAPORA uses five integrated Woodpecker CI/CD pipelines for complete deployment automation. These pipelines are self-hosted alternatives to GitHub Actions, providing full control over infrastructure and execution environment.

Pipeline Architecture

Push to Repository
    ↓
[Validate & Build] - Generates artifacts
    ↓
    ├── Manual Promotion → [Deploy to Docker]
    │                          ↓
    │                    Health checks → Services running locally
    │
    └── Manual Promotion → [Deploy to Kubernetes] with dry-run
                                ↓
                        Review changes
                                ↓
                        Actual deployment
                                ↓
                        [Health Check] (automatic every 15min/6hr)
                                ↓
                        [Rollback] if issues detected

---

Quick Reference

Pipeline Files

.woodpecker/
├── validate-and-build.yml       # Validates configs, generates artifacts
├── deploy-docker.yml            # Deploys to Docker Compose
├── deploy-kubernetes.yml        # Deploys to Kubernetes
├── health-check.yml             # Continuous monitoring (scheduled)
├── rollback.yml                 # Safe deployment rollback
├── SETUP.md                     # Installation and configuration
└── WOODPECKER_GUIDE.md          # This file

Pipeline Triggers

Pipeline	Trigger	Branch	Manual
validate-and-build	Push, PR	main/develop	Yes
deploy-docker	Manual promotion	main/develop	Yes
deploy-kubernetes	Manual promotion	main/develop	Yes
health-check	Cron (15min, 6hr)	Any	Yes
rollback	Manual promotion	main/develop	Yes

Environment Variables

All pipelines use:

ARTIFACTS_DIR=provisioning/artifacts    # Generated configs
LOG_DIR=provisioning/logs               # Pipeline logs
VAPORA_NAMESPACE=vapora                 # K8s namespace

---

Workflows in Detail

1. Validate & Build (validate-and-build.yml)

Purpose: Validate all configurations and generate deployment artifacts

Triggers:

• Push to main or develop branches (if provisioning files change)
• Manual promotion from Woodpecker UI
• Pull requests affecting provisioning

Execution Flow:

setup
  └─ prepare: Create directories, display info
      ↓
install_dependencies
  └─ install_tools: Install Rust, Nushell, Nickel, jinja2, yq
      ↓
validate_solo/multiuser/enterprise (parallel)
  └─ validate_*: Run mode-specific validation
      ↓
build_artifacts
  ├─ install_tools: Reinstall tools (cached layer)
  ├─ build_artifacts: Run CI pipeline to generate outputs
  ├─ verify_artifacts: Validate JSON, YAML, TOML formats
  └─ generate_manifest: Create README documenting outputs
      ↓
publish
  └─ publish_artifacts: Display artifact summary

Duration: ~5 minutes

Outputs:

provisioning/artifacts/
├── config-solo.json
├── config-multiuser.json
├── config-enterprise.json
├── vapora-solo.toml/yaml
├── vapora-multiuser.toml/yaml
├── vapora-enterprise.toml/yaml
├── configmap.yaml
├── deployment.yaml
├── docker-compose.yml
└── README.md

Usage:

# Automatic (on push)
git commit -m "Update provisioning config"
git push origin main

# Manual (from Woodpecker UI)
1. Go to repository → Latest build
2. Click "Promote" button
3. Select "validate-and-build" pipeline
4. Click "Promote"

Expected Output:

✓ Solo configuration validated
✓ Multiuser configuration validated
✓ Enterprise configuration validated
✓ JSON outputs validated
✓ YAML outputs validated
✓ TOML outputs validated
✓ Manifests generated
✓ Artifacts ready for deployment

---

2. Deploy to Docker (deploy-docker.yml)

Purpose: Deploy VAPORA to Docker Compose for local/staging testing

Triggers:

• Manual promotion from completed validate-and-build build
• Manual promotion from Woodpecker UI

Execution Flow:

setup
  └─ prepare: Display deployment info
      ↓
install_dependencies
  └─ install_tools: Install tools + Docker
      ↓
download_artifacts
  └─ fetch_latest_artifacts: Get configs from workspace
      ↓
validate_docker_config
  └─ validate_compose: Validate docker-compose.yml format
      ↓
deploy_docker_compose
  ├─ pull_images: Download container images
  └─ compose_up: Start services with docker compose
      ↓
health_checks
  ├─ verify_services: Check HTTP endpoints
  └─ collect_logs: Gather service logs
      ↓
verify_endpoints
  └─ test_endpoints: Test API calls to running services
      ↓
generate_report
  └─ create_deployment_report: Generate deployment summary
      ↓
publish
  └─ publish_results & notify_slack

Duration: ~3 minutes

Service Endpoints (after deployment):

- Backend API: http://localhost:8001
- Frontend UI: http://localhost:3000
- Agents: http://localhost:8002
- LLM Router: http://localhost:8003
- SurrealDB: http://localhost:8000
- Health: http://localhost:8001/health

Usage:

# From Woodpecker UI (after validate-and-build)
1. Go to completed validate-and-build build
2. Click "Promote"
3. Select "deploy-docker"
4. Click "Promote"

# Monitor via Woodpecker UI
1. Go to repository → Active builds
2. Watch deploy-docker build progress
3. Check each stage for logs

Local Testing:

# Download artifacts from Woodpecker workspace
# Extract provisioning/artifacts/docker-compose.yml

# Start services
docker compose -f docker-compose.yml up -d

# Check health
curl http://localhost:8001/health

# View logs
docker compose logs -f backend

# Stop services
docker compose down

Verification:

• ✓ Backend responds at port 8001
• ✓ Frontend accessible at port 3000
• ✓ Agents running at port 8002
• ✓ LLM Router at port 8003
• ✓ SurrealDB at port 8000
• ✓ Health endpoint returns 200 OK

---

3. Deploy to Kubernetes (deploy-kubernetes.yml)

Purpose: Deploy VAPORA to Kubernetes with dry-run validation

Triggers:

• Manual promotion from completed validate-and-build
• Manual promotion from Woodpecker UI

Execution Flow:

setup
  └─ prepare: Display deployment info
      ↓
install_dependencies
  └─ install_tools: Install kubectl, tools
      ↓
configure_kubernetes
  ├─ setup_kubeconfig_staging/production: Decode kubeconfig
  └─ verify_cluster: Test cluster access
      ↓
validate_manifests
  ├─ validate_kubernetes_manifests: Check manifest validity
  └─ dry_run_validation: Kubernetes dry-run check
      ↓
create_namespace
  ├─ ensure_namespace: Create vapora namespace
  └─ setup_rbac: Configure service accounts
      ↓
deploy_configmap
  └─ apply_configmap: Deploy configuration
      ↓
deploy_services (with monitoring)
  ├─ apply_deployments: Deploy all three services
  ├─ monitor_rollout_backend: Wait for backend ready
  ├─ monitor_rollout_agents: Wait for agents ready
  └─ monitor_rollout_llm_router: Wait for router ready
      ↓
verify_deployment
  ├─ check_pods: Verify pod status
  ├─ check_services: Verify service endpoints
  ├─ collect_logs: Gather deployment logs
  └─ annotate_deployment: Add metadata
      ↓
generate_report
  └─ create_deployment_report: Generate summary
      ↓
publish
  └─ publish_results & notify_slack

Duration: ~5-10 minutes (includes rollout waits)

Deployment Options:

# Via Woodpecker UI Promotion
1. Select environment: staging or production
2. Select deployment mode: solo, multiuser, enterprise
3. Set dry_run: true (first), then false (actual)
4. Set rollout_timeout: 300 (seconds)

Dry-Run Usage (Recommended):

# Step 1: Promote with dry-run enabled
Mode: enterprise
Environment: staging
Dry-run: true
Rollout timeout: 300

# Step 2: Review dry-run output in logs
# Check proposed changes to deployments

# Step 3: If satisfied, promote again with dry-run disabled
Dry-run: false

# Step 4: Monitor rollout
# Watch rollout status and pod health

Verification Commands (after deployment):

# Check deployments
kubectl get deployments -n vapora

# Check pods
kubectl get pods -n vapora -o wide

# Check services
kubectl get services -n vapora

# View logs
kubectl logs -f deployment/vapora-backend -n vapora

# Check events
kubectl get events -n vapora --sort-by='.lastTimestamp'

# Port forward for local testing
kubectl port-forward -n vapora svc/vapora-backend 8001:8001
curl http://localhost:8001/health

# Check rollout history
kubectl rollout history deployment/vapora-backend -n vapora

Deployment Modes:

Mode	Replicas	Resources	Use Case
solo	1	Minimal	Development, testing
multiuser	2	Standard	Team/staging environments
enterprise	3	Optimized	Production with HA

---

4. Health Check & Monitoring (health-check.yml)

Purpose: Continuous health monitoring across Docker and Kubernetes

Triggers:

• Schedule: Every 15 minutes (quick check)
• Schedule: Every 6 hours (comprehensive diagnostics)
• Manual promotion from Woodpecker UI

Execution Flow:

setup
  └─ prepare: Display check info
      ↓
install_dependencies
  └─ install_tools: Install kubectl, Docker tools
      ↓
configure_kubernetes
  └─ setup_kubeconfig: Configure cluster access
      ↓
health_check_docker (if available)
  ├─ check_docker_containers: Container status
  ├─ check_docker_endpoints: HTTP health checks
  └─ collect_docker_diagnostics: System resource info
      ↓
health_check_kubernetes
  ├─ check_k8s_deployments: Deployment replica status
  ├─ check_k8s_services: Service endpoints
  ├─ check_k8s_events: Recent cluster events
  └─ collect_pod_logs: Application logs
      ↓
analyze_health
  ├─ generate_health_report: Create summary
  └─ check_health_status: Determine overall status
      ↓
publish
  └─ publish_reports & notify_slack

Duration: ~5 minutes

Checked Resources (Docker):

• Container status (Up/Down)
• HTTP endpoints (8001, 8002, 8003, 3000, 8000)
• Network connectivity
• Resource usage

Checked Resources (Kubernetes):

• Deployment replica status
• Pod readiness conditions
• Service availability
• ConfigMap data
• Recent cluster events
• Pod logs (last 100 lines)

Reports Generated:

provisioning/logs/health-checks/
├── docker-containers.log
├── docker-endpoints.log
├── docker-diagnostics.log
├── k8s-deployments.log
├── k8s-services.log
├── k8s-events.log
├── k8s-diagnostics.log
├── pods/
│   ├── backend.log
│   ├── agents.log
│   └── llm-router.log
└── HEALTH_REPORT.md

Manual Trigger:

# From Woodpecker UI
1. Click "Promote" on any completed build
2. Select "health-check" pipeline
3. Click "Promote"

# View results
1. Wait for build to complete
2. Check "Artifacts" for health reports
3. Review pod logs for errors

Alert Conditions:

• ❌ Pod in CrashLoopBackOff state
• ❌ Endpoint not responding
• ❌ Service not running
• ❌ Recent error events in cluster

---

5. Rollback Deployment (rollback.yml)

Purpose: Safe deployment rollback with pre-checks and verification

Triggers:

• Manual promotion only (safety feature)

Execution Flow:

pre_rollback_checks
  └─ verify_environment: Confirm rollback parameters
      ↓
install_dependencies
  └─ install_tools: Install kubectl, tools
      ↓
configure_kubernetes
  └─ setup_kubeconfig: Configure target cluster
      ↓
store_deployment_history
  └─ snapshot_current_state: Backup current deployments
      ↓
kubernetes_rollback
  ├─ perform_rollback: Execute kubectl rollout undo
  ├─ verify_rollback: Check rollback status
  └─ check_pod_health: Verify pod readiness
      ↓
docker_rollback_guide
  ├─ generate_docker_guide: Create manual instructions
  └─ store_docker_state: Backup docker-compose.yml
      ↓
post_rollback_verification
  └─ generate_rollback_report: Create summary
      ↓
publish
  └─ publish_artifacts & notify_slack

Duration: ~3-5 minutes

Rollback Parameters:

Target:
  - kubernetes    # Automatic K8s rollback
  - docker        # Guided Docker rollback

Environment:
  - staging       # Staging cluster
  - production    # Production cluster

Deployment:
  - all           # Rollback all services
  - backend       # Rollback specific service
  - agents
  - llm-router

Revision:
  - 0             # Previous revision (default)
  - 1, 2, 3...    # Specific revision number

Usage:

# Kubernetes Rollback (Automatic)
1. Go to Woodpecker UI
2. Click "Promote"
3. Select "rollback" pipeline
4. Set:
   - Target: kubernetes
   - Environment: production
   - Deployment: all
   - Revision: 0 (previous)
5. Click "Promote"
6. Monitor rollout status

# Docker Rollback (Manual Guide)
1. Follow generated DOCKER_ROLLBACK_GUIDE.md
2. Execute git/docker commands as instructed
3. Verify services running with health checks

Verification After Rollback:

# Kubernetes
kubectl get pods -n vapora
kubectl logs -f deployment/vapora-backend -n vapora
kubectl rollout history deployment/vapora-backend -n vapora

# Docker
docker compose ps
docker compose logs -f
curl http://localhost:8001/health

Rollback History:

# View deployment revisions
kubectl rollout history deployment/vapora-backend -n vapora

# Output example:
REVISION  CHANGE-CAUSE
1         <initial>
2         Deployment rolled out
3         Deployment rolled out

# Find the working revision and use that number

---

Integration Patterns

Pattern 1: Automatic Validation on Every Push

Developer pushes feature branch
    ↓
Git webhook triggers Woodpecker
    ↓
[Validate & Build] runs automatically
    ↓
Artifacts generated in workspace
    ↓
Build completes (visible in Woodpecker UI)

Pattern 2: Staging Deployment

1. Merge PR to develop branch
    ↓
2. [Validate & Build] runs automatically
    ↓
3. In Woodpecker UI → Promote to deploy-kubernetes
    - Mode: multiuser
    - Environment: staging
    - Dry-run: true
    ↓
4. Review dry-run output
    ↓
5. Promote again with dry-run: false
    ↓
6. [Health Check] runs (automatic in 15min)
    ↓
7. Staging live

Pattern 3: Production Deployment

1. Code review approved
    ↓
2. Merge PR to main branch
    ↓
3. [Validate & Build] runs automatically
    ↓
4. In Woodpecker UI → Promote to deploy-kubernetes
    - Mode: enterprise
    - Environment: production
    - Dry-run: true
    ↓
5. **CAREFULLY** review all changes
    ↓
6. Promote again with dry-run: false
    ↓
7. [Health Check] monitoring starts (every 6 hours)
    ↓
8. Production deployment complete

Pattern 4: Emergency Rollback

1. Production issue detected
    ↓
2. [Health Check] alerts in Slack (if configured)
    ↓
3. In Woodpecker UI → Promote to rollback
    - Target: kubernetes
    - Environment: production
    - Deployment: all
    - Revision: 0 (previous)
    ↓
4. Monitor rollout status
    ↓
5. Services restored
    ↓
6. Investigate root cause
    ↓
7. Plan corrected deployment

---

Configuration & Secrets

Secrets Required

# Kubernetes kubeconfigs (base64 encoded)
KUBE_CONFIG_STAGING         # For staging deployments

KUBE_CONFIG_PRODUCTION      # For production deployments

# Optional: Slack notifications
SLACK_WEBHOOK               # General notifications

SLACK_WEBHOOK_ALERTS        # Critical alerts only

Adding Secrets in Woodpecker UI

1. Go to repository → Settings → Secrets
2. Click "Add secret"
3. Enter name: KUBE_CONFIG_STAGING
4. Paste base64-encoded kubeconfig value
5. Click "Add"
6. Repeat for other secrets

Encoding Kubeconfig

# Get kubeconfig and encode
cat ~/.kube/config | base64

# Verify locally before adding
echo "base64_value_here" | base64 -d | kubectl cluster-info

Environment Variables Available in Pipelines

# Woodpecker System Variables
CI_BUILD_LINK              # Link to build in UI
CI_COMMIT_SHA              # Full commit hash
CI_COMMIT_BRANCH           # Branch name
CI_COMMIT_AUTHOR           # Commit author

# Pipeline-Defined Variables
ARTIFACTS_DIR              # provisioning/artifacts
LOG_DIR                    # provisioning/logs
VAPORA_NAMESPACE           # vapora (K8s namespace)

---

Monitoring & Troubleshooting

Checking Build Status

Via Woodpecker UI:

1. Go to repository page
2. See "Active builds" and "Previous builds"
3. Click a build to see pipeline execution
4. Click a stage to see detailed logs

Via Terminal:

# If using woodpecker-cli
woodpecker-cli build list -r owner/repo

# View specific build
woodpecker-cli build view -r owner/repo -b <BUILD_NUMBER>

# Watch build live
woodpecker-cli build watch -r owner/repo -b <BUILD_NUMBER>

Accessing Logs

From Woodpecker UI:

1. Click build → see stages
2. Click stage → see full logs
3. Scroll through logs or search

From Workspace:

# Logs persisted in workspace (visible as artifacts)
provisioning/logs/
├── validate-solo.log
├── build.log
├── docker/
├── kubernetes/
└── health-checks/

Common Issues

Issue 1: "Pipeline not triggering"

Symptoms: Push doesn't start validate-and-build

Diagnose:

1. Check webhook in GitHub settings
2. Verify repository authorized in Woodpecker
3. Check file paths match trigger.paths.include
4. Review Woodpecker logs: WOODPECKER_LOG_LEVEL=debug

Fix:

# Manually re-authorize in Woodpecker UI
# Settings → Repositories → VAPORA → Activate

# Test webhook
curl -X POST https://your-woodpecker/hook \
  -H "X-GitHub-Event: push" \
  -d '{"ref":"refs/heads/main"}'

Issue 2: "Secret not found"

Symptoms: Stage fails with "secret not found"

Diagnose:

1. Go to repository → Settings → Secrets
2. Verify secret exists and name matches exactly
3. Check secret value is not empty

Fix:

# Re-add secret in UI
# Make sure spelling is exact (case-sensitive)

# Test secret locally
echo "secret_value" | base64 -d

Issue 3: "Kubeconfig decode error"

Symptoms: base64: invalid input during kubectl setup

Diagnose:

1. Check if base64 value is valid
2. Test decode locally

Fix:

# Test locally first
echo "kube_config_base64_value" | base64 -d | kubectl cluster-info

# If invalid, re-encode
cat ~/.kube/config | base64

# Add to Woodpecker secret

Issue 4: "Deployment timeout"

Symptoms: Waiting for pod readiness timeout

Diagnose:

1. Check pod logs: kubectl logs -n vapora <pod>
2. Check pod events: kubectl describe pod -n vapora <pod>
3. Check resource constraints

Fix:

# Increase timeout in deploy-kubernetes.yml
rollout_timeout: 600  # 10 minutes

# Check pod logs for errors
kubectl logs -n vapora deployment/vapora-backend --tail=50

# Check resource availability
kubectl top nodes
kubectl top pods -n vapora

Issue 5: "Docker connection failed"

Symptoms: Cannot connect to Docker daemon in deploy-docker

Diagnose:

1. Check Docker socket mounted
2. Verify Docker daemon running

Fix:

# Verify socket mounted in agent
docker exec woodpecker-agent ls -la /var/run/docker.sock

# Test Docker access
docker ps

# Restart Docker if needed
sudo systemctl restart docker

---

Performance Tuning

Parallel Validation

Validation stages run in parallel (solo, multiuser, enterprise):

validate_solo:
  depends_on: [install_dependencies]
  # Runs while multiuser and enterprise also run

validate_multiuser:
  depends_on: [install_dependencies]
  # All three in parallel, not sequential

Impact: Reduces validation time by ~3x

Caching

Tool installation caches automatically:

# First run: downloads and installs
- cargo install nu --locked

# Subsequent runs: uses cached Docker layer

Workspace Cleanup

Between builds, workspace persists. To reclaim space:

1. Delete old workspace volumes
2. Configure retention policy in Woodpecker
3. Use docker volume prune carefully

---

Security Considerations

Secret Management

✅ Best Practices:

• Store all sensitive values as secrets
• Use environment-specific secrets (staging vs prod)
• Rotate secrets quarterly
• Never log secret values
• Use unique kubeconfigs per environment

❌ Anti-Patterns:

• Hardcoding secrets in YAML
• Using same secret for all environments
• Storing secrets in git history
• Logging secret values during debug

RBAC & Access Control

# Kubernetes: Limit service account permissions
kubectl create serviceaccount vapora-deployer -n vapora

# Assign minimal necessary permissions
kubectl create role vapora-deployer \
  --verb=get,list,watch,create,update,patch \
  --resource=deployments,configmaps,pods

# Bind role to service account
kubectl create rolebinding vapora-deployer \
  --role=vapora-deployer \
  --serviceaccount=vapora:vapora-deployer

Pipeline Execution

• Pipelines run in isolated Docker containers
• Limited to workspace directory
• No access to host filesystem (unless mounted)
• Network isolation between stages possible

---

Advanced Topics

Custom Pipeline Parameters

Use Woodpecker promotions to pass parameters:

deploy-kubernetes.yml:
  environment:
    - Deploy_Environment    # Read from promotion UI
    - Rollback_Target
    - Rollback_Revision

Multi-Agent Setup

Deploy multiple agents for distributed execution:

# Agent 1: Docker builds
- WOODPECKER_FILTER_LABELS=type:docker

# Agent 2: Kubernetes operations
- WOODPECKER_FILTER_LABELS=type:kubernetes

# In pipeline, require specific agent
labels:
  - type:kubernetes

Conditional Execution

Skip stages based on conditions:

deploy-production:
  when:
    evaluate: 'return build.Deploy_Environment == "production"'
    # Only runs if Deploy_Environment is production

---

Comparison with GitHub Actions

Feature Comparison

Feature	Woodpecker	GitHub Actions
Hosting	Self-hosted	GitHub-hosted
Infrastructure Control	✓ Full control	Limited
YAML Syntax	Similar but different	GitHub-specific
PR Integration	Limited	Native
Manual Dispatch	Via promotions	workflow_dispatch
Secrets Management	Built-in UI	GitHub secrets
Artifact Storage	Workspace + volumes	Actions API
Cost (self-hosted)	Infrastructure only	GitHub minutes quota
Dry-run Support	✓ First-class	Manual pattern

When to Choose Woodpecker

✓ Want to self-host CI/CD ✓ Need full infrastructure control ✓ Prefer to avoid vendor lock-in ✓ Have compliance/data residency requirements ✓ Want to run multiple repos unified CI/CD

When to Choose GitHub Actions

✓ Want GitHub-hosted runners ✓ Prefer tight GitHub integration ✓ Want PR comments and status checks ✓ Don't want infrastructure overhead

---

Support & Resources

• Woodpecker Documentation: https://woodpecker-ci.org/docs
• VAPORA Repository: https://github.com/your-org/vapora
• GitHub Actions Guide: ./../.github/GITHUB_ACTIONS_GUIDE.md
• Nushell Scripts: provisioning/scripts/*.nu

---

Quick Start Checklist

• Install Woodpecker server
• Configure GitHub OAuth app
• Authorize VAPORA repository
• Add KUBE_CONFIG_STAGING secret
• Add KUBE_CONFIG_PRODUCTION secret
• Test: Push to feature branch
• Verify: validate-and-build completes
• Test: Promote to deploy-docker
• Test: Promote to deploy-kubernetes (dry-run)
• Configure: Slack webhooks (optional)
• Document: Team runbooks

---

Generated: 2026-01-12 Status: Production-ready Pipelines: 5 comprehensive workflows Documentation: Complete reference guide