provisioning/docs/src/testing/taskserv-validation-guide.md

Taskserv Validation and Testing Guide

Version: 1.0.0 Date: 2025-10-06 Status: Production Ready

---

Overview

The taskserv validation and testing system provides comprehensive evaluation of infrastructure services before deployment, reducing errors and increasing confidence in deployments.

Validation Levels

1. Static Validation

Validates configuration files, templates, and scripts without requiring infrastructure access.

What it checks:

• KCL schema syntax and semantics
• Jinja2 template syntax
• Shell script syntax (with shellcheck if available)
• File structure and naming conventions

Command:

provisioning taskserv validate kubernetes --level static

2. Dependency Validation

Checks taskserv dependencies, conflicts, and requirements.

What it checks:

• Required dependencies are available
• Optional dependencies status
• Conflicting taskservs
• Resource requirements (memory, CPU, disk)
• Health check configuration

Command:

provisioning taskserv validate kubernetes --level dependencies

Check against infrastructure:

provisioning taskserv check-deps kubernetes --infra my-project

3. Check Mode (Dry-Run)

Enhanced check mode that performs validation and previews deployment without making changes.

What it does:

• Runs static validation
• Validates dependencies
• Previews configuration generation
• Lists files to be deployed
• Checks prerequisites (without SSH in check mode)

Command:

provisioning taskserv create kubernetes --check

4. Sandbox Testing

Tests taskserv in isolated container environment before actual deployment.

What it tests:

• Package prerequisites
• Configuration validity
• Script execution
• Health check simulation

Command:

# Test with Docker
provisioning taskserv test kubernetes --runtime docker

# Test with Podman
provisioning taskserv test kubernetes --runtime podman

# Keep container for inspection
provisioning taskserv test kubernetes --runtime docker --keep

---

Complete Validation Workflow

Recommended Validation Sequence

# 1. Static validation (fastest, no infrastructure needed)
provisioning taskserv validate kubernetes --level static -v

# 2. Dependency validation
provisioning taskserv check-deps kubernetes --infra my-project

# 3. Check mode (dry-run with full validation)
provisioning taskserv create kubernetes --check -v

# 4. Sandbox testing (optional, requires Docker/Podman)
provisioning taskserv test kubernetes --runtime docker

# 5. Actual deployment (after all validations pass)
provisioning taskserv create kubernetes

Quick Validation (All Levels)

# Run all validation levels
provisioning taskserv validate kubernetes --level all -v

---

Validation Commands Reference

provisioning taskserv validate <taskserv>

Multi-level validation framework.

Options:

• --level <level> - Validation level: static, dependencies, health, all (default: all)
• --infra <name> - Infrastructure context
• --settings <path> - Settings file path
• --verbose - Verbose output
• --out <format> - Output format: json, yaml, text

Examples:

# Complete validation
provisioning taskserv validate kubernetes

# Only static validation
provisioning taskserv validate kubernetes --level static

# With verbose output
provisioning taskserv validate kubernetes -v

# JSON output
provisioning taskserv validate kubernetes --out json

provisioning taskserv check-deps <taskserv>

Check dependencies against infrastructure.

Options:

• --infra <name> - Infrastructure context
• --settings <path> - Settings file path
• --verbose - Verbose output

Examples:

# Check dependencies
provisioning taskserv check-deps kubernetes --infra my-project

# Verbose output
provisioning taskserv check-deps kubernetes --infra my-project -v

provisioning taskserv create <taskserv> --check

Enhanced check mode with full validation and preview.

Options:

• --check - Enable check mode (no actual deployment)
• --verbose - Verbose output
• All standard create options

Examples:

# Check mode with verbose output
provisioning taskserv create kubernetes --check -v

# Check specific server
provisioning taskserv create kubernetes server-01 --check

provisioning taskserv test <taskserv>

Sandbox testing in isolated environment.

Options:

• --runtime <name> - Runtime: docker, podman, native (default: docker)
• --infra <name> - Infrastructure context
• --settings <path> - Settings file path
• --keep - Keep container after test
• --verbose - Verbose output

Examples:

# Test with Docker
provisioning taskserv test kubernetes --runtime docker

# Test with Podman
provisioning taskserv test kubernetes --runtime podman

# Keep container for debugging
provisioning taskserv test kubernetes --keep -v

# Connect to kept container
docker exec -it taskserv-test-kubernetes bash

---

Validation Output

Static Validation

Taskserv Validation
Taskserv: kubernetes
Level: static

Validating Nickel schemas for kubernetes...
  Checking main.ncl...
    ✓ Valid
  Checking version.ncl...
    ✓ Valid
  Checking dependencies.ncl...
    ✓ Valid

Validating templates for kubernetes...
  Checking env-kubernetes.j2...
    ✓ Basic syntax OK
  Checking install-kubernetes.sh...
    ✓ Basic syntax OK

Validation Summary
✓ nickel: 0 errors, 0 warnings
✓ templates: 0 errors, 0 warnings
✓ scripts: 0 errors, 0 warnings

Overall Status
✓ VALID - 0 warnings

Dependency Validation

Dependency Validation Report
Taskserv: kubernetes

Status: VALID

Required Dependencies:
  • containerd
  • etcd
  • os

Optional Dependencies:
  • cilium
  • helm

Conflicts:
  • docker
  • podman

Check Mode Output

Check Mode: kubernetes on server-01

→ Running static validation...
  ✓ Static validation passed

→ Checking dependencies...
  ✓ Dependencies OK
    Required: containerd, etcd, os

→ Previewing configuration generation...
  ✓ Configuration preview generated
    Files to process: 15

→ Checking prerequisites...
  ℹ Prerequisite checks (preview mode):
    ⊘ Server accessibility: Check mode - SSH not tested
    ℹ Directory /tmp: Would verify directory exists
    ℹ Command bash: Would verify command is available

Check Mode Summary
✓ All validations passed

💡 Taskserv can be deployed with: provisioning taskserv create kubernetes

Test Output

Taskserv Sandbox Testing
Taskserv: kubernetes
Runtime: docker

→ Running pre-test validation...
✓ Validation passed

→ Preparing sandbox environment...
  Using base image: ubuntu:22.04
✓ Sandbox prepared: a1b2c3d4e5f6

→ Running tests in sandbox...
  Test 1: Package prerequisites...
  Test 2: Configuration validity...
  Test 3: Script execution...
  Test 4: Health check simulation...

Test Summary
Total tests: 4
Passed: 4
Failed: 0
Skipped: 0

Detailed Results:
  ✓ Package prerequisites: Package manager accessible
  ✓ Configuration validity: 3 configuration files validated
  ✓ Script execution: 2 scripts validated
  ✓ Health check: Health check configuration valid: http://localhost:6443/healthz

✓ All tests passed

---

Integration with CI/CD

GitLab CI Example

validate-taskservs:
  stage: validate
  script:
    - provisioning taskserv validate kubernetes --level all --out json
    - provisioning taskserv check-deps kubernetes --infra production

test-taskservs:
  stage: test
  script:
    - provisioning taskserv test kubernetes --runtime docker
  dependencies:
    - validate-taskservs

deploy-taskservs:
  stage: deploy
  script:
    - provisioning taskserv create kubernetes
  dependencies:
    - test-taskservs
  only:
    - main

GitHub Actions Example

name: Taskserv Validation

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Validate Taskservs
        run: |
          provisioning taskserv validate kubernetes --level all -v

      - name: Check Dependencies
        run: |
          provisioning taskserv check-deps kubernetes --infra production

      - name: Test in Sandbox
        run: |
          provisioning taskserv test kubernetes --runtime docker

---

Troubleshooting

shellcheck not found

If shellcheck is not available, script validation will be skipped with a warning.

Install shellcheck:

# macOS
brew install shellcheck

# Ubuntu/Debian
apt install shellcheck

# Fedora
dnf install shellcheck

Docker/Podman not available

Sandbox testing requires Docker or Podman.

Check runtime:

# Docker
docker ps

# Podman
podman ps

# Use native mode (limited testing)
provisioning taskserv test kubernetes --runtime native

Nickel type checking errors

Nickel type checking errors indicate syntax or type problems.

Common fixes:

• Check schema syntax in .ncl files
• Validate imports and dependencies
• Run nickel format to format files
• Check manifest.toml dependencies

Dependency conflicts

If conflicting taskservs are detected:

• Remove conflicting taskserv first
• Check infrastructure configuration
• Review dependency declarations in dependencies.ncl

---

Advanced Usage

Custom Validation Scripts

You can create custom validation scripts by extending the validation framework:

# custom_validation.nu
use provisioning/core/nulib/taskservs/validate.nu *

def custom-validate [taskserv: string] {
    # Custom validation logic
    let result = (validate-nickel-schemas $taskserv --verbose=true)

    # Additional custom checks
    # ...

    return $result
}

Batch Validation

Validate multiple taskservs:

# Validate all taskservs in infrastructure
for taskserv in (provisioning taskserv list | get name) {
    provisioning taskserv validate $taskserv
}

Automated Testing

Create test suite for all taskservs:

#!/usr/bin/env nu

let taskservs = ["kubernetes", "containerd", "cilium", "etcd"]

for ts in $taskservs {
    print $"Testing ($ts)..."
    provisioning taskserv test $ts --runtime docker
}

---

Best Practices

Before Deployment

1. Always validate before deploying to production
2. Run check mode to preview changes
3. Test in sandbox for critical services
4. Check dependencies in infrastructure context

During Development

1. Validate frequently during taskserv development
2. Use verbose mode to understand validation details
3. Fix warnings even if validation passes
4. Keep containers for debugging test failures

In CI/CD

1. Fail fast on validation errors
2. Require all tests pass before merge
3. Generate reports in JSON format for analysis
4. Archive test results for audit trail

---

Related Documentation

• Taskserv Development Guide
• KCL Schema Reference
• Dependency Management
• CI/CD Integration

---

Version History

Version	Date	Changes
1.0.0	2025-10-06	Initial validation and testing guide

---

Maintained By: Infrastructure Team Review Cycle: Quarterly