diff --git a/.woodpecker/README.md b/.woodpecker/README.md index 11d2da0..ec4c5a5 100644 --- a/.woodpecker/README.md +++ b/.woodpecker/README.md @@ -1 +1,79 @@ -# Woodpecker CI Configuration\n\nPipelines for Gitea/Forgejo + Woodpecker CI.\n\n## Files\n\n- **`ci.yml`** - Main CI pipeline (push, pull requests)\n- **`Dockerfile`** - Custom CI image with pre-installed tools\n- **`Dockerfile.cross`** - Cross-compilation image for multi-platform builds\n\n## Setup\n\n### 1. Activate Woodpecker CI\n\nEnable Woodpecker CI in your Gitea/Forgejo repository settings.\n\n### 2. (Optional) Build Custom Image\n\nSpeeds up CI by pre-installing tools (~5 min faster per run).\n\n```\n# Build CI image\ndocker build -t your-registry/ci:latest -f .woodpecker/Dockerfile .\n\n# Push to your registry\ndocker push your-registry/ci:latest\n\n# Update .woodpecker/ci.yml\n# Change: image: rust:latest\n# To: image: your-registry/ci:latest\n```\n\n### 3. Cross-Compilation Setup\n\nFor multi-platform builds:\n\n```\n# Build cross-compilation image\ndocker build -t your-registry/ci-cross:latest -f .woodpecker/Dockerfile.cross .\n\n# Push to registry\ndocker push your-registry/ci-cross:latest\n```\n\n## CI Pipeline (`ci.yml`)\n\n**Triggers**: Push to `main`/`develop`, Pull Requests\n\n**Jobs**:\n\n1. Lint (Rust, Bash, Nickel, Nushell, Markdown) - Parallel\n2. Test (all features)\n3. Build (release)\n4. Security audit\n5. License compliance check\n\n**Duration**: ~15-20 minutes (without custom image), ~10-15 minutes (with custom image)\n\n## Triggering Pipelines\n\n```\n# CI pipeline (automatic on push/PR)\ngit push origin main\n```\n\n## Viewing Results\n\n- **Gitea/Forgejo**: Repository → Actions → Pipeline runs\n- **Woodpecker UI**: \n\n## Differences from GitHub Actions\n\n| Feature | GitHub Actions | Woodpecker CI |\n| --------- | --------------- | --------------- |\n| Matrix builds | ✅ 3 OS | ❌ Linux only* |\n| Caching | ✅ Built-in | ⚠️ Server-side** |\n\n\* Multi-OS builds require multiple Woodpecker agents\n\*\* Configure in Woodpecker server settings +# Woodpecker CI Configuration + +Pipelines for Gitea/Forgejo + Woodpecker CI. + +## Files + +- **`ci.yml`** - Main CI pipeline (push, pull requests) +- **`Dockerfile`** - Custom CI image with pre-installed tools +- **`Dockerfile.cross`** - Cross-compilation image for multi-platform builds + +## Setup + +### 1. Activate Woodpecker CI + +Enable Woodpecker CI in your Gitea/Forgejo repository settings. + +### 2. (Optional) Build Custom Image + +Speeds up CI by pre-installing tools (~5 min faster per run). + +```bash +# Build CI image +docker build -t your-registry/ci:latest -f .woodpecker/Dockerfile . + +# Push to your registry +docker push your-registry/ci:latest + +# Update .woodpecker/ci.yml +# Change: image: rust:latest +# To: image: your-registry/ci:latest +``` + +### 3. Cross-Compilation Setup + +For multi-platform builds: + +```bash +# Build cross-compilation image +docker build -t your-registry/ci-cross:latest -f .woodpecker/Dockerfile.cross . + +# Push to registry +docker push your-registry/ci-cross:latest +``` + +## CI Pipeline (`ci.yml`) + +**Triggers**: Push to `main`/`develop`, Pull Requests + +**Jobs**: + +1. Lint (Rust, Bash, Nickel, Nushell, Markdown) - Parallel +2. Test (all features) +3. Build (release) +4. Security audit +5. License compliance check + +**Duration**: ~15-20 minutes (without custom image), ~10-15 minutes (with custom image) + +## Triggering Pipelines + +```bash +# CI pipeline (automatic on push/PR) +git push origin main +``` + +## Viewing Results + +- **Gitea/Forgejo**: Repository → Actions → Pipeline runs +- **Woodpecker UI**: + +## Differences from GitHub Actions + +| Feature | GitHub Actions | Woodpecker CI | +| --------- | --------------- | --------------- | +| Matrix builds | ✅ 3 OS | ❌ Linux only* | +| Caching | ✅ Built-in | ⚠️ Server-side** | + +\* Multi-OS builds require multiple Woodpecker agents +\*\* Configure in Woodpecker server settings \ No newline at end of file diff --git a/locales/TRANSLATIONS_STATUS.md b/locales/TRANSLATIONS_STATUS.md index 3a50de2..de469d7 100644 --- a/locales/TRANSLATIONS_STATUS.md +++ b/locales/TRANSLATIONS_STATUS.md @@ -1 +1,345 @@ -# Multilingual Provisioning System - Translation Status\n\n**Last Updated**: 2026-01-13\n**Status**: 100% Complete (Phases 1-4D)\n\n## Executive Summary\n\nThe Provisioning system now supports comprehensive multilingual interfaces across all components:\n\n- **Help System**: 65 strings extracted and translated\n- **Forms**: ~180 strings covering setup, authentication, and infrastructure management\n- **Supported Languages**: English (en-US), Spanish (es-ES) with infrastructure for future additions\n\n## Language Coverage\n\n### English (en-US) - 100% Complete ✅\n\n**Source of Truth**: `/provisioning/locales/en-US/`\n\n| Component | File | Strings | Status |\n|-----------|------|---------|--------|\n| Help System | `help.ftl` | 65 | ✅ Complete |\n| Forms | `forms.ftl` | 180 | ✅ Complete |\n| **Total** | | **245** | **✅ Complete** |\n\n### Spanish (es-ES) - 100% Complete ✅\n\n**Source of Truth**: `/provisioning/locales/es-ES/`\n\n| Component | File | Strings | Status |\n|-----------|------|---------|--------|\n| Help System | `help.ftl` | 65 | ✅ Complete |\n| Forms | `forms.ftl` | 180 | ✅ Complete |\n| **Total** | | **245** | **✅ Complete** |\n\n## String Breakdown by Category\n\n### Help System (65 strings)\n\n**Main Menu** (26 strings):\n- Title, subtitle, category hint\n- 13 category names and descriptions (infrastructure, orchestration, development, workspace, platform, setup, authentication, plugins, utilities, tools, vm, diagnostics, concepts, guides, integrations)\n- 3 error messages\n\n**Infrastructure** (21 strings):\n- Section title\n- Server operations (4 strings)\n- TaskServ management (4 strings)\n- Cluster management (4 strings)\n- VM operations (4 strings)\n- Infrastructure tip\n\n**Orchestration** (18 strings):\n- Section title\n- Orchestrator management (5 strings)\n- Workflow operations (5 strings)\n- Batch operations (7 strings)\n- Batch workflows tip + example\n\n### Forms (180 strings)\n\n**Unified Setup Form** (~140 strings):\n- Project information (project name, version, description)\n- Database configuration (PostgreSQL, MySQL, MongoDB, SQLite)\n- API service configuration (name, ports, health check, replicas)\n- Deployment options (Docker Compose, Kubernetes, Cloud)\n- Advanced options (monitoring, TLS)\n- Security & authentication (JWT, OAuth2, SAML, None)\n- Terms & conditions (terms, newsletter, save address)\n\n**Core Forms** (~40 strings):\n- Auth login form (username, password, remember me, forgot password)\n- Setup wizard (quick, standard, advanced)\n- MFA enrollment (TOTP, SMS, backup codes, device name)\n\n**Infrastructure Forms** (interactive):\n- Delete confirmations (30+ potential variations for different resources)\n- Resource confirmation prompts\n- Data retention options\n\n## TypeDialog Integration\n\n### Configured Forms\n\nThe following root forms have `locales_path` configured to use Fluent catalogs:\n\n| Form | Path | locales_path |\n|------|------|--------------|\n| Unified Setup | `.typedialog/provisioning/form.toml` | `../../../locales` |\n| CI Configuration | `.typedialog/ci/form.toml` | `../../../locales` |\n| Core Auth | `.typedialog/core/forms/auth-login.toml` | `../../../../../locales` |\n| Core Setup Wizard | `.typedialog/core/forms/setup-wizard.toml` | `../../../../../locales` |\n| Core MFA | `.typedialog/core/forms/mfa-enroll.toml` | `../../../../../locales` |\n\n**Note**: Fragment forms (70+ files) inherit locales from their parent forms through TypeDialog's `load_fragments` mechanism.\n\n## Fluent File Organization\n\n```\nprovisioning/locales/\n├── i18n-config.toml # Central i18n configuration\n├── TRANSLATIONS_STATUS.md # This file\n├── en-US/ # English base language\n│ ├── help.ftl # Help system strings (65 keys)\n│ └── forms.ftl # Form strings (180 keys)\n└── es-ES/ # Spanish translations\n ├── help.ftl # Help system translations\n └── forms.ftl # Form translations\n```\n\n## Fallback Chains\n\nWhen a string is missing in the active locale, TypeDialog automatically falls back to the configured chain:\n\n```\nes-ES → en-US (default)\n```\n\n**Configuration** in `i18n-config.toml`:\n```\n[fallback_chains]\nes-ES = ["en-US"]\n```\n\nThis ensures that if any Spanish translation is missing, the English version is displayed as a fallback.\n\n## Feature Configuration\n\nThe i18n system supports these features (enabled in `i18n-config.toml`):\n\n| Feature | Status | Purpose |\n|---------|--------|---------|\n| Pluralization | ✅ Enabled | Support plural forms in translations |\n| Number Formatting | ✅ Enabled | Locale-specific number/currency formatting |\n| Date Formatting | ✅ Enabled | Locale-specific date formats |\n| Fallback Chains | ✅ Enabled | Automatic fallback to English |\n| Gender Agreement | ⚠️ Disabled | Spanish doesn't need gender in these strings |\n| RTL Support | ⚠️ Disabled | No RTL languages configured yet |\n\n## Translation Quality Standards\n\n### Naming Conventions\n\nAll Fluent keys follow a consistent pattern:\n\n- **Help strings**: `help-{category}-{element}` (e.g., `help-infra-server-create`)\n- **Form prompts**: `form-{element}-prompt` (e.g., `form-project_name-prompt`)\n- **Form help**: `form-{element}-help` (e.g., `form-project_name-help`)\n- **Form placeholders**: `form-{element}-placeholder`\n- **Form options**: `form-{element}-option-{value}` (e.g., `form-database_type-option-postgres`)\n- **Section headers**: `section-{name}-title`\n\n### Coverage Requirements\n\nFrom `i18n-config.toml`:\n\n- **Required Coverage**: 95% (critical locales: en-US, es-ES)\n- **Warning Threshold**: 80%\n- **Validation**: Missing keys trigger warnings during build\n\n## Testing & Validation\n\n### Locale Resolution\n\nThe system uses the `LANG` environment variable for locale selection:\n\n```\n# English (default)\n$ LANG=en_US provisioning help infrastructure\n# Output: SERVER & INFRASTRUCTURE...\n\n# Spanish\n$ LANG=es_ES provisioning help infrastructure\n# Output: SERVIDOR E INFRAESTRUCTURA...\n\n# Fallback to English if missing locale file\n$ LANG=fr_FR provisioning help infrastructure\n# Output: SERVER & INFRASTRUCTURE... (fallback)\n```\n\n### Form Testing\n\nTypeDialog forms automatically use the configured locale:\n\n```\n# Display Unified Setup in English\n$ LANG=en_US provisioning setup profile\n\n# Display Unified Setup in Spanish\n$ LANG=es_ES provisioning setup profile\n```\n\n### Coverage Validation\n\nTo validate translation coverage:\n\n```\n# Check translation status\nprovisioning i18n status\n\n# Generate coverage report\nprovisioning i18n coverage --locale es-ES\n# Expected: 100% (245/245 strings translated)\n\n# Validate Fluent files\nprovisioning i18n validate\n```\n\n## Future Expansion\n\nThe infrastructure supports adding new languages. To add a new locale (e.g., Portuguese):\n\n### 1. Add Locale Configuration\n\nIn `i18n-config.toml`:\n\n```\n[locales.pt-BR]\nname = "Portuguese (Brazil)"\ndirection = "ltr"\nplurals = 2\ndecimal_separator = ","\nthousands_separator = "."\ndate_format = "DD/MM/YYYY"\n\n[fallback_chains]\npt-BR = ["pt-PT", "es-ES", "en-US"]\n```\n\n### 2. Create Locale Directory\n\n```\nmkdir -p provisioning/locales/pt-BR\n```\n\n### 3. Create Translation Files\n\n```\ncp provisioning/locales/en-US/help.ftl provisioning/locales/pt-BR/help.ftl\ncp provisioning/locales/en-US/forms.ftl provisioning/locales/pt-BR/forms.ftl\n```\n\n### 4. Translate Strings\n\nUpdate `pt-BR/help.ftl` and `pt-BR/forms.ftl` with Portuguese translations.\n\n### 5. Validate\n\n```\nprovisioning i18n validate --locale pt-BR\n```\n\n## Architecture & Implementation Details\n\n### Mozilla Fluent Format\n\nAll translations use Mozilla Fluent (.ftl files), which offers:\n\n- **Simple Syntax**: `key = value` format\n- **Rich Features**: Pluralization, gender agreement, attributes\n- **Fallback Support**: Automatic chain resolution\n- **Extensibility**: Support for custom functions and formatting\n\n**Example**:\n```\nhelp-infra-server-create = Create a new server\nform-database_type-option-postgres = PostgreSQL (Recommended)\n```\n\n### TypeDialog Integration\n\nTypeDialog forms reference Fluent keys via `locales_path`:\n\n```\nlocales_path = "../../../locales"\n\n[[elements]]\nname = "project_name"\nprompt = "form-project_name-prompt" # References: locales/*/forms.ftl\nhelp = "form-project_name-help"\nplaceholder = "form-project_name-placeholder"\n```\n\nTypeDialog's locale resolution:\n\n1. Check `locales_path` configuration\n2. Look for `LANG` environment variable (e.g., `es_ES`)\n3. Find corresponding Fluent file (e.g., `es-ES/forms.ftl`)\n4. Resolve key → value\n5. Fallback to parent locale chain if missing\n6. Use literal key if no translation found\n\n## Coverage Metrics\n\n### String Count Summary\n\n| Category | en-US | es-ES | Coverage |\n|----------|-------|-------|----------|\n| Help System | 65 | 65 | 100% ✅ |\n| Forms | 180 | 180 | 100% ✅ |\n| **Total** | **245** | **245** | **100%** ✅ |\n\n### Language Support\n\n| Locale | Strings | Status | Notes |\n|--------|---------|--------|-------|\n| en-US | 245 | ✅ Complete | Base language |\n| es-ES | 245 | ✅ Complete | Full translation |\n| pt-BR | - | 🔄 Planned | Infrastructure ready |\n| fr-FR | - | 🔄 Planned | Infrastructure ready |\n| ja-JP | - | 🔄 Planned | Infrastructure ready |\n\n## Maintenance & Updates\n\n### Adding Translations\n\nWhen new forms or help sections are added:\n\n1. Extract strings using extraction tools\n2. Add Fluent keys to `en-US/*.ftl`\n3. Translate to `es-ES/*.ftl`\n4. Update this status document\n\n### Validation Checklist\n\n- [ ] All new strings have Fluent keys\n- [ ] Keys follow naming conventions\n- [ ] English translation complete\n- [ ] Spanish translation complete\n- [ ] Fallback chains tested\n- [ ] LANG environment variable works\n- [ ] TypeDialog forms display correctly\n\n## References\n\n- **Fluent Documentation**: https://projectfluent.org/\n- **TypeDialog i18n**: TypeDialog embedded documentation\n- **i18n Configuration**: See `provisioning/locales/i18n-config.toml`\n- **Help System**: See `provisioning/core/nulib/main_provisioning/help_system.nu`\n\n---\n\n**Status**: ✅ Complete\n**Phases Completed**: 1, 2, 3, 4A, 4B, 4C, 4D\n**Ready for**: Production deployment, further language additions, testing\n +# Multilingual Provisioning System - Translation Status + +**Last Updated**: 2026-01-13 +**Status**: 100% Complete (Phases 1-4D) + +## Executive Summary + +The Provisioning system now supports comprehensive multilingual interfaces across all components: + +- **Help System**: 65 strings extracted and translated +- **Forms**: ~180 strings covering setup, authentication, and infrastructure management +- **Supported Languages**: English (en-US), Spanish (es-ES) with infrastructure for future additions + +## Language Coverage + +### English (en-US) - 100% Complete ✅ + +**Source of Truth**: `/provisioning/locales/en-US/` + +| Component | File | Strings | Status | +|-----------|------|---------|--------| +| Help System | `help.ftl` | 65 | ✅ Complete | +| Forms | `forms.ftl` | 180 | ✅ Complete | +| **Total** | | **245** | **✅ Complete** | + +### Spanish (es-ES) - 100% Complete ✅ + +**Source of Truth**: `/provisioning/locales/es-ES/` + +| Component | File | Strings | Status | +|-----------|------|---------|--------| +| Help System | `help.ftl` | 65 | ✅ Complete | +| Forms | `forms.ftl` | 180 | ✅ Complete | +| **Total** | | **245** | **✅ Complete** | + +## String Breakdown by Category + +### Help System (65 strings) + +**Main Menu** (26 strings): +- Title, subtitle, category hint +- 13 category names and descriptions (infrastructure, orchestration, development, workspace, platform, setup, authentication, plugins, utilities, tools, vm, diagnostics, concepts, guides, integrations) +- 3 error messages + +**Infrastructure** (21 strings): +- Section title +- Server operations (4 strings) +- TaskServ management (4 strings) +- Cluster management (4 strings) +- VM operations (4 strings) +- Infrastructure tip + +**Orchestration** (18 strings): +- Section title +- Orchestrator management (5 strings) +- Workflow operations (5 strings) +- Batch operations (7 strings) +- Batch workflows tip + example + +### Forms (180 strings) + +**Unified Setup Form** (~140 strings): +- Project information (project name, version, description) +- Database configuration (PostgreSQL, MySQL, MongoDB, SQLite) +- API service configuration (name, ports, health check, replicas) +- Deployment options (Docker Compose, Kubernetes, Cloud) +- Advanced options (monitoring, TLS) +- Security & authentication (JWT, OAuth2, SAML, None) +- Terms & conditions (terms, newsletter, save address) + +**Core Forms** (~40 strings): +- Auth login form (username, password, remember me, forgot password) +- Setup wizard (quick, standard, advanced) +- MFA enrollment (TOTP, SMS, backup codes, device name) + +**Infrastructure Forms** (interactive): +- Delete confirmations (30+ potential variations for different resources) +- Resource confirmation prompts +- Data retention options + +## TypeDialog Integration + +### Configured Forms + +The following root forms have `locales_path` configured to use Fluent catalogs: + +| Form | Path | locales_path | +|------|------|--------------| +| Unified Setup | `.typedialog/provisioning/form.toml` | `../../../locales` | +| CI Configuration | `.typedialog/ci/form.toml` | `../../../locales` | +| Core Auth | `.typedialog/core/forms/auth-login.toml` | `../../../../../locales` | +| Core Setup Wizard | `.typedialog/core/forms/setup-wizard.toml` | `../../../../../locales` | +| Core MFA | `.typedialog/core/forms/mfa-enroll.toml` | `../../../../../locales` | + +**Note**: Fragment forms (70+ files) inherit locales from their parent forms through TypeDialog's `load_fragments` mechanism. + +## Fluent File Organization + +```bash +provisioning/locales/ +├── i18n-config.toml # Central i18n configuration +├── TRANSLATIONS_STATUS.md # This file +├── en-US/ # English base language +│ ├── help.ftl # Help system strings (65 keys) +│ └── forms.ftl # Form strings (180 keys) +└── es-ES/ # Spanish translations + ├── help.ftl # Help system translations + └── forms.ftl # Form translations +``` + +## Fallback Chains + +When a string is missing in the active locale, TypeDialog automatically falls back to the configured chain: + +```toml +es-ES → en-US (default) +``` + +**Configuration** in `i18n-config.toml`: +```toml +[fallback_chains] +es-ES = ["en-US"] +``` + +This ensures that if any Spanish translation is missing, the English version is displayed as a fallback. + +## Feature Configuration + +The i18n system supports these features (enabled in `i18n-config.toml`): + +| Feature | Status | Purpose | +|---------|--------|---------| +| Pluralization | ✅ Enabled | Support plural forms in translations | +| Number Formatting | ✅ Enabled | Locale-specific number/currency formatting | +| Date Formatting | ✅ Enabled | Locale-specific date formats | +| Fallback Chains | ✅ Enabled | Automatic fallback to English | +| Gender Agreement | ⚠️ Disabled | Spanish doesn't need gender in these strings | +| RTL Support | ⚠️ Disabled | No RTL languages configured yet | + +## Translation Quality Standards + +### Naming Conventions + +All Fluent keys follow a consistent pattern: + +- **Help strings**: `help-{category}-{element}` (e.g., `help-infra-server-create`) +- **Form prompts**: `form-{element}-prompt` (e.g., `form-project_name-prompt`) +- **Form help**: `form-{element}-help` (e.g., `form-project_name-help`) +- **Form placeholders**: `form-{element}-placeholder` +- **Form options**: `form-{element}-option-{value}` (e.g., `form-database_type-option-postgres`) +- **Section headers**: `section-{name}-title` + +### Coverage Requirements + +From `i18n-config.toml`: + +- **Required Coverage**: 95% (critical locales: en-US, es-ES) +- **Warning Threshold**: 80% +- **Validation**: Missing keys trigger warnings during build + +## Testing & Validation + +### Locale Resolution + +The system uses the `LANG` environment variable for locale selection: + +```bash +# English (default) +$ LANG=en_US provisioning help infrastructure +# Output: SERVER & INFRASTRUCTURE... + +# Spanish +$ LANG=es_ES provisioning help infrastructure +# Output: SERVIDOR E INFRAESTRUCTURA... + +# Fallback to English if missing locale file +$ LANG=fr_FR provisioning help infrastructure +# Output: SERVER & INFRASTRUCTURE... (fallback) +``` + +### Form Testing + +TypeDialog forms automatically use the configured locale: + +```toml +# Display Unified Setup in English +$ LANG=en_US provisioning setup profile + +# Display Unified Setup in Spanish +$ LANG=es_ES provisioning setup profile +``` + +### Coverage Validation + +To validate translation coverage: + +```bash +# Check translation status +provisioning i18n status + +# Generate coverage report +provisioning i18n coverage --locale es-ES +# Expected: 100% (245/245 strings translated) + +# Validate Fluent files +provisioning i18n validate +``` + +## Future Expansion + +The infrastructure supports adding new languages. To add a new locale (e.g., Portuguese): + +### 1. Add Locale Configuration + +In `i18n-config.toml`: + +```toml +[locales.pt-BR] +name = "Portuguese (Brazil)" +direction = "ltr" +plurals = 2 +decimal_separator = "," +thousands_separator = "." +date_format = "DD/MM/YYYY" + +[fallback_chains] +pt-BR = ["pt-PT", "es-ES", "en-US"] +``` + +### 2. Create Locale Directory + +```bash +mkdir -p provisioning/locales/pt-BR +``` + +### 3. Create Translation Files + +```bash +cp provisioning/locales/en-US/help.ftl provisioning/locales/pt-BR/help.ftl +cp provisioning/locales/en-US/forms.ftl provisioning/locales/pt-BR/forms.ftl +``` + +### 4. Translate Strings + +Update `pt-BR/help.ftl` and `pt-BR/forms.ftl` with Portuguese translations. + +### 5. Validate + +```bash +provisioning i18n validate --locale pt-BR +``` + +## Architecture & Implementation Details + +### Mozilla Fluent Format + +All translations use Mozilla Fluent (.ftl files), which offers: + +- **Simple Syntax**: `key = value` format +- **Rich Features**: Pluralization, gender agreement, attributes +- **Fallback Support**: Automatic chain resolution +- **Extensibility**: Support for custom functions and formatting + +**Example**: +```bash +help-infra-server-create = Create a new server +form-database_type-option-postgres = PostgreSQL (Recommended) +``` + +### TypeDialog Integration + +TypeDialog forms reference Fluent keys via `locales_path`: + +```bash +locales_path = "../../../locales" + +[[elements]] +name = "project_name" +prompt = "form-project_name-prompt" # References: locales/*/forms.ftl +help = "form-project_name-help" +placeholder = "form-project_name-placeholder" +``` + +TypeDialog's locale resolution: + +1. Check `locales_path` configuration +2. Look for `LANG` environment variable (e.g., `es_ES`) +3. Find corresponding Fluent file (e.g., `es-ES/forms.ftl`) +4. Resolve key → value +5. Fallback to parent locale chain if missing +6. Use literal key if no translation found + +## Coverage Metrics + +### String Count Summary + +| Category | en-US | es-ES | Coverage | +|----------|-------|-------|----------| +| Help System | 65 | 65 | 100% ✅ | +| Forms | 180 | 180 | 100% ✅ | +| **Total** | **245** | **245** | **100%** ✅ | + +### Language Support + +| Locale | Strings | Status | Notes | +|--------|---------|--------|-------| +| en-US | 245 | ✅ Complete | Base language | +| es-ES | 245 | ✅ Complete | Full translation | +| pt-BR | - | 🔄 Planned | Infrastructure ready | +| fr-FR | - | 🔄 Planned | Infrastructure ready | +| ja-JP | - | 🔄 Planned | Infrastructure ready | + +## Maintenance & Updates + +### Adding Translations + +When new forms or help sections are added: + +1. Extract strings using extraction tools +2. Add Fluent keys to `en-US/*.ftl` +3. Translate to `es-ES/*.ftl` +4. Update this status document + +### Validation Checklist + +- [ ] All new strings have Fluent keys +- [ ] Keys follow naming conventions +- [ ] English translation complete +- [ ] Spanish translation complete +- [ ] Fallback chains tested +- [ ] LANG environment variable works +- [ ] TypeDialog forms display correctly + +## References + +- **Fluent Documentation**: https://projectfluent.org/ +- **TypeDialog i18n**: TypeDialog embedded documentation +- **i18n Configuration**: See `provisioning/locales/i18n-config.toml` +- **Help System**: See `provisioning/core/nulib/main_provisioning/help_system.nu` + +--- + +**Status**: ✅ Complete +**Phases Completed**: 1, 2, 3, 4A, 4B, 4C, 4D +**Ready for**: Production deployment, further language additions, testing \ No newline at end of file diff --git a/resources/images/how-to-use.md b/resources/images/how-to-use.md index 3e04285..7c77db1 100644 --- a/resources/images/how-to-use.md +++ b/resources/images/how-to-use.md @@ -1 +1,5 @@ -provisioning_logo-dark.svg logo image for dark mode\nprovisioning_logo-light.svg logo image for normal mode\nprovisioning_logo-text-dark.svg logo text for dark mode\nprovisioning_logo-text-light.svg logo text for normal mode\nprovisioning_logo-image.svg main image +provisioning_logo-dark.svg logo image for dark mode +provisioning_logo-light.svg logo image for normal mode +provisioning_logo-text-dark.svg logo text for dark mode +provisioning_logo-text-light.svg logo text for normal mode +provisioning_logo-image.svg main image diff --git a/scripts/setup-platform-config.sh.md b/scripts/setup-platform-config.sh.md index 2aaba50..d3a6f05 100644 --- a/scripts/setup-platform-config.sh.md +++ b/scripts/setup-platform-config.sh.md @@ -1 +1,402 @@ -# Platform Services Configuration Setup Script\n\n**Path**: `provisioning/scripts/setup-platform-config.sh`\n\nSetup and manage platform service configurations in `provisioning/config/runtime/`.\n\n## Features\n\n- ✅ **Interactive Mode**: Guided setup with TypeDialog or quick mode\n- ✅ **Interactive TypeDialog**: Web/TUI/CLI form-based configuration\n- ✅ **Quick Mode**: Auto-setup from defaults + mode overlays\n- ✅ **Automatic TOML Export**: Generates TOML files for Rust services\n- ✅ **Runtime Detection**: Detect existing configs and offer update/replace options\n- ✅ **Batch Operations**: Configure all 8 services at once\n- ✅ **Cleanup Management**: Remove/reset configurations safely\n\n## Usage\n\n### Interactive Setup (Recommended)\n\n```\n# Start interactive wizard\n./provisioning/scripts/setup-platform-config.sh\n\n# Prompts for:\n# 1. Action: TypeDialog, Quick Mode, Clean, or List\n# 2. Service (if TypeDialog/Quick)\n# 3. Mode (solo/multiuser/cicd/enterprise)\n# 4. Backend (web/tui/cli, if TypeDialog)\n```\n\n### Command-Line Options\n\n```\n# Configure specific service via TypeDialog\n./provisioning/scripts/setup-platform-config.sh \n --service orchestrator \n --mode solo \n --backend web\n\n# Quick setup all services for enterprise mode\n./provisioning/scripts/setup-platform-config.sh \n --quick-mode \n --mode enterprise\n\n# Regenerate TOML files from existing .ncl configs\n./provisioning/scripts/setup-platform-config.sh \n --generate-toml\n\n# List available options\n./provisioning/scripts/setup-platform-config.sh --list-modes\n./provisioning/scripts/setup-platform-config.sh --list-services\n./provisioning/scripts/setup-platform-config.sh --list-configs\n\n# Clean all runtime configurations\n./provisioning/scripts/setup-platform-config.sh --clean\n```\n\n## Workflow\n\n### 1. Initial Setup (Empty Runtime)\n\n```\nInteractive Prompt\n ↓\n├─ TypeDialog (Recommended)\n│ ├─ Load form definitions\n│ ├─ User fills form (web/tui/cli)\n│ └─ Generates orchestrator.solo.ncl\n│ ↓\n│ Auto-export to orchestrator.solo.toml\n│\n└─ Quick Mode\n ├─ Select mode (solo/multiuser/cicd/enterprise)\n ├─ Compose all services: defaults + mode overlay\n ├─ Create 8 .ncl files\n └─ Auto-export to 8 .toml files\n```\n\n### 2. Update Existing Configuration\n\n```\nDetect Existing Config\n ↓\nChoose Action:\n ├─ Clean up & start fresh\n ├─ Update via TypeDialog (edit existing)\n ├─ Use quick mode (regenerate)\n └─ List current configs\n```\n\n### 3. Manual NCL Edits\n\n```\nUser edits: provisioning/config/runtime/orchestrator.solo.ncl\n ↓\nRun: ./setup-platform-config.sh --generate-toml\n ↓\nAuto-exports to: provisioning/config/runtime/generated/orchestrator.solo.toml\n ↓\nService loads TOML automatically\n```\n\n## Configuration Layers\n\nThe script composes configurations from multiple layers:\n\n```\n1. Schema (TYPE-SAFE CONTRACT)\n ↓\n provisioning/schemas/platform/schemas/orchestrator.ncl\n (Defines valid fields, types, constraints)\n\n2. Service Defaults (BASE VALUES)\n ↓\n provisioning/schemas/platform/defaults/orchestrator-defaults.ncl\n (Default values for all orchestrator settings)\n\n3. Mode Overlay (MODE-SPECIFIC TUNING)\n ↓\n provisioning/schemas/platform/defaults/deployment/solo-defaults.ncl\n (Resource limits for solo mode: 2 CPU, 4GB RAM)\n\n4. Composition (MERGE)\n ↓\n defaults.merge_config_with_mode(mode_config)\n (Merges base + mode overlay)\n\n5. Runtime Config (USER CUSTOMIZATION)\n ↓\n provisioning/config/runtime/orchestrator.solo.ncl\n (Final config, can be hand-edited)\n\n6. TOML Export (SERVICE CONSUMPTION)\n ↓\n provisioning/config/runtime/generated/orchestrator.solo.toml\n (Rust service reads this)\n```\n\n## Services & Modes\n\n### 8 Available Services\n\n```\n1. orchestrator - Main orchestration engine\n2. control-center - Web UI and management console\n3. mcp-server - Model Context Protocol server\n4. vault-service - Secrets management and encryption\n5. extension-registry - Extension distribution system\n6. rag - Retrieval-Augmented Generation\n7. ai-service - AI model integration\n8. provisioning-daemon - Background operations\n```\n\n### 4 Deployment Modes\n\n| Mode | Specs | Use Case |\n| ------ | ------- | ---------- |\n| `solo` | 2 CPU, 4GB RAM | Development, testing |\n| `multiuser` | 4 CPU, 8GB RAM | Team staging |\n| `cicd` | 8 CPU, 16GB RAM | CI/CD pipelines |\n| `enterprise` | 16+ CPU, 32+ GB | Production HA |\n\n## Directory Structure\n\n```\nprovisioning/\n├── config/\n│ └── runtime/ # 🔒 PRIVATE (gitignored)\n│ ├── .gitignore\n│ ├── orchestrator.solo.ncl # Runtime config (user editable)\n│ ├── vault-service.multiuser.ncl # Runtime config\n│ └── generated/ # TOMLs (auto-generated)\n│ ├── orchestrator.solo.toml # For Rust services\n│ └── vault-service.multiuser.toml\n│\n├── schemas/platform/ # 📘 PUBLIC (versionable)\n│ ├── schemas/ # Type contracts\n│ ├── defaults/ # Base values\n│ │ ├── orchestrator-defaults.ncl\n│ │ └── deployment/\n│ │ ├── solo-defaults.ncl\n│ │ ├── multiuser-defaults.ncl\n│ │ ├── cicd-defaults.ncl\n│ │ └── enterprise-defaults.ncl\n│ └── validators/ # Business logic\n│\n└── scripts/\n └── setup-platform-config.sh # This script\n```\n\n## Requirements\n\n- **Bash 4.0+**\n- **Nickel 0.10+** - Configuration language\n- **Nushell 0.109+** - Script engine\n- **TypeDialog** (optional, for interactive setup)\n\n## Integration with Provisioning Installer\n\n### ⚠️ Current Status: Installer NOT YET IMPLEMENTED\n\nThe `setup-platform-config.sh` script is a **standalone tool** ready to use independently.\n\n**For now**: Call the script manually before running services\n\n```\n# Step 1: Setup platform configurations (MANUAL)\n./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo\n\n# Step 2: Run services\nexport ORCHESTRATOR_MODE=solo\ncargo run -p orchestrator\n```\n\n### Future: When Installer is Implemented\n\nOnce `provisioning/scripts/install.sh` is ready, it will look like:\n\n```\n#!/bin/bash\n# provisioning/scripts/install.sh (FUTURE)\n\n# Pre-flight checks\ncheck_dependencies() {\n command -v nickel >/dev/null || { echo "Nickel required"; exit 1; }\n command -v nu >/dev/null || { echo "Nushell required"; exit 1; }\n}\ncheck_dependencies\n\n# Install provisioning system\necho "Installing provisioning system..."\n# (implementation here)\n\n# Setup platform configurations\necho "Setting up platform configurations..."\n./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo\n\n# Build and verify\necho "Building platform services..."\ncargo build -p orchestrator -p control-center -p mcp-server\n\necho "Installation complete!"\n```\n\n### CI/CD Integration (Available Now)\n\nFor CI/CD pipelines that don't require the full installer:\n\n```\n#!/bin/bash\n# ci/setup.sh\n\n# Setup configurations for CI/CD mode\n./provisioning/scripts/setup-platform-config.sh \n --quick-mode \n --mode cicd\n\n# Run tests\ncargo test --all\n\n# Deploy\ndocker-compose -f provisioning/platform/infrastructure/docker/docker-compose.cicd.yml up\n```\n\n## Important Notes\n\n### ⚠️ Manual Edits\n\nIf you manually edit `.ncl` files in `provisioning/config/runtime/`:\n\n```\n# Always regenerate TOMLs afterward\n./provisioning/scripts/setup-platform-config.sh --generate-toml\n```\n\n### 🔒 Private Configurations\n\nFiles in `provisioning/config/runtime/` are **gitignored**:\n- `.ncl` files (may contain secrets/encrypted values)\n- `generated/*.toml` files (auto-generated, no need to version)\n\n### 📘 Public Schemas\n\nSchemas in `provisioning/schemas/platform/` are **public**:\n- Source of truth for configuration structure\n- Versionable and shared across team\n- Can be committed to git\n\n### 🔄 Regeneration\n\nThe script is **idempotent** - run it multiple times safely:\n\n```\n# Safe: Re-runs setup, updates configs\n./provisioning/scripts/setup-platform-config.sh --quick-mode --mode multiuser\n\n# Does NOT overwrite manually edited files (unless --clean is used)\n```\n\n## Troubleshooting\n\n### Nickel Validation Fails\n\n```\n# Check syntax of generated config\nnickel typecheck provisioning/config/runtime/orchestrator.solo.ncl\n\n# View detailed error\nnickel export --format toml provisioning/config/runtime/orchestrator.solo.ncl\n```\n\n### TOML Export Fails\n\n```\n# Check if Nickel config is valid\nnickel typecheck provisioning/config/runtime/orchestrator.solo.ncl\n\n# Try manual export\nnickel export --format toml provisioning/config/runtime/orchestrator.solo.ncl\n```\n\n### Service Won't Start\n\n```\n# Verify TOML exists\nls -la provisioning/config/runtime/generated/orchestrator.solo.toml\n\n# Check TOML syntax\ncat provisioning/config/runtime/generated/orchestrator.solo.toml | head -20\n\n# Verify service can read TOML\nORCHESTRATOR_MODE=solo cargo run -p orchestrator --\n```\n\n## Examples\n\n### Example 1: Quick Setup for Development\n\n```\n# Setup all services for solo mode (2 CPU, 4GB RAM)\n./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo\n\n# Result:\n# ✓ Created provisioning/config/runtime/orchestrator.solo.ncl\n# ✓ Created provisioning/config/runtime/control-center.solo.ncl\n# ✓ ... (8 services total)\n# ✓ Generated 8 TOML files in provisioning/config/runtime/generated/\n\n# Run service:\nexport ORCHESTRATOR_MODE=solo\ncargo run -p orchestrator\n```\n\n### Example 2: Interactive TypeDialog Setup\n\n```\n# Configure orchestrator in multiuser mode with web UI\n./provisioning/scripts/setup-platform-config.sh \n --service orchestrator \n --mode multiuser \n --backend web\n\n# TypeDialog opens browser, user fills form\n# Result:\n# ✓ Created provisioning/config/runtime/orchestrator.multiuser.ncl\n# ✓ Generated provisioning/config/runtime/generated/orchestrator.multiuser.toml\n\n# Run service:\nexport ORCHESTRATOR_MODE=multiuser\ncargo run -p orchestrator\n```\n\n### Example 3: Update After Manual Edit\n\n```\n# Edit config manually\nvim provisioning/config/runtime/orchestrator.solo.ncl\n\n# Regenerate TOML (critical!)\n./provisioning/scripts/setup-platform-config.sh --generate-toml\n\n# Verify changes\ncat provisioning/config/runtime/generated/orchestrator.solo.toml | head -20\n\n# Restart service with new config\npkill orchestrator\nORCHESTRATOR_MODE=solo cargo run -p orchestrator\n```\n\n## Performance Notes\n\n- **Quick Mode**: ~1-2 seconds (no user interaction, direct composition)\n- **TypeDialog (web)**: 10-30 seconds (server startup + UI loading)\n- **TOML Export**: <1 second per service\n- **Full Setup**: 5-10 minutes (8 services via TypeDialog)\n\n---\n\n**Version**: 1.0.0\n**Created**: 2026-01-05\n**Last Updated**: 2026-01-05 \ No newline at end of file +# Platform Services Configuration Setup Script + +**Path**: `provisioning/scripts/setup-platform-config.sh` + +Setup and manage platform service configurations in `provisioning/config/runtime/`. + +## Features + +- ✅ **Interactive Mode**: Guided setup with TypeDialog or quick mode +- ✅ **Interactive TypeDialog**: Web/TUI/CLI form-based configuration +- ✅ **Quick Mode**: Auto-setup from defaults + mode overlays +- ✅ **Automatic TOML Export**: Generates TOML files for Rust services +- ✅ **Runtime Detection**: Detect existing configs and offer update/replace options +- ✅ **Batch Operations**: Configure all 8 services at once +- ✅ **Cleanup Management**: Remove/reset configurations safely + +## Usage + +### Interactive Setup (Recommended) + +```bash +# Start interactive wizard +./provisioning/scripts/setup-platform-config.sh + +# Prompts for: +# 1. Action: TypeDialog, Quick Mode, Clean, or List +# 2. Service (if TypeDialog/Quick) +# 3. Mode (solo/multiuser/cicd/enterprise) +# 4. Backend (web/tui/cli, if TypeDialog) +``` + +### Command-Line Options + +```bash +# Configure specific service via TypeDialog +./provisioning/scripts/setup-platform-config.sh + --service orchestrator + --mode solo + --backend web + +# Quick setup all services for enterprise mode +./provisioning/scripts/setup-platform-config.sh + --quick-mode + --mode enterprise + +# Regenerate TOML files from existing .ncl configs +./provisioning/scripts/setup-platform-config.sh + --generate-toml + +# List available options +./provisioning/scripts/setup-platform-config.sh --list-modes +./provisioning/scripts/setup-platform-config.sh --list-services +./provisioning/scripts/setup-platform-config.sh --list-configs + +# Clean all runtime configurations +./provisioning/scripts/setup-platform-config.sh --clean +``` + +## Workflow + +### 1. Initial Setup (Empty Runtime) + +```bash +Interactive Prompt + ↓ +├─ TypeDialog (Recommended) +│ ├─ Load form definitions +│ ├─ User fills form (web/tui/cli) +│ └─ Generates orchestrator.solo.ncl +│ ↓ +│ Auto-export to orchestrator.solo.toml +│ +└─ Quick Mode + ├─ Select mode (solo/multiuser/cicd/enterprise) + ├─ Compose all services: defaults + mode overlay + ├─ Create 8 .ncl files + └─ Auto-export to 8 .toml files +``` + +### 2. Update Existing Configuration + +```toml +Detect Existing Config + ↓ +Choose Action: + ├─ Clean up & start fresh + ├─ Update via TypeDialog (edit existing) + ├─ Use quick mode (regenerate) + └─ List current configs +``` + +### 3. Manual NCL Edits + +```bash +User edits: provisioning/config/runtime/orchestrator.solo.ncl + ↓ +Run: ./setup-platform-config.sh --generate-toml + ↓ +Auto-exports to: provisioning/config/runtime/generated/orchestrator.solo.toml + ↓ +Service loads TOML automatically +``` + +## Configuration Layers + +The script composes configurations from multiple layers: + +```toml +1. Schema (TYPE-SAFE CONTRACT) + ↓ + provisioning/schemas/platform/schemas/orchestrator.ncl + (Defines valid fields, types, constraints) + +2. Service Defaults (BASE VALUES) + ↓ + provisioning/schemas/platform/defaults/orchestrator-defaults.ncl + (Default values for all orchestrator settings) + +3. Mode Overlay (MODE-SPECIFIC TUNING) + ↓ + provisioning/schemas/platform/defaults/deployment/solo-defaults.ncl + (Resource limits for solo mode: 2 CPU, 4GB RAM) + +4. Composition (MERGE) + ↓ + defaults.merge_config_with_mode(mode_config) + (Merges base + mode overlay) + +5. Runtime Config (USER CUSTOMIZATION) + ↓ + provisioning/config/runtime/orchestrator.solo.ncl + (Final config, can be hand-edited) + +6. TOML Export (SERVICE CONSUMPTION) + ↓ + provisioning/config/runtime/generated/orchestrator.solo.toml + (Rust service reads this) +``` + +## Services & Modes + +### 8 Available Services + +```bash +1. orchestrator - Main orchestration engine +2. control-center - Web UI and management console +3. mcp-server - Model Context Protocol server +4. vault-service - Secrets management and encryption +5. extension-registry - Extension distribution system +6. rag - Retrieval-Augmented Generation +7. ai-service - AI model integration +8. provisioning-daemon - Background operations +``` + +### 4 Deployment Modes + +| Mode | Specs | Use Case | +| ------ | ------- | ---------- | +| `solo` | 2 CPU, 4GB RAM | Development, testing | +| `multiuser` | 4 CPU, 8GB RAM | Team staging | +| `cicd` | 8 CPU, 16GB RAM | CI/CD pipelines | +| `enterprise` | 16+ CPU, 32+ GB | Production HA | + +## Directory Structure + +```bash +provisioning/ +├── config/ +│ └── runtime/ # 🔒 PRIVATE (gitignored) +│ ├── .gitignore +│ ├── orchestrator.solo.ncl # Runtime config (user editable) +│ ├── vault-service.multiuser.ncl # Runtime config +│ └── generated/ # TOMLs (auto-generated) +│ ├── orchestrator.solo.toml # For Rust services +│ └── vault-service.multiuser.toml +│ +├── schemas/platform/ # 📘 PUBLIC (versionable) +│ ├── schemas/ # Type contracts +│ ├── defaults/ # Base values +│ │ ├── orchestrator-defaults.ncl +│ │ └── deployment/ +│ │ ├── solo-defaults.ncl +│ │ ├── multiuser-defaults.ncl +│ │ ├── cicd-defaults.ncl +│ │ └── enterprise-defaults.ncl +│ └── validators/ # Business logic +│ +└── scripts/ + └── setup-platform-config.sh # This script +``` + +## Requirements + +- **Bash 4.0+** +- **Nickel 0.10+** - Configuration language +- **Nushell 0.109+** - Script engine +- **TypeDialog** (optional, for interactive setup) + +## Integration with Provisioning Installer + +### ⚠️ Current Status: Installer NOT YET IMPLEMENTED + +The `setup-platform-config.sh` script is a **standalone tool** ready to use independently. + +**For now**: Call the script manually before running services + +```bash +# Step 1: Setup platform configurations (MANUAL) +./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo + +# Step 2: Run services +export ORCHESTRATOR_MODE=solo +cargo run -p orchestrator +``` + +### Future: When Installer is Implemented + +Once `provisioning/scripts/install.sh` is ready, it will look like: + +```bash +#!/bin/bash +# provisioning/scripts/install.sh (FUTURE) + +# Pre-flight checks +check_dependencies() { + command -v nickel >/dev/null || { echo "Nickel required"; exit 1; } + command -v nu >/dev/null || { echo "Nushell required"; exit 1; } +} +check_dependencies + +# Install provisioning system +echo "Installing provisioning system..." +# (implementation here) + +# Setup platform configurations +echo "Setting up platform configurations..." +./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo + +# Build and verify +echo "Building platform services..." +cargo build -p orchestrator -p control-center -p mcp-server + +echo "Installation complete!" +``` + +### CI/CD Integration (Available Now) + +For CI/CD pipelines that don't require the full installer: + +```bash +#!/bin/bash +# ci/setup.sh + +# Setup configurations for CI/CD mode +./provisioning/scripts/setup-platform-config.sh + --quick-mode + --mode cicd + +# Run tests +cargo test --all + +# Deploy +docker-compose -f provisioning/platform/infrastructure/docker/docker-compose.cicd.yml up +``` + +## Important Notes + +### ⚠️ Manual Edits + +If you manually edit `.ncl` files in `provisioning/config/runtime/`: + +```nickel +# Always regenerate TOMLs afterward +./provisioning/scripts/setup-platform-config.sh --generate-toml +``` + +### 🔒 Private Configurations + +Files in `provisioning/config/runtime/` are **gitignored**: +- `.ncl` files (may contain secrets/encrypted values) +- `generated/*.toml` files (auto-generated, no need to version) + +### 📘 Public Schemas + +Schemas in `provisioning/schemas/platform/` are **public**: +- Source of truth for configuration structure +- Versionable and shared across team +- Can be committed to git + +### 🔄 Regeneration + +The script is **idempotent** - run it multiple times safely: + +```bash +# Safe: Re-runs setup, updates configs +./provisioning/scripts/setup-platform-config.sh --quick-mode --mode multiuser + +# Does NOT overwrite manually edited files (unless --clean is used) +``` + +## Troubleshooting + +### Nickel Validation Fails + +```nickel +# Check syntax of generated config +nickel typecheck provisioning/config/runtime/orchestrator.solo.ncl + +# View detailed error +nickel export --format toml provisioning/config/runtime/orchestrator.solo.ncl +``` + +### TOML Export Fails + +```toml +# Check if Nickel config is valid +nickel typecheck provisioning/config/runtime/orchestrator.solo.ncl + +# Try manual export +nickel export --format toml provisioning/config/runtime/orchestrator.solo.ncl +``` + +### Service Won't Start + +```bash +# Verify TOML exists +ls -la provisioning/config/runtime/generated/orchestrator.solo.toml + +# Check TOML syntax +cat provisioning/config/runtime/generated/orchestrator.solo.toml | head -20 + +# Verify service can read TOML +ORCHESTRATOR_MODE=solo cargo run -p orchestrator -- +``` + +## Examples + +### Example 1: Quick Setup for Development + +```bash +# Setup all services for solo mode (2 CPU, 4GB RAM) +./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo + +# Result: +# ✓ Created provisioning/config/runtime/orchestrator.solo.ncl +# ✓ Created provisioning/config/runtime/control-center.solo.ncl +# ✓ ... (8 services total) +# ✓ Generated 8 TOML files in provisioning/config/runtime/generated/ + +# Run service: +export ORCHESTRATOR_MODE=solo +cargo run -p orchestrator +``` + +### Example 2: Interactive TypeDialog Setup + +```bash +# Configure orchestrator in multiuser mode with web UI +./provisioning/scripts/setup-platform-config.sh + --service orchestrator + --mode multiuser + --backend web + +# TypeDialog opens browser, user fills form +# Result: +# ✓ Created provisioning/config/runtime/orchestrator.multiuser.ncl +# ✓ Generated provisioning/config/runtime/generated/orchestrator.multiuser.toml + +# Run service: +export ORCHESTRATOR_MODE=multiuser +cargo run -p orchestrator +``` + +### Example 3: Update After Manual Edit + +```bash +# Edit config manually +vim provisioning/config/runtime/orchestrator.solo.ncl + +# Regenerate TOML (critical!) +./provisioning/scripts/setup-platform-config.sh --generate-toml + +# Verify changes +cat provisioning/config/runtime/generated/orchestrator.solo.toml | head -20 + +# Restart service with new config +pkill orchestrator +ORCHESTRATOR_MODE=solo cargo run -p orchestrator +``` + +## Performance Notes + +- **Quick Mode**: ~1-2 seconds (no user interaction, direct composition) +- **TypeDialog (web)**: 10-30 seconds (server startup + UI loading) +- **TOML Export**: <1 second per service +- **Full Setup**: 5-10 minutes (8 services via TypeDialog) + +--- + +**Version**: 1.0.0 +**Created**: 2026-01-05 +**Last Updated**: 2026-01-05 \ No newline at end of file diff --git a/tests/integration/README.md b/tests/integration/README.md index 373b5ae..1103784 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -1 +1,588 @@ -# Integration Testing Suite\n\n**Version**: 1.0.0\n**Status**: ✅ Complete\n**Test Coverage**: 140 tests across 4 modes, 15+ services\n\n---\n\n## Overview\n\nThis directory contains the comprehensive integration testing suite for the provisioning platform. Tests validate all four execution modes (solo, multi-user, CI/CD, enterprise) with full service integration, workflow testing, and end-to-end scenarios.\n\n**Key Features**:\n\n- ✅ **4 Execution Modes**: Solo, Multi-User, CI/CD, Enterprise\n- ✅ **15+ Services**: Orchestrator, CoreDNS, Gitea, OCI registries, PostgreSQL, Prometheus, etc.\n- ✅ **OrbStack Integration**: Deployable to isolated OrbStack machine\n- ✅ **Parallel Execution**: Run tests in parallel for speed\n- ✅ **Multiple Report Formats**: JUnit XML, HTML, JSON\n- ✅ **Automatic Cleanup**: Resources cleaned up after tests\n\n---\n\n## Quick Start\n\n### 1. Prerequisites\n\n```\n# Install OrbStack\nbrew install --cask orbstack\n\n# Create OrbStack machine\norb create provisioning --cpu 4 --memory 8192 --disk 100\n\n# Verify machine is running\norb status provisioning\n```\n\n### 2. Run Tests\n\n```\n# Run all tests for solo mode\nnu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n# Run all tests for all modes\nnu provisioning/tests/integration/framework/test_runner.nu\n\n# Run with HTML report\nnu provisioning/tests/integration/framework/test_runner.nu --report test-report.html\n```\n\n### 3. View Results\n\n```\n# View JUnit report\ncat /tmp/provisioning-test-reports/junit-results.xml\n\n# View HTML report\nopen test-report.html\n\n# View logs\ncat /tmp/provisioning-test.log\n```\n\n---\n\n## Directory Structure\n\n```\nprovisioning/tests/integration/\n├── README.md # This file\n├── test_config.yaml # Test configuration\n├── setup_test_environment.nu # Environment setup\n├── teardown_test_environment.nu # Cleanup script\n├── framework/ # Test framework\n│ ├── test_helpers.nu # Common utilities (400 lines)\n│ ├── orbstack_helpers.nu # OrbStack integration (250 lines)\n│ └── test_runner.nu # Test orchestrator (500 lines)\n├── modes/ # Mode-specific tests\n│ ├── test_solo_mode.nu # Solo mode (400 lines, 8 tests)\n│ ├── test_multiuser_mode.nu # Multi-user (500 lines, 10 tests)\n│ ├── test_cicd_mode.nu # CI/CD (450 lines, 8 tests)\n│ └── test_enterprise_mode.nu # Enterprise (600 lines, 6 tests)\n├── services/ # Service integration tests\n│ ├── test_dns_integration.nu # CoreDNS (300 lines, 8 tests)\n│ ├── test_gitea_integration.nu # Gitea (350 lines, 10 tests)\n│ ├── test_oci_integration.nu # OCI registries (400 lines, 12 tests)\n│ └── test_service_orchestration.nu # Service manager (350 lines, 10 tests)\n├── workflows/ # Workflow tests\n│ ├── test_extension_loading.nu # Extension loading (400 lines, 12 tests)\n│ └── test_batch_workflows.nu # Batch workflows (500 lines, 12 tests)\n├── e2e/ # End-to-end tests\n│ ├── test_complete_deployment.nu # Full deployment (600 lines, 6 tests)\n│ └── test_disaster_recovery.nu # Backup/restore (400 lines, 6 tests)\n├── performance/ # Performance tests\n│ ├── test_concurrency.nu # Concurrency (350 lines, 6 tests)\n│ └── test_scalability.nu # Scalability (300 lines, 6 tests)\n├── security/ # Security tests\n│ ├── test_rbac_enforcement.nu # RBAC (400 lines, 10 tests)\n│ └── test_kms_integration.nu # KMS (300 lines, 5 tests)\n└── docs/ # Documentation\n ├── TESTING_GUIDE.md # Complete testing guide (800 lines)\n ├── ORBSTACK_SETUP.md # OrbStack setup (300 lines)\n └── TEST_COVERAGE.md # Coverage report (400 lines)\n```\n\n**Total**: ~7,500 lines of test code + ~1,500 lines of documentation\n\n---\n\n## Test Modes\n\n### Solo Mode (8 Tests)\n\n**Services**: Orchestrator, CoreDNS, Zot OCI registry\n\n**Tests**:\n\n- ✅ Minimal services running\n- ✅ Single-user operations (no auth)\n- ✅ No multi-user services\n- ✅ Workspace creation\n- ✅ Server deployment with DNS registration\n- ✅ Taskserv installation\n- ✅ Extension loading from OCI\n- ✅ Admin permissions\n\n**Run**:\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu --mode solo\n```\n\n### Multi-User Mode (10 Tests)\n\n**Services**: Solo services + Gitea, PostgreSQL\n\n**Tests**:\n\n- ✅ Multi-user services running\n- ✅ User authentication\n- ✅ Role-based permissions (viewer, developer, operator, admin)\n- ✅ Workspace collaboration (clone, push, pull)\n- ✅ Distributed locking via Gitea issues\n- ✅ Concurrent operations\n- ✅ Extension publishing to Gitea\n- ✅ Extension downloading from Gitea\n- ✅ DNS for multiple servers\n- ✅ User isolation\n\n**Run**:\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu --mode multiuser\n```\n\n### CI/CD Mode (8 Tests)\n\n**Services**: Multi-user services + API server, Prometheus\n\n**Tests**:\n\n- ✅ API server accessibility\n- ✅ Service account JWT authentication\n- ✅ API server creation\n- ✅ API taskserv installation\n- ✅ Batch workflow submission via API\n- ✅ Remote workflow monitoring\n- ✅ Automated deployment pipeline\n- ✅ Prometheus metrics collection\n\n**Run**:\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu --mode cicd\n```\n\n### Enterprise Mode (6 Tests)\n\n**Services**: CI/CD services + Harbor, Grafana, KMS, Elasticsearch\n\n**Tests**:\n\n- ✅ All enterprise services running (Harbor, Grafana, Prometheus, KMS)\n- ✅ SSH keys stored in KMS\n- ✅ Full RBAC enforcement\n- ✅ Audit logging for all operations\n- ✅ Harbor OCI registry operational\n- ✅ Monitoring stack (Prometheus + Grafana)\n\n**Run**:\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu --mode enterprise\n```\n\n---\n\n## Service Integration Tests\n\n### CoreDNS Integration (8 Tests)\n\n- DNS registration on server creation\n- DNS resolution\n- DNS cleanup on server deletion\n- DNS updates on IP change\n- External DNS queries\n- Multiple server DNS records\n- Zone transfers (if enabled)\n- DNS caching\n\n### Gitea Integration (10 Tests)\n\n- Gitea initialization\n- Workspace git clone\n- Workspace git push\n- Workspace git pull\n- Distributed locking (acquire/release)\n- Extension publishing to releases\n- Extension downloading from releases\n- Gitea webhooks\n- Gitea API access\n\n### OCI Registry Integration (12 Tests)\n\n- Zot registry (solo/multi-user modes)\n- Harbor registry (enterprise mode)\n- Push/pull KCL packages\n- Push/pull extension artifacts\n- List artifacts\n- Verify manifests\n- Delete artifacts\n- Authentication\n- Catalog API\n- Blob upload\n\n### Orchestrator Integration (10 Tests)\n\n- Health endpoint\n- Task submission\n- Task status queries\n- Task completion\n- Failure handling\n- Retry logic\n- Task queue processing\n- Workflow submission\n- Workflow monitoring\n- REST API endpoints\n\n---\n\n## Workflow Tests\n\n### Extension Loading (12 Tests)\n\n- Load taskserv from OCI\n- Load provider from Gitea\n- Load cluster from local path\n- Dependency resolution\n- Version conflict resolution\n- Extension caching\n- Lazy loading\n- Semver version resolution\n- Extension updates\n- Extension rollback\n- Multi-source loading\n- Extension validation\n\n### Batch Workflows (12 Tests)\n\n- Batch submission\n- Batch status queries\n- Batch monitoring\n- Multi-server creation\n- Multi-taskserv installation\n- Cluster deployment\n- Mixed providers (AWS + UpCloud + local)\n- Dependency resolution\n- Rollback on failure\n- Partial failure handling\n- Parallel execution\n- Checkpoint recovery\n\n---\n\n## End-to-End Tests\n\n### Complete Deployment (6 Tests)\n\n**Scenario**: Deploy 3-node Kubernetes cluster from scratch\n\n1. Initialize workspace\n2. Load extensions (containerd, etcd, kubernetes, cilium)\n3. Create 3 servers (1 control-plane, 2 workers)\n4. Verify DNS registration\n5. Install containerd on all servers\n6. Install etcd on control-plane\n7. Install kubernetes on all servers\n8. Install cilium for networking\n9. Verify cluster health\n10. Deploy test application\n11. Verify application accessible via DNS\n12. Cleanup\n\n### Disaster Recovery (6 Tests)\n\n- Workspace backup\n- Data loss simulation\n- Workspace restore\n- Data integrity verification\n- Platform service backup\n- Platform service restore\n\n---\n\n## Performance Tests\n\n### Concurrency (6 Tests)\n\n- 10 concurrent server creations\n- 20 concurrent DNS registrations\n- 5 concurrent workflow submissions\n- Throughput measurement\n- Latency measurement\n- Resource contention handling\n\n### Scalability (6 Tests)\n\n- 100 server creations\n- 100 taskserv installations\n- 100 DNS records\n- 1000 OCI artifacts\n- Performance degradation analysis\n- Resource usage tracking\n\n---\n\n## Security Tests\n\n### RBAC Enforcement (10 Tests)\n\n- Viewer cannot create servers\n- Developer can deploy to dev, not prod\n- Operator can manage infrastructure\n- Admin has full access\n- Service account automation permissions\n- Role escalation prevention\n- Permission inheritance\n- Workspace isolation\n- API endpoint authorization\n- CLI command authorization\n\n### KMS Integration (5 Tests)\n\n- SSH key storage\n- SSH key retrieval\n- SSH key usage for server access\n- SSH key rotation\n- Audit logging for key access\n\n---\n\n## Test Runner Options\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu [OPTIONS]\n```\n\n**Options**:\n\n| Option | Description | Example |\n| -------- | ------------- | --------- |\n| `--mode ` | Test specific mode (solo, multiuser, cicd, enterprise) | `--mode solo` |\n| `--filter ` | Filter tests by regex pattern | `--filter "dns"` |\n| `--parallel ` | Number of parallel workers | `--parallel 4` |\n| `--verbose` | Detailed output | `--verbose` |\n| `--report ` | Generate HTML report | `--report test-report.html` |\n| `--skip-setup` | Skip environment setup | `--skip-setup` |\n| `--skip-teardown` | Skip environment teardown (for debugging) | `--skip-teardown` |\n\n**Examples**:\n\n```\n# Run all tests for all modes (sequential)\nnu provisioning/tests/integration/framework/test_runner.nu\n\n# Run solo mode tests only\nnu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n# Run DNS-related tests across all modes\nnu provisioning/tests/integration/framework/test_runner.nu --filter "dns"\n\n# Run tests in parallel with 4 workers\nnu provisioning/tests/integration/framework/test_runner.nu --parallel 4\n\n# Generate HTML report\nnu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html\n\n# Run tests without cleanup (for debugging failures)\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown\n```\n\n---\n\n## CI/CD Integration\n\n### GitHub Actions\n\nSee `.github/workflows/integration-tests.yml` for complete workflow.\n\n**Trigger**: PR, push to main, nightly\n\n**Matrix**: All 4 modes tested in parallel\n\n**Artifacts**: Test reports, logs uploaded on failure\n\n### GitLab CI\n\nSee `.gitlab-ci.yml` for complete configuration.\n\n**Stages**: Test\n\n**Parallel**: All 4 modes\n\n**Artifacts**: JUnit XML, HTML reports\n\n---\n\n## Test Results\n\n### Expected Duration\n\n| Mode | Sequential | Parallel (4 workers) |\n| ------ | ------------ | ---------------------- |\n| Solo | 10 min | 3 min |\n| Multi-User | 15 min | 4 min |\n| CI/CD | 20 min | 5 min |\n| Enterprise | 30 min | 8 min |\n| **Total** | **75 min** | **20 min** |\n\n### Report Formats\n\n**JUnit XML**: `/tmp/provisioning-test-reports/junit-results.xml`\n\n- For CI/CD integration\n- Compatible with all CI systems\n\n**HTML Report**: Generated with `--report` flag\n\n- Beautiful visual report\n- Test details, duration, errors\n- Pass/fail summary\n\n**JSON Report**: `/tmp/provisioning-test-reports/test-results.json`\n\n- Machine-readable format\n- For custom analysis\n\n---\n\n## Troubleshooting\n\n### Common Issues\n\n**OrbStack machine not found**:\n\n```\norb create provisioning --cpu 4 --memory 8192\n```\n\n**Docker connection failed**:\n\n```\norb restart provisioning\ndocker -H /var/run/docker.sock ps\n```\n\n**Service health check timeout**:\n\n```\n# Check logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# Increase timeout in test_config.yaml\n# test_execution.timeouts.test_timeout_seconds: 600\n```\n\n**Test environment cleanup failed**:\n\n```\n# Manual cleanup\nnu provisioning/tests/integration/teardown_test_environment.nu --force\n```\n\n**For more troubleshooting**, see [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md#troubleshooting)\n\n---\n\n## Documentation\n\n- **[TESTING_GUIDE.md](docs/TESTING_GUIDE.md)**: Complete testing guide (800 lines)\n- **[ORBSTACK_SETUP.md](docs/ORBSTACK_SETUP.md)**: OrbStack machine setup (300 lines)\n- **[TEST_COVERAGE.md](docs/TEST_COVERAGE.md)**: Coverage report (400 lines)\n\n---\n\n## Contributing\n\n### Writing New Tests\n\n1. **Choose appropriate directory**: `modes/`, `services/`, `workflows/`, `e2e/`, `performance/`, `security/`\n2. **Follow naming convention**: `test__.nu`\n3. **Use test helpers**: Import from `framework/test_helpers.nu`\n4. **Add assertions**: Use `assert-*` helpers\n5. **Cleanup resources**: Always cleanup, even on failure\n6. **Update coverage**: Add test to TEST_COVERAGE.md\n\n### Example Test\n\n```\nuse std log\nuse ../framework/test_helpers.nu *\n\ndef test-my-feature [test_config: record] {\n run-test "my-feature-test" {\n log info "Testing my feature..."\n\n # Setup\n let resource = create-test-resource\n\n # Test\n let result = perform-operation $resource\n\n # Assert\n assert-eq $result.status "success"\n\n # Cleanup\n cleanup-test-resource $resource\n\n log info "✓ My feature works"\n }\n}\n```\n\n---\n\n## Metrics\n\n### Test Suite Statistics\n\n- **Total Tests**: 140\n- **Total Lines of Code**: ~7,500\n- **Documentation Lines**: ~1,500\n- **Coverage**: 88.5% (Rust orchestrator code)\n- **Flaky Tests**: 0%\n- **Success Rate**: 99.8%\n\n### Bug Detection\n\n- **Bugs Caught by Integration Tests**: 92%\n- **Bugs Caught by Unit Tests**: 90%\n- **Bugs Found in Production**: 2.7%\n\n---\n\n## License\n\nSame as provisioning platform (check root LICENSE file)\n\n---\n\n## Maintainers\n\nPlatform Team\n\n**Last Updated**: 2025-10-06\n**Next Review**: 2025-11-06\n\n---\n\n## Quick Links\n\n- [Setup OrbStack](docs/ORBSTACK_SETUP.md#creating-the-provisioning-machine)\n- [Run First Test](docs/TESTING_GUIDE.md#quick-start)\n- [Writing Tests](docs/TESTING_GUIDE.md#writing-new-tests)\n- [CI/CD Integration](docs/TESTING_GUIDE.md#cicd-integration)\n- [Troubleshooting](docs/TESTING_GUIDE.md#troubleshooting)\n- [Test Coverage Report](docs/TEST_COVERAGE.md) +# Integration Testing Suite + +**Version**: 1.0.0 +**Status**: ✅ Complete +**Test Coverage**: 140 tests across 4 modes, 15+ services + +--- + +## Overview + +This directory contains the comprehensive integration testing suite for the provisioning platform. Tests validate all four execution modes (solo, multi-user, CI/CD, enterprise) with full service integration, workflow testing, and end-to-end scenarios. + +**Key Features**: + +- ✅ **4 Execution Modes**: Solo, Multi-User, CI/CD, Enterprise +- ✅ **15+ Services**: Orchestrator, CoreDNS, Gitea, OCI registries, PostgreSQL, Prometheus, etc. +- ✅ **OrbStack Integration**: Deployable to isolated OrbStack machine +- ✅ **Parallel Execution**: Run tests in parallel for speed +- ✅ **Multiple Report Formats**: JUnit XML, HTML, JSON +- ✅ **Automatic Cleanup**: Resources cleaned up after tests + +--- + +## Quick Start + +### 1. Prerequisites + +```bash +# Install OrbStack +brew install --cask orbstack + +# Create OrbStack machine +orb create provisioning --cpu 4 --memory 8192 --disk 100 + +# Verify machine is running +orb status provisioning +``` + +### 2. Run Tests + +```bash +# Run all tests for solo mode +nu provisioning/tests/integration/framework/test_runner.nu --mode solo + +# Run all tests for all modes +nu provisioning/tests/integration/framework/test_runner.nu + +# Run with HTML report +nu provisioning/tests/integration/framework/test_runner.nu --report test-report.html +``` + +### 3. View Results + +```bash +# View JUnit report +cat /tmp/provisioning-test-reports/junit-results.xml + +# View HTML report +open test-report.html + +# View logs +cat /tmp/provisioning-test.log +``` + +--- + +## Directory Structure + +```bash +provisioning/tests/integration/ +├── README.md # This file +├── test_config.yaml # Test configuration +├── setup_test_environment.nu # Environment setup +├── teardown_test_environment.nu # Cleanup script +├── framework/ # Test framework +│ ├── test_helpers.nu # Common utilities (400 lines) +│ ├── orbstack_helpers.nu # OrbStack integration (250 lines) +│ └── test_runner.nu # Test orchestrator (500 lines) +├── modes/ # Mode-specific tests +│ ├── test_solo_mode.nu # Solo mode (400 lines, 8 tests) +│ ├── test_multiuser_mode.nu # Multi-user (500 lines, 10 tests) +│ ├── test_cicd_mode.nu # CI/CD (450 lines, 8 tests) +│ └── test_enterprise_mode.nu # Enterprise (600 lines, 6 tests) +├── services/ # Service integration tests +│ ├── test_dns_integration.nu # CoreDNS (300 lines, 8 tests) +│ ├── test_gitea_integration.nu # Gitea (350 lines, 10 tests) +│ ├── test_oci_integration.nu # OCI registries (400 lines, 12 tests) +│ └── test_service_orchestration.nu # Service manager (350 lines, 10 tests) +├── workflows/ # Workflow tests +│ ├── test_extension_loading.nu # Extension loading (400 lines, 12 tests) +│ └── test_batch_workflows.nu # Batch workflows (500 lines, 12 tests) +├── e2e/ # End-to-end tests +│ ├── test_complete_deployment.nu # Full deployment (600 lines, 6 tests) +│ └── test_disaster_recovery.nu # Backup/restore (400 lines, 6 tests) +├── performance/ # Performance tests +│ ├── test_concurrency.nu # Concurrency (350 lines, 6 tests) +│ └── test_scalability.nu # Scalability (300 lines, 6 tests) +├── security/ # Security tests +│ ├── test_rbac_enforcement.nu # RBAC (400 lines, 10 tests) +│ └── test_kms_integration.nu # KMS (300 lines, 5 tests) +└── docs/ # Documentation + ├── TESTING_GUIDE.md # Complete testing guide (800 lines) + ├── ORBSTACK_SETUP.md # OrbStack setup (300 lines) + └── TEST_COVERAGE.md # Coverage report (400 lines) +``` + +**Total**: ~7,500 lines of test code + ~1,500 lines of documentation + +--- + +## Test Modes + +### Solo Mode (8 Tests) + +**Services**: Orchestrator, CoreDNS, Zot OCI registry + +**Tests**: + +- ✅ Minimal services running +- ✅ Single-user operations (no auth) +- ✅ No multi-user services +- ✅ Workspace creation +- ✅ Server deployment with DNS registration +- ✅ Taskserv installation +- ✅ Extension loading from OCI +- ✅ Admin permissions + +**Run**: + +```nushell +nu provisioning/tests/integration/framework/test_runner.nu --mode solo +``` + +### Multi-User Mode (10 Tests) + +**Services**: Solo services + Gitea, PostgreSQL + +**Tests**: + +- ✅ Multi-user services running +- ✅ User authentication +- ✅ Role-based permissions (viewer, developer, operator, admin) +- ✅ Workspace collaboration (clone, push, pull) +- ✅ Distributed locking via Gitea issues +- ✅ Concurrent operations +- ✅ Extension publishing to Gitea +- ✅ Extension downloading from Gitea +- ✅ DNS for multiple servers +- ✅ User isolation + +**Run**: + +```nushell +nu provisioning/tests/integration/framework/test_runner.nu --mode multiuser +``` + +### CI/CD Mode (8 Tests) + +**Services**: Multi-user services + API server, Prometheus + +**Tests**: + +- ✅ API server accessibility +- ✅ Service account JWT authentication +- ✅ API server creation +- ✅ API taskserv installation +- ✅ Batch workflow submission via API +- ✅ Remote workflow monitoring +- ✅ Automated deployment pipeline +- ✅ Prometheus metrics collection + +**Run**: + +```nushell +nu provisioning/tests/integration/framework/test_runner.nu --mode cicd +``` + +### Enterprise Mode (6 Tests) + +**Services**: CI/CD services + Harbor, Grafana, KMS, Elasticsearch + +**Tests**: + +- ✅ All enterprise services running (Harbor, Grafana, Prometheus, KMS) +- ✅ SSH keys stored in KMS +- ✅ Full RBAC enforcement +- ✅ Audit logging for all operations +- ✅ Harbor OCI registry operational +- ✅ Monitoring stack (Prometheus + Grafana) + +**Run**: + +```nushell +nu provisioning/tests/integration/framework/test_runner.nu --mode enterprise +``` + +--- + +## Service Integration Tests + +### CoreDNS Integration (8 Tests) + +- DNS registration on server creation +- DNS resolution +- DNS cleanup on server deletion +- DNS updates on IP change +- External DNS queries +- Multiple server DNS records +- Zone transfers (if enabled) +- DNS caching + +### Gitea Integration (10 Tests) + +- Gitea initialization +- Workspace git clone +- Workspace git push +- Workspace git pull +- Distributed locking (acquire/release) +- Extension publishing to releases +- Extension downloading from releases +- Gitea webhooks +- Gitea API access + +### OCI Registry Integration (12 Tests) + +- Zot registry (solo/multi-user modes) +- Harbor registry (enterprise mode) +- Push/pull KCL packages +- Push/pull extension artifacts +- List artifacts +- Verify manifests +- Delete artifacts +- Authentication +- Catalog API +- Blob upload + +### Orchestrator Integration (10 Tests) + +- Health endpoint +- Task submission +- Task status queries +- Task completion +- Failure handling +- Retry logic +- Task queue processing +- Workflow submission +- Workflow monitoring +- REST API endpoints + +--- + +## Workflow Tests + +### Extension Loading (12 Tests) + +- Load taskserv from OCI +- Load provider from Gitea +- Load cluster from local path +- Dependency resolution +- Version conflict resolution +- Extension caching +- Lazy loading +- Semver version resolution +- Extension updates +- Extension rollback +- Multi-source loading +- Extension validation + +### Batch Workflows (12 Tests) + +- Batch submission +- Batch status queries +- Batch monitoring +- Multi-server creation +- Multi-taskserv installation +- Cluster deployment +- Mixed providers (AWS + UpCloud + local) +- Dependency resolution +- Rollback on failure +- Partial failure handling +- Parallel execution +- Checkpoint recovery + +--- + +## End-to-End Tests + +### Complete Deployment (6 Tests) + +**Scenario**: Deploy 3-node Kubernetes cluster from scratch + +1. Initialize workspace +2. Load extensions (containerd, etcd, kubernetes, cilium) +3. Create 3 servers (1 control-plane, 2 workers) +4. Verify DNS registration +5. Install containerd on all servers +6. Install etcd on control-plane +7. Install kubernetes on all servers +8. Install cilium for networking +9. Verify cluster health +10. Deploy test application +11. Verify application accessible via DNS +12. Cleanup + +### Disaster Recovery (6 Tests) + +- Workspace backup +- Data loss simulation +- Workspace restore +- Data integrity verification +- Platform service backup +- Platform service restore + +--- + +## Performance Tests + +### Concurrency (6 Tests) + +- 10 concurrent server creations +- 20 concurrent DNS registrations +- 5 concurrent workflow submissions +- Throughput measurement +- Latency measurement +- Resource contention handling + +### Scalability (6 Tests) + +- 100 server creations +- 100 taskserv installations +- 100 DNS records +- 1000 OCI artifacts +- Performance degradation analysis +- Resource usage tracking + +--- + +## Security Tests + +### RBAC Enforcement (10 Tests) + +- Viewer cannot create servers +- Developer can deploy to dev, not prod +- Operator can manage infrastructure +- Admin has full access +- Service account automation permissions +- Role escalation prevention +- Permission inheritance +- Workspace isolation +- API endpoint authorization +- CLI command authorization + +### KMS Integration (5 Tests) + +- SSH key storage +- SSH key retrieval +- SSH key usage for server access +- SSH key rotation +- Audit logging for key access + +--- + +## Test Runner Options + +```nushell +nu provisioning/tests/integration/framework/test_runner.nu [OPTIONS] +``` + +**Options**: + +| Option | Description | Example | +| -------- | ------------- | --------- | +| `--mode ` | Test specific mode (solo, multiuser, cicd, enterprise) | `--mode solo` | +| `--filter ` | Filter tests by regex pattern | `--filter "dns"` | +| `--parallel ` | Number of parallel workers | `--parallel 4` | +| `--verbose` | Detailed output | `--verbose` | +| `--report ` | Generate HTML report | `--report test-report.html` | +| `--skip-setup` | Skip environment setup | `--skip-setup` | +| `--skip-teardown` | Skip environment teardown (for debugging) | `--skip-teardown` | + +**Examples**: + +```bash +# Run all tests for all modes (sequential) +nu provisioning/tests/integration/framework/test_runner.nu + +# Run solo mode tests only +nu provisioning/tests/integration/framework/test_runner.nu --mode solo + +# Run DNS-related tests across all modes +nu provisioning/tests/integration/framework/test_runner.nu --filter "dns" + +# Run tests in parallel with 4 workers +nu provisioning/tests/integration/framework/test_runner.nu --parallel 4 + +# Generate HTML report +nu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html + +# Run tests without cleanup (for debugging failures) +nu provisioning/tests/integration/framework/test_runner.nu --skip-teardown +``` + +--- + +## CI/CD Integration + +### GitHub Actions + +See `.github/workflows/integration-tests.yml` for complete workflow. + +**Trigger**: PR, push to main, nightly + +**Matrix**: All 4 modes tested in parallel + +**Artifacts**: Test reports, logs uploaded on failure + +### GitLab CI + +See `.gitlab-ci.yml` for complete configuration. + +**Stages**: Test + +**Parallel**: All 4 modes + +**Artifacts**: JUnit XML, HTML reports + +--- + +## Test Results + +### Expected Duration + +| Mode | Sequential | Parallel (4 workers) | +| ------ | ------------ | ---------------------- | +| Solo | 10 min | 3 min | +| Multi-User | 15 min | 4 min | +| CI/CD | 20 min | 5 min | +| Enterprise | 30 min | 8 min | +| **Total** | **75 min** | **20 min** | + +### Report Formats + +**JUnit XML**: `/tmp/provisioning-test-reports/junit-results.xml` + +- For CI/CD integration +- Compatible with all CI systems + +**HTML Report**: Generated with `--report` flag + +- Beautiful visual report +- Test details, duration, errors +- Pass/fail summary + +**JSON Report**: `/tmp/provisioning-test-reports/test-results.json` + +- Machine-readable format +- For custom analysis + +--- + +## Troubleshooting + +### Common Issues + +**OrbStack machine not found**: + +```bash +orb create provisioning --cpu 4 --memory 8192 +``` + +**Docker connection failed**: + +```bash +orb restart provisioning +docker -H /var/run/docker.sock ps +``` + +**Service health check timeout**: + +```bash +# Check logs +nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator + +# Increase timeout in test_config.yaml +# test_execution.timeouts.test_timeout_seconds: 600 +``` + +**Test environment cleanup failed**: + +```bash +# Manual cleanup +nu provisioning/tests/integration/teardown_test_environment.nu --force +``` + +**For more troubleshooting**, see [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md#troubleshooting) + +--- + +## Documentation + +- **[TESTING_GUIDE.md](docs/TESTING_GUIDE.md)**: Complete testing guide (800 lines) +- **[ORBSTACK_SETUP.md](docs/ORBSTACK_SETUP.md)**: OrbStack machine setup (300 lines) +- **[TEST_COVERAGE.md](docs/TEST_COVERAGE.md)**: Coverage report (400 lines) + +--- + +## Contributing + +### Writing New Tests + +1. **Choose appropriate directory**: `modes/`, `services/`, `workflows/`, `e2e/`, `performance/`, `security/` +2. **Follow naming convention**: `test__.nu` +3. **Use test helpers**: Import from `framework/test_helpers.nu` +4. **Add assertions**: Use `assert-*` helpers +5. **Cleanup resources**: Always cleanup, even on failure +6. **Update coverage**: Add test to TEST_COVERAGE.md + +### Example Test + +```bash +use std log +use ../framework/test_helpers.nu * + +def test-my-feature [test_config: record] { + run-test "my-feature-test" { + log info "Testing my feature..." + + # Setup + let resource = create-test-resource + + # Test + let result = perform-operation $resource + + # Assert + assert-eq $result.status "success" + + # Cleanup + cleanup-test-resource $resource + + log info "✓ My feature works" + } +} +``` + +--- + +## Metrics + +### Test Suite Statistics + +- **Total Tests**: 140 +- **Total Lines of Code**: ~7,500 +- **Documentation Lines**: ~1,500 +- **Coverage**: 88.5% (Rust orchestrator code) +- **Flaky Tests**: 0% +- **Success Rate**: 99.8% + +### Bug Detection + +- **Bugs Caught by Integration Tests**: 92% +- **Bugs Caught by Unit Tests**: 90% +- **Bugs Found in Production**: 2.7% + +--- + +## License + +Same as provisioning platform (check root LICENSE file) + +--- + +## Maintainers + +Platform Team + +**Last Updated**: 2025-10-06 +**Next Review**: 2025-11-06 + +--- + +## Quick Links + +- [Setup OrbStack](docs/ORBSTACK_SETUP.md#creating-the-provisioning-machine) +- [Run First Test](docs/TESTING_GUIDE.md#quick-start) +- [Writing Tests](docs/TESTING_GUIDE.md#writing-new-tests) +- [CI/CD Integration](docs/TESTING_GUIDE.md#cicd-integration) +- [Troubleshooting](docs/TESTING_GUIDE.md#troubleshooting) +- [Test Coverage Report](docs/TEST_COVERAGE.md) \ No newline at end of file diff --git a/tests/integration/docs/orbstack-setup.md b/tests/integration/docs/orbstack-setup.md index 28ca4e9..9347a82 100644 --- a/tests/integration/docs/orbstack-setup.md +++ b/tests/integration/docs/orbstack-setup.md @@ -1 +1,501 @@ -# OrbStack Machine Setup Guide\n\n**Version**: 1.0.0\n**Last Updated**: 2025-10-06\n\nThis guide walks through setting up an OrbStack machine named "provisioning" for integration testing.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Prerequisites](#prerequisites)\n3. [Installing OrbStack](#installing-orbstack)\n4. [Creating the Provisioning Machine](#creating-the-provisioning-machine)\n5. [Configuring Resources](#configuring-resources)\n6. [Installing Prerequisites](#installing-prerequisites)\n7. [Deploying Platform for Testing](#deploying-platform-for-testing)\n8. [Verifying Setup](#verifying-setup)\n9. [Troubleshooting](#troubleshooting)\n\n---\n\n## Overview\n\nOrbStack is a lightweight, fast Docker and Linux environment for macOS. We use it to run integration tests in an isolated environment without \naffecting the host system.\n\n**Why OrbStack?**\n\n- ✅ **Fast**: Boots in seconds, much faster than traditional VMs\n- ✅ **Lightweight**: Uses minimal resources\n- ✅ **Native macOS Integration**: Seamless file sharing and networking\n- ✅ **Docker Compatible**: Full Docker API compatibility\n- ✅ **Easy Management**: Simple CLI for machine management\n\n---\n\n## Prerequisites\n\n- **macOS 12.0+** (Monterey or later)\n- **Homebrew** package manager\n- **4 GB+ RAM** available for OrbStack machine\n- **50 GB+ disk space** for containers and images\n\n---\n\n## Installing OrbStack\n\n### Option 1: Homebrew (Recommended)\n\n```\n# Install OrbStack via Homebrew\nbrew install --cask orbstack\n```\n\n### Option 2: Direct Download\n\n1. Download OrbStack from \n2. Open the downloaded DMG file\n3. Drag OrbStack to Applications folder\n4. Launch OrbStack from Applications\n\n### Verify Installation\n\n```\n# Check OrbStack CLI is available\norb version\n\n# Expected output:\n# OrbStack 1.x.x\n```\n\n---\n\n## Creating the Provisioning Machine\n\n### Create Machine\n\n```\n# Create machine named "provisioning"\norb create provisioning\n\n# Output:\n# Creating machine "provisioning"...\n# Machine "provisioning" created successfully\n```\n\n### Start Machine\n\n```\n# Start the machine\norb start provisioning\n\n# Verify machine is running\norb status provisioning\n\n# Output:\n# Machine: provisioning\n# State: running\n# CPU: 4 cores\n# Memory: 8192 MB\n# Disk: 100 GB\n```\n\n### List All Machines\n\n```\n# List all OrbStack machines\norb list\n\n# Output (JSON):\n# [\n# {\n# "name": "provisioning",\n# "state": "running",\n# "cpu_cores": 4,\n# "memory_mb": 8192,\n# "disk_gb": 100\n# }\n# ]\n```\n\n---\n\n## Configuring Resources\n\n### Set CPU Cores\n\n```\n# Set CPU cores to 4\norb config provisioning --cpu 4\n```\n\n### Set Memory\n\n```\n# Set memory to 8 GB (8192 MB)\norb config provisioning --memory 8192\n```\n\n### Set Disk Size\n\n```\n# Set disk size to 100 GB\norb config provisioning --disk 100\n```\n\n### Apply All Settings at Once\n\n```\n# Configure all resources during creation\norb create provisioning --cpu 4 --memory 8192 --disk 100\n```\n\n### Recommended Resources\n\n| Component | Minimum | Recommended |\n| ----------- | --------- | ------------- |\n| CPU Cores | 2 | 4 |\n| Memory | 4 GB | 8 GB |\n| Disk | 50 GB | 100 GB |\n\n**Note**: Enterprise mode tests require more resources due to additional services (Harbor, ELK, etc.)\n\n---\n\n## Installing Prerequisites\n\n### Install Docker CLI\n\nOrbStack includes Docker, but you may need the Docker CLI:\n\n```\n# Install Docker CLI via Homebrew\nbrew install docker\n\n# Verify Docker is available\ndocker version\n```\n\n### Install Nushell\n\n```\n# Install Nushell\nbrew install nushell\n\n# Verify Nushell is installed\nnu --version\n\n# Expected: 0.107.1 or later\n```\n\n### Install Additional Tools\n\n```\n# Install dig for DNS testing\nbrew install bind\n\n# Install psql for PostgreSQL testing\nbrew install postgresql@15\n\n# Install git for Gitea testing\nbrew install git\n```\n\n---\n\n## Deploying Platform for Testing\n\n### Deploy Solo Mode\n\n```\n# Navigate to project directory\ncd /Users/Akasha/project-provisioning\n\n# Deploy solo mode to OrbStack\nnu provisioning/tests/integration/setup_test_environment.nu --mode solo\n```\n\n**Deployed Services**:\n\n- Orchestrator (172.20.0.10:9090)\n- CoreDNS (172.20.0.2:53)\n- Zot OCI Registry (172.20.0.20:5000)\n\n### Deploy Multi-User Mode\n\n```\n# Deploy multi-user mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode multiuser\n```\n\n**Deployed Services**:\n\n- Solo mode services +\n- Gitea (172.20.0.30:3000)\n- PostgreSQL (172.20.0.40:5432)\n\n### Deploy CI/CD Mode\n\n```\n# Deploy CI/CD mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode cicd\n```\n\n**Deployed Services**:\n\n- Multi-user mode services +\n- API Server (enabled in orchestrator)\n- Prometheus (172.20.0.50:9090)\n\n### Deploy Enterprise Mode\n\n```\n# Deploy enterprise mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode enterprise\n```\n\n**Deployed Services**:\n\n- CI/CD mode services +\n- Harbor OCI Registry (172.20.0.21:443)\n- Grafana (172.20.0.51:3000)\n- KMS (integrated with orchestrator)\n- Elasticsearch (for audit logging)\n\n---\n\n## Verifying Setup\n\n### Verify Machine is Running\n\n```\n# Check machine status\norb status provisioning\n\n# Expected: state = "running"\n```\n\n### Verify Docker Connectivity\n\n```\n# List running containers\ndocker -H /var/run/docker.sock ps\n\n# Expected: List of running containers\n```\n\n### Verify Services are Healthy\n\n```\n# Check orchestrator health\ncurl http://172.20.0.10:9090/health\n\n# Expected: {"status": "healthy"}\n\n# Check CoreDNS\ndig @172.20.0.2 test.local\n\n# Expected: DNS query response\n\n# Check OCI registry\ncurl http://172.20.0.20:5000/v2/\n\n# Expected: {}\n```\n\n### Run Smoke Test\n\n```\n# Run a simple smoke test\nnu provisioning/tests/integration/framework/test_runner.nu --filter "health" --mode solo\n\n# Expected: All health check tests pass\n```\n\n---\n\n## Troubleshooting\n\n### Machine Won't Start\n\n**Symptom**: `orb start provisioning` fails\n\n**Solutions**:\n\n```\n# Check OrbStack daemon\nps aux | grep orbstack\n\n# Restart OrbStack app\nkillall OrbStack\nopen -a OrbStack\n\n# Recreate machine\norb delete provisioning\norb create provisioning\n```\n\n### Docker Connection Failed\n\n**Symptom**: `docker -H /var/run/docker.sock ps` fails\n\n**Solutions**:\n\n```\n# Verify Docker socket exists\nls -la /var/run/docker.sock\n\n# Check OrbStack is running\norb status provisioning\n\n# Restart machine\norb restart provisioning\n```\n\n### Network Connectivity Issues\n\n**Symptom**: Cannot connect to services\n\n**Solutions**:\n\n```\n# Check Docker network\ndocker -H /var/run/docker.sock network ls\n\n# Recreate provisioning network\ndocker -H /var/run/docker.sock network rm provisioning-net\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-create-network\n\n# Verify network exists\ndocker -H /var/run/docker.sock network inspect provisioning-net\n```\n\n### Resource Exhaustion\n\n**Symptom**: Services fail to start due to lack of resources\n\n**Solutions**:\n\n```\n# Increase machine resources\norb config provisioning --cpu 8 --memory 16384\n\n# Restart machine\norb restart provisioning\n\n# Check resource usage\ndocker -H /var/run/docker.sock stats\n```\n\n### Service Container Crashes\n\n**Symptom**: Container exits immediately after start\n\n**Solutions**:\n\n```\n# Check container logs\ndocker -H /var/run/docker.sock logs \n\n# Check container exit code\ndocker -H /var/run/docker.sock inspect | grep ExitCode\n\n# Restart container\ndocker -H /var/run/docker.sock restart \n```\n\n---\n\n## Advanced Configuration\n\n### Custom Network Subnet\n\nEdit `provisioning/tests/integration/test_config.yaml`:\n\n```\norbstack:\n network:\n subnet: "172.30.0.0/16" # Custom subnet\n gateway: "172.30.0.1"\n dns: ["172.30.0.2"]\n```\n\n### Persistent Volumes\n\n```\n# Create named volume for data persistence\ndocker -H /var/run/docker.sock volume create provisioning-data\n\n# Mount volume in container\ndocker -H /var/run/docker.sock run -v provisioning-data:/data ...\n```\n\n### SSH Access to Machine\n\n```\n# SSH into OrbStack machine\norb ssh provisioning\n\n# Now you're inside the machine\n# Install additional tools if needed\napt-get update && apt-get install -y vim curl\n```\n\n---\n\n## Cleanup\n\n### Stop Machine\n\n```\n# Stop machine (preserves data)\norb stop provisioning\n```\n\n### Delete Machine\n\n```\n# Delete machine (removes all data)\norb delete provisioning\n\n# Confirm deletion\n# This will remove all containers, volumes, and data\n```\n\n### Cleanup Docker Resources\n\n```\n# Remove all containers\ndocker -H /var/run/docker.sock rm -f $(docker -H /var/run/docker.sock ps -aq)\n\n# Remove all volumes\ndocker -H /var/run/docker.sock volume prune -f\n\n# Remove all networks\ndocker -H /var/run/docker.sock network prune -f\n```\n\n---\n\n## Best Practices\n\n1. **Regular Cleanup**: Clean up unused containers and volumes regularly\n2. **Resource Monitoring**: Monitor resource usage to prevent exhaustion\n3. **Automated Setup**: Use setup scripts for consistent environments\n4. **Version Control**: Track OrbStack machine configuration in version control\n5. **Backup Important Data**: Backup test data before major changes\n\n---\n\n## References\n\n- [OrbStack Official Documentation](https://orbstack.dev/docs)\n- [OrbStack GitHub](https://github.com/orbstack/orbstack)\n- [Docker Documentation](https://docs.docker.com)\n- [Integration Testing Guide](TESTING_GUIDE.md)\n\n---\n\n**Maintained By**: Platform Team\n**Last Updated**: 2025-10-06 +# OrbStack Machine Setup Guide + +**Version**: 1.0.0 +**Last Updated**: 2025-10-06 + +This guide walks through setting up an OrbStack machine named "provisioning" for integration testing. + +## Table of Contents + +1. [Overview](#overview) +2. [Prerequisites](#prerequisites) +3. [Installing OrbStack](#installing-orbstack) +4. [Creating the Provisioning Machine](#creating-the-provisioning-machine) +5. [Configuring Resources](#configuring-resources) +6. [Installing Prerequisites](#installing-prerequisites) +7. [Deploying Platform for Testing](#deploying-platform-for-testing) +8. [Verifying Setup](#verifying-setup) +9. [Troubleshooting](#troubleshooting) + +--- + +## Overview + +OrbStack is a lightweight, fast Docker and Linux environment for macOS. We use it to run integration tests in an isolated environment without +affecting the host system. + +**Why OrbStack?** + +- ✅ **Fast**: Boots in seconds, much faster than traditional VMs +- ✅ **Lightweight**: Uses minimal resources +- ✅ **Native macOS Integration**: Seamless file sharing and networking +- ✅ **Docker Compatible**: Full Docker API compatibility +- ✅ **Easy Management**: Simple CLI for machine management + +--- + +## Prerequisites + +- **macOS 12.0+** (Monterey or later) +- **Homebrew** package manager +- **4 GB+ RAM** available for OrbStack machine +- **50 GB+ disk space** for containers and images + +--- + +## Installing OrbStack + +### Option 1: Homebrew (Recommended) + +```bash +# Install OrbStack via Homebrew +brew install --cask orbstack +``` + +### Option 2: Direct Download + +1. Download OrbStack from +2. Open the downloaded DMG file +3. Drag OrbStack to Applications folder +4. Launch OrbStack from Applications + +### Verify Installation + +```bash +# Check OrbStack CLI is available +orb version + +# Expected output: +# OrbStack 1.x.x +``` + +--- + +## Creating the Provisioning Machine + +### Create Machine + +```bash +# Create machine named "provisioning" +orb create provisioning + +# Output: +# Creating machine "provisioning"... +# Machine "provisioning" created successfully +``` + +### Start Machine + +```bash +# Start the machine +orb start provisioning + +# Verify machine is running +orb status provisioning + +# Output: +# Machine: provisioning +# State: running +# CPU: 4 cores +# Memory: 8192 MB +# Disk: 100 GB +``` + +### List All Machines + +```bash +# List all OrbStack machines +orb list + +# Output (JSON): +# [ +# { +# "name": "provisioning", +# "state": "running", +# "cpu_cores": 4, +# "memory_mb": 8192, +# "disk_gb": 100 +# } +# ] +``` + +--- + +## Configuring Resources + +### Set CPU Cores + +```bash +# Set CPU cores to 4 +orb config provisioning --cpu 4 +``` + +### Set Memory + +```bash +# Set memory to 8 GB (8192 MB) +orb config provisioning --memory 8192 +``` + +### Set Disk Size + +```bash +# Set disk size to 100 GB +orb config provisioning --disk 100 +``` + +### Apply All Settings at Once + +```toml +# Configure all resources during creation +orb create provisioning --cpu 4 --memory 8192 --disk 100 +``` + +### Recommended Resources + +| Component | Minimum | Recommended | +| ----------- | --------- | ------------- | +| CPU Cores | 2 | 4 | +| Memory | 4 GB | 8 GB | +| Disk | 50 GB | 100 GB | + +**Note**: Enterprise mode tests require more resources due to additional services (Harbor, ELK, etc.) + +--- + +## Installing Prerequisites + +### Install Docker CLI + +OrbStack includes Docker, but you may need the Docker CLI: + +```bash +# Install Docker CLI via Homebrew +brew install docker + +# Verify Docker is available +docker version +``` + +### Install Nushell + +```nushell +# Install Nushell +brew install nushell + +# Verify Nushell is installed +nu --version + +# Expected: 0.107.1 or later +``` + +### Install Additional Tools + +```bash +# Install dig for DNS testing +brew install bind + +# Install psql for PostgreSQL testing +brew install postgresql@15 + +# Install git for Gitea testing +brew install git +``` + +--- + +## Deploying Platform for Testing + +### Deploy Solo Mode + +```bash +# Navigate to project directory +cd /Users/Akasha/project-provisioning + +# Deploy solo mode to OrbStack +nu provisioning/tests/integration/setup_test_environment.nu --mode solo +``` + +**Deployed Services**: + +- Orchestrator (172.20.0.10:9090) +- CoreDNS (172.20.0.2:53) +- Zot OCI Registry (172.20.0.20:5000) + +### Deploy Multi-User Mode + +```bash +# Deploy multi-user mode +nu provisioning/tests/integration/setup_test_environment.nu --mode multiuser +``` + +**Deployed Services**: + +- Solo mode services + +- Gitea (172.20.0.30:3000) +- PostgreSQL (172.20.0.40:5432) + +### Deploy CI/CD Mode + +```bash +# Deploy CI/CD mode +nu provisioning/tests/integration/setup_test_environment.nu --mode cicd +``` + +**Deployed Services**: + +- Multi-user mode services + +- API Server (enabled in orchestrator) +- Prometheus (172.20.0.50:9090) + +### Deploy Enterprise Mode + +```bash +# Deploy enterprise mode +nu provisioning/tests/integration/setup_test_environment.nu --mode enterprise +``` + +**Deployed Services**: + +- CI/CD mode services + +- Harbor OCI Registry (172.20.0.21:443) +- Grafana (172.20.0.51:3000) +- KMS (integrated with orchestrator) +- Elasticsearch (for audit logging) + +--- + +## Verifying Setup + +### Verify Machine is Running + +```bash +# Check machine status +orb status provisioning + +# Expected: state = "running" +``` + +### Verify Docker Connectivity + +```bash +# List running containers +docker -H /var/run/docker.sock ps + +# Expected: List of running containers +``` + +### Verify Services are Healthy + +```bash +# Check orchestrator health +curl http://172.20.0.10:9090/health + +# Expected: {"status": "healthy"} + +# Check CoreDNS +dig @172.20.0.2 test.local + +# Expected: DNS query response + +# Check OCI registry +curl http://172.20.0.20:5000/v2/ + +# Expected: {} +``` + +### Run Smoke Test + +```bash +# Run a simple smoke test +nu provisioning/tests/integration/framework/test_runner.nu --filter "health" --mode solo + +# Expected: All health check tests pass +``` + +--- + +## Troubleshooting + +### Machine Won't Start + +**Symptom**: `orb start provisioning` fails + +**Solutions**: + +```bash +# Check OrbStack daemon +ps aux | grep orbstack + +# Restart OrbStack app +killall OrbStack +open -a OrbStack + +# Recreate machine +orb delete provisioning +orb create provisioning +``` + +### Docker Connection Failed + +**Symptom**: `docker -H /var/run/docker.sock ps` fails + +**Solutions**: + +```bash +# Verify Docker socket exists +ls -la /var/run/docker.sock + +# Check OrbStack is running +orb status provisioning + +# Restart machine +orb restart provisioning +``` + +### Network Connectivity Issues + +**Symptom**: Cannot connect to services + +**Solutions**: + +```bash +# Check Docker network +docker -H /var/run/docker.sock network ls + +# Recreate provisioning network +docker -H /var/run/docker.sock network rm provisioning-net +nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-create-network + +# Verify network exists +docker -H /var/run/docker.sock network inspect provisioning-net +``` + +### Resource Exhaustion + +**Symptom**: Services fail to start due to lack of resources + +**Solutions**: + +```bash +# Increase machine resources +orb config provisioning --cpu 8 --memory 16384 + +# Restart machine +orb restart provisioning + +# Check resource usage +docker -H /var/run/docker.sock stats +``` + +### Service Container Crashes + +**Symptom**: Container exits immediately after start + +**Solutions**: + +```bash +# Check container logs +docker -H /var/run/docker.sock logs + +# Check container exit code +docker -H /var/run/docker.sock inspect | grep ExitCode + +# Restart container +docker -H /var/run/docker.sock restart +``` + +--- + +## Advanced Configuration + +### Custom Network Subnet + +Edit `provisioning/tests/integration/test_config.yaml`: + +```yaml +orbstack: + network: + subnet: "172.30.0.0/16" # Custom subnet + gateway: "172.30.0.1" + dns: ["172.30.0.2"] +``` + +### Persistent Volumes + +```bash +# Create named volume for data persistence +docker -H /var/run/docker.sock volume create provisioning-data + +# Mount volume in container +docker -H /var/run/docker.sock run -v provisioning-data:/data ... +``` + +### SSH Access to Machine + +```bash +# SSH into OrbStack machine +orb ssh provisioning + +# Now you're inside the machine +# Install additional tools if needed +apt-get update && apt-get install -y vim curl +``` + +--- + +## Cleanup + +### Stop Machine + +```bash +# Stop machine (preserves data) +orb stop provisioning +``` + +### Delete Machine + +```bash +# Delete machine (removes all data) +orb delete provisioning + +# Confirm deletion +# This will remove all containers, volumes, and data +``` + +### Cleanup Docker Resources + +```bash +# Remove all containers +docker -H /var/run/docker.sock rm -f $(docker -H /var/run/docker.sock ps -aq) + +# Remove all volumes +docker -H /var/run/docker.sock volume prune -f + +# Remove all networks +docker -H /var/run/docker.sock network prune -f +``` + +--- + +## Best Practices + +1. **Regular Cleanup**: Clean up unused containers and volumes regularly +2. **Resource Monitoring**: Monitor resource usage to prevent exhaustion +3. **Automated Setup**: Use setup scripts for consistent environments +4. **Version Control**: Track OrbStack machine configuration in version control +5. **Backup Important Data**: Backup test data before major changes + +--- + +## References + +- [OrbStack Official Documentation](https://orbstack.dev/docs) +- [OrbStack GitHub](https://github.com/orbstack/orbstack) +- [Docker Documentation](https://docs.docker.com) +- [Integration Testing Guide](TESTING_GUIDE.md) + +--- + +**Maintained By**: Platform Team +**Last Updated**: 2025-10-06 \ No newline at end of file diff --git a/tests/integration/docs/test-coverage.md b/tests/integration/docs/test-coverage.md index 305a1fa..960fc41 100644 --- a/tests/integration/docs/test-coverage.md +++ b/tests/integration/docs/test-coverage.md @@ -1 +1,400 @@ -# Integration Test Coverage Report\n\n**Version**: 1.0.0\n**Last Updated**: 2025-10-06\n**Test Suite Version**: 1.0.0\n\nThis document provides a comprehensive overview of integration test coverage for the provisioning platform.\n\n## Table of Contents\n\n1. [Summary](#summary)\n2. [Mode Coverage](#mode-coverage)\n3. [Service Coverage](#service-coverage)\n4. [Workflow Coverage](#workflow-coverage)\n5. [Edge Cases Covered](#edge-cases-covered)\n6. [Coverage Gaps](#coverage-gaps)\n7. [Future Enhancements](#future-enhancements)\n\n---\n\n## Summary\n\n### Overall Coverage\n\n| Category | Coverage | Tests | Status |\n| ---------- | ---------- | ------- | -------- |\n| **Modes** | 4/4 (100%) | 32 | ✅ Complete |\n| **Services** | 15/15 (100%) | 45 | ✅ Complete |\n| **Workflows** | 8/8 (100%) | 24 | ✅ Complete |\n| **E2E Scenarios** | 6/6 (100%) | 12 | ✅ Complete |\n| **Security** | 5/5 (100%) | 15 | ✅ Complete |\n| **Performance** | 4/4 (100%) | 12 | ✅ Complete |\n| **Total** | **42/42** | **140** | ✅ **Complete** |\n\n### Test Distribution\n\n```\nTotal Integration Tests: 140\n├── Mode Tests: 32 (23%)\n│ ├── Solo: 8\n│ ├── Multi-User: 10\n│ ├── CI/CD: 8\n│ └── Enterprise: 6\n├── Service Tests: 45 (32%)\n│ ├── DNS: 8\n│ ├── Gitea: 10\n│ ├── OCI Registry: 12\n│ ├── Orchestrator: 10\n│ └── Others: 5\n├── Workflow Tests: 24 (17%)\n│ ├── Extension Loading: 12\n│ └── Batch Workflows: 12\n├── E2E Tests: 12 (9%)\n│ ├── Complete Deployment: 6\n│ └── Disaster Recovery: 6\n├── Security Tests: 15 (11%)\n│ ├── RBAC: 10\n│ └── KMS: 5\n└── Performance Tests: 12 (8%)\n ├── Concurrency: 6\n └── Scalability: 6\n```\n\n---\n\n## Mode Coverage\n\n### Solo Mode (8 Tests) ✅\n\n| Test | Description | Status |\n| ------ | ------------- | -------- |\n| `test-minimal-services` | Verify orchestrator, CoreDNS, Zot running | ✅ Pass |\n| `test-single-user-operations` | All operations work without authentication | ✅ Pass |\n| `test-no-multiuser-services` | Gitea, PostgreSQL not running | ✅ Pass |\n| `test-workspace-creation` | Create workspace in solo mode | ✅ Pass |\n| `test-server-deployment-with-dns` | Server creation triggers DNS registration | ✅ Pass |\n| `test-taskserv-installation` | Install kubernetes taskserv | ✅ Pass |\n| `test-extension-loading-from-oci` | Load extensions from Zot registry | ✅ Pass |\n| `test-admin-permissions` | Admin has full permissions | ✅ Pass |\n\n**Coverage**: 100%\n**Critical Paths**: ✅ All covered\n**Edge Cases**: ✅ Handled\n\n### Multi-User Mode (10 Tests) ✅\n\n| Test | Description | Status |\n| ------ | ------------- | -------- |\n| `test-multiuser-services-running` | Gitea, PostgreSQL running | ✅ Pass |\n| `test-user-authentication` | Users can authenticate | ✅ Pass |\n| `test-role-based-permissions` | Roles enforced (viewer, developer, operator, admin) | ✅ Pass |\n| `test-workspace-collaboration` | Multiple users can clone/push workspaces | ✅ Pass |\n| `test-workspace-locking` | Distributed locking via Gitea issues | ✅ Pass |\n| `test-concurrent-operations` | Multiple users work simultaneously | ✅ Pass |\n| `test-extension-publishing` | Publish extensions to Gitea releases | ✅ Pass |\n| `test-extension-downloading` | Download extensions from Gitea | ✅ Pass |\n| `test-dns-multi-server` | DNS registration for multiple servers | ✅ Pass |\n| `test-user-isolation` | Users can only access their resources | ✅ Pass |\n\n**Coverage**: 100%\n**Critical Paths**: ✅ All covered\n**Edge Cases**: ✅ Handled\n\n### CI/CD Mode (8 Tests) ✅\n\n| Test | Description | Status |\n| ------ | ------------- | -------- |\n| `test-api-server-running` | API server accessible | ✅ Pass |\n| `test-service-account-auth` | Service accounts can authenticate with JWT | ✅ Pass |\n| `test-api-server-creation` | Create server via API | ✅ Pass |\n| `test-api-taskserv-installation` | Install taskserv via API | ✅ Pass |\n| `test-batch-workflow-submission` | Submit batch workflow via API | ✅ Pass |\n| `test-workflow-monitoring` | Monitor workflow progress remotely | ✅ Pass |\n| `test-automated-pipeline` | Complete automated deployment pipeline | ✅ Pass |\n| `test-prometheus-metrics` | Metrics collected and queryable | ✅ Pass |\n\n**Coverage**: 100%\n**Critical Paths**: ✅ All covered\n**Edge Cases**: ✅ Handled\n\n### Enterprise Mode (6 Tests) ✅\n\n| Test | Description | Status |\n| ------ | ------------- | -------- |\n| `test-enterprise-services-running` | Harbor, Grafana, Prometheus, KMS running | ✅ Pass |\n| `test-kms-ssh-key-storage` | SSH keys stored in KMS | ✅ Pass |\n| `test-rbac-full-enforcement` | RBAC enforced at all levels | ✅ Pass |\n| `test-audit-logging` | All operations logged | ✅ Pass |\n| `test-harbor-registry` | Harbor OCI registry operational | ✅ Pass |\n| `test-monitoring-stack` | Prometheus + Grafana operational | ✅ Pass |\n\n**Coverage**: 100%\n**Critical Paths**: ✅ All covered\n**Edge Cases**: ✅ Handled\n\n---\n\n## Service Coverage\n\n### CoreDNS (8 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-dns-registration` | Server creation triggers DNS A record | ✅ |\n| `test-dns-resolution` | DNS queries resolve correctly | ✅ |\n| `test-dns-cleanup` | DNS records removed on server deletion | ✅ |\n| `test-dns-update` | DNS records updated on IP change | ✅ |\n| `test-dns-external-query` | External clients can query DNS | ✅ |\n| `test-dns-multiple-records` | Multiple servers get unique records | ✅ |\n| `test-dns-zone-transfer` | Zone transfers work (if enabled) | ✅ |\n| `test-dns-caching` | DNS caching works correctly | ✅ |\n\n**Coverage**: 100%\n\n### Gitea (10 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-gitea-initialization` | Gitea initializes with default settings | ✅ |\n| `test-git-clone` | Clone workspace repository | ✅ |\n| `test-git-push` | Push workspace changes | ✅ |\n| `test-git-pull` | Pull workspace updates | ✅ |\n| `test-workspace-locking-acquire` | Acquire workspace lock via issue | ✅ |\n| `test-workspace-locking-release` | Release workspace lock | ✅ |\n| `test-extension-publish` | Publish extension to Gitea release | ✅ |\n| `test-extension-download` | Download extension from release | ✅ |\n| `test-gitea-webhooks` | Webhooks trigger on push | ✅ |\n| `test-gitea-api-access` | Gitea API accessible | ✅ |\n\n**Coverage**: 100%\n\n### OCI Registry (12 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-zot-registry-running` | Zot registry accessible (solo/multi-user) | ✅ |\n| `test-harbor-registry-running` | Harbor registry accessible (enterprise) | ✅ |\n| `test-oci-push-kcl-package` | Push KCL package to OCI | ✅ |\n| `test-oci-pull-kcl-package` | Pull KCL package from OCI | ✅ |\n| `test-oci-push-extension` | Push extension artifact to OCI | ✅ |\n| `test-oci-pull-extension` | Pull extension artifact from OCI | ✅ |\n| `test-oci-list-artifacts` | List artifacts in namespace | ✅ |\n| `test-oci-verify-manifest` | Verify OCI manifest contents | ✅ |\n| `test-oci-delete-artifact` | Delete artifact from registry | ✅ |\n| `test-oci-authentication` | Authentication with OCI registry | ✅ |\n| `test-oci-catalog` | Catalog API works | ✅ |\n| `test-oci-blob-upload` | Blob upload works | ✅ |\n\n**Coverage**: 100%\n\n### Orchestrator (10 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-orchestrator-health` | Health endpoint returns healthy | ✅ |\n| `test-task-submission` | Submit task to orchestrator | ✅ |\n| `test-task-status` | Query task status | ✅ |\n| `test-task-completion` | Task completes successfully | ✅ |\n| `test-task-failure-handling` | Failed tasks handled correctly | ✅ |\n| `test-task-retry` | Tasks retry on transient failure | ✅ |\n| `test-task-queue` | Task queue processes tasks in order | ✅ |\n| `test-workflow-submission` | Submit workflow | ✅ |\n| `test-workflow-monitoring` | Monitor workflow progress | ✅ |\n| `test-orchestrator-api` | REST API endpoints work | ✅ |\n\n**Coverage**: 100%\n\n### PostgreSQL (5 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-postgres-running` | PostgreSQL accessible | ✅ |\n| `test-database-creation` | Create database | ✅ |\n| `test-user-creation` | Create database user | ✅ |\n| `test-data-persistence` | Data persists across restarts | ✅ |\n| `test-connection-pool` | Connection pooling works | ✅ |\n\n**Coverage**: 100%\n\n---\n\n## Workflow Coverage\n\n### Extension Loading (12 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-load-taskserv-from-oci` | Load taskserv from OCI registry | ✅ |\n| `test-load-provider-from-gitea` | Load provider from Gitea release | ✅ |\n| `test-load-cluster-from-local` | Load cluster from local path | ✅ |\n| `test-dependency-resolution` | Resolve extension dependencies | ✅ |\n| `test-version-conflict-resolution` | Handle version conflicts | ✅ |\n| `test-extension-caching` | Cache extension artifacts | ✅ |\n| `test-extension-lazy-loading` | Extensions loaded on-demand | ✅ |\n| `test-semver-resolution` | Semver version resolution | ✅ |\n| `test-extension-update` | Update extension to newer version | ✅ |\n| `test-extension-rollback` | Rollback extension to previous version | ✅ |\n| `test-multi-source-loading` | Load from multiple sources in one workflow | ✅ |\n| `test-extension-validation` | Validate extension before loading | ✅ |\n\n**Coverage**: 100%\n\n### Batch Workflows (12 Tests) ✅\n\n| Test | Description | Coverage |\n| ------ | ------------- | ---------- |\n| `test-batch-submit` | Submit batch workflow | ✅ |\n| `test-batch-status` | Query batch status | ✅ |\n| `test-batch-monitor` | Monitor batch progress | ✅ |\n| `test-batch-multi-server-creation` | Create multiple servers in batch | ✅ |\n| `test-batch-multi-taskserv-install` | Install taskservs on multiple servers | ✅ |\n| `test-batch-cluster-deployment` | Deploy complete cluster in batch | ✅ |\n| `test-batch-mixed-providers` | Batch with AWS + UpCloud + local | ✅ |\n| `test-batch-dependencies` | Batch operations with dependencies | ✅ |\n| `test-batch-rollback` | Rollback failed batch operation | ✅ |\n| `test-batch-partial-failure` | Handle partial batch failures | ✅ |\n| `test-batch-parallel-execution` | Parallel execution within batch | ✅ |\n| `test-batch-checkpoint-recovery` | Recovery from checkpoint after failure | ✅ |\n\n**Coverage**: 100%\n\n---\n\n## Edge Cases Covered\n\n### Authentication & Authorization\n\n| Edge Case | Test Coverage | Status |\n| ----------- | --------------- | -------- |\n| Unauthenticated request | ✅ Rejected in multi-user mode | ✅ |\n| Invalid JWT token | ✅ Rejected with 401 | ✅ |\n| Expired JWT token | ✅ Rejected with 401 | ✅ |\n| Insufficient permissions | ✅ Rejected with 403 | ✅ |\n| Role escalation attempt | ✅ Blocked by RBAC | ✅ |\n\n### Resource Management\n\n| Edge Case | Test Coverage | Status |\n| ----------- | --------------- | -------- |\n| Resource exhaustion | ✅ Graceful degradation | ✅ |\n| Concurrent resource access | ✅ Locking prevents conflicts | ✅ |\n| Resource cleanup failure | ✅ Retry with backoff | ✅ |\n| Orphaned resources | ✅ Cleanup job removes | ✅ |\n\n### Network Operations\n\n| Edge Case | Test Coverage | Status |\n| ----------- | --------------- | -------- |\n| Network timeout | ✅ Retry with exponential backoff | ✅ |\n| DNS resolution failure | ✅ Fallback to IP address | ✅ |\n| Service unavailable | ✅ Circuit breaker pattern | ✅ |\n| Partial network partition | ✅ Retry and eventual consistency | ✅ |\n\n### Data Consistency\n\n| Edge Case | Test Coverage | Status |\n| ----------- | --------------- | -------- |\n| Concurrent writes | ✅ Last-write-wins with timestamps | ✅ |\n| Split-brain scenario | ✅ Distributed lock prevents | ✅ |\n| Data corruption | ✅ Checksum validation | ✅ |\n| Incomplete transactions | ✅ Rollback on failure | ✅ |\n\n---\n\n## Coverage Gaps\n\n### Known Limitations\n\n1. **Load Testing**: No tests for extreme load (1000+ concurrent requests)\n - **Impact**: Medium\n - **Mitigation**: Planned for v1.1.0\n\n2. **Disaster Recovery**: Limited testing of backup/restore under load\n - **Impact**: Low\n - **Mitigation**: Manual testing procedures documented\n\n3. **Network Partitions**: Limited testing of split-brain scenarios\n - **Impact**: Low (distributed locking mitigates)\n - **Mitigation**: Planned for v1.2.0\n\n4. **Security Penetration Testing**: No automated penetration tests\n - **Impact**: Medium\n - **Mitigation**: Annual security audit\n\n### Planned Enhancements\n\n- [ ] Chaos engineering tests (inject failures)\n- [ ] Load testing with 10,000+ concurrent operations\n- [ ] Extended disaster recovery scenarios\n- [ ] Fuzz testing for API endpoints\n- [ ] Performance regression detection\n\n---\n\n## Future Enhancements\n\n### v1.1.0 (Next Release)\n\n- **Load Testing Suite**: 1000+ concurrent operations\n- **Chaos Engineering**: Inject random failures\n- **Extended Security Tests**: Penetration testing automation\n- **Performance Benchmarks**: Baseline performance metrics\n\n### v1.2.0 (Q2 2025)\n\n- **Multi-Cloud Integration**: Test AWS + UpCloud + GCP simultaneously\n- **Network Partition Testing**: Advanced split-brain scenarios\n- **Compliance Testing**: GDPR, SOC2 compliance validation\n- **Visual Regression Testing**: UI component testing\n\n### v2.0.0 (Future)\n\n- **AI-Powered Test Generation**: Generate tests from user scenarios\n- **Property-Based Testing**: QuickCheck-style property testing\n- **Mutation Testing**: Detect untested code paths\n- **Continuous Fuzzing**: 24/7 fuzz testing\n\n---\n\n## Test Quality Metrics\n\n### Code Coverage (Orchestrator Rust Code)\n\n| Module | Coverage | Tests |\n| -------- | ---------- | ------- |\n| `main.rs` | 85% | 12 |\n| `config.rs` | 92% | 8 |\n| `queue.rs` | 88% | 10 |\n| `batch.rs` | 90% | 15 |\n| `dependency.rs` | 87% | 12 |\n| `rollback.rs` | 89% | 14 |\n| **Average** | **88.5%** | **71** |\n\n### Test Reliability\n\n- **Flaky Tests**: 0%\n- **Test Success Rate**: 99.8%\n- **Average Test Duration**: 15 minutes (full suite)\n- **Parallel Execution Speedup**: 4x (with 4 workers)\n\n### Bug Detection Rate\n\n- **Bugs Caught by Integration Tests**: 23/25 (92%)\n- **Bugs Caught by Unit Tests**: 45/50 (90%)\n- **Bugs Found in Production**: 2/75 (2.7%)\n\n---\n\n## References\n\n- [Integration Testing Guide](TESTING_GUIDE.md)\n- [OrbStack Setup Guide](ORBSTACK_SETUP.md)\n- [Platform Architecture](/docs/architecture/)\n- [CI/CD Pipeline](/.github/workflows/)\n\n---\n\n**Maintained By**: Platform Team\n**Last Updated**: 2025-10-06\n**Next Review**: 2025-11-06 +# Integration Test Coverage Report + +**Version**: 1.0.0 +**Last Updated**: 2025-10-06 +**Test Suite Version**: 1.0.0 + +This document provides a comprehensive overview of integration test coverage for the provisioning platform. + +## Table of Contents + +1. [Summary](#summary) +2. [Mode Coverage](#mode-coverage) +3. [Service Coverage](#service-coverage) +4. [Workflow Coverage](#workflow-coverage) +5. [Edge Cases Covered](#edge-cases-covered) +6. [Coverage Gaps](#coverage-gaps) +7. [Future Enhancements](#future-enhancements) + +--- + +## Summary + +### Overall Coverage + +| Category | Coverage | Tests | Status | +| ---------- | ---------- | ------- | -------- | +| **Modes** | 4/4 (100%) | 32 | ✅ Complete | +| **Services** | 15/15 (100%) | 45 | ✅ Complete | +| **Workflows** | 8/8 (100%) | 24 | ✅ Complete | +| **E2E Scenarios** | 6/6 (100%) | 12 | ✅ Complete | +| **Security** | 5/5 (100%) | 15 | ✅ Complete | +| **Performance** | 4/4 (100%) | 12 | ✅ Complete | +| **Total** | **42/42** | **140** | ✅ **Complete** | + +### Test Distribution + +```bash +Total Integration Tests: 140 +├── Mode Tests: 32 (23%) +│ ├── Solo: 8 +│ ├── Multi-User: 10 +│ ├── CI/CD: 8 +│ └── Enterprise: 6 +├── Service Tests: 45 (32%) +│ ├── DNS: 8 +│ ├── Gitea: 10 +│ ├── OCI Registry: 12 +│ ├── Orchestrator: 10 +│ └── Others: 5 +├── Workflow Tests: 24 (17%) +│ ├── Extension Loading: 12 +│ └── Batch Workflows: 12 +├── E2E Tests: 12 (9%) +│ ├── Complete Deployment: 6 +│ └── Disaster Recovery: 6 +├── Security Tests: 15 (11%) +│ ├── RBAC: 10 +│ └── KMS: 5 +└── Performance Tests: 12 (8%) + ├── Concurrency: 6 + └── Scalability: 6 +``` + +--- + +## Mode Coverage + +### Solo Mode (8 Tests) ✅ + +| Test | Description | Status | +| ------ | ------------- | -------- | +| `test-minimal-services` | Verify orchestrator, CoreDNS, Zot running | ✅ Pass | +| `test-single-user-operations` | All operations work without authentication | ✅ Pass | +| `test-no-multiuser-services` | Gitea, PostgreSQL not running | ✅ Pass | +| `test-workspace-creation` | Create workspace in solo mode | ✅ Pass | +| `test-server-deployment-with-dns` | Server creation triggers DNS registration | ✅ Pass | +| `test-taskserv-installation` | Install kubernetes taskserv | ✅ Pass | +| `test-extension-loading-from-oci` | Load extensions from Zot registry | ✅ Pass | +| `test-admin-permissions` | Admin has full permissions | ✅ Pass | + +**Coverage**: 100% +**Critical Paths**: ✅ All covered +**Edge Cases**: ✅ Handled + +### Multi-User Mode (10 Tests) ✅ + +| Test | Description | Status | +| ------ | ------------- | -------- | +| `test-multiuser-services-running` | Gitea, PostgreSQL running | ✅ Pass | +| `test-user-authentication` | Users can authenticate | ✅ Pass | +| `test-role-based-permissions` | Roles enforced (viewer, developer, operator, admin) | ✅ Pass | +| `test-workspace-collaboration` | Multiple users can clone/push workspaces | ✅ Pass | +| `test-workspace-locking` | Distributed locking via Gitea issues | ✅ Pass | +| `test-concurrent-operations` | Multiple users work simultaneously | ✅ Pass | +| `test-extension-publishing` | Publish extensions to Gitea releases | ✅ Pass | +| `test-extension-downloading` | Download extensions from Gitea | ✅ Pass | +| `test-dns-multi-server` | DNS registration for multiple servers | ✅ Pass | +| `test-user-isolation` | Users can only access their resources | ✅ Pass | + +**Coverage**: 100% +**Critical Paths**: ✅ All covered +**Edge Cases**: ✅ Handled + +### CI/CD Mode (8 Tests) ✅ + +| Test | Description | Status | +| ------ | ------------- | -------- | +| `test-api-server-running` | API server accessible | ✅ Pass | +| `test-service-account-auth` | Service accounts can authenticate with JWT | ✅ Pass | +| `test-api-server-creation` | Create server via API | ✅ Pass | +| `test-api-taskserv-installation` | Install taskserv via API | ✅ Pass | +| `test-batch-workflow-submission` | Submit batch workflow via API | ✅ Pass | +| `test-workflow-monitoring` | Monitor workflow progress remotely | ✅ Pass | +| `test-automated-pipeline` | Complete automated deployment pipeline | ✅ Pass | +| `test-prometheus-metrics` | Metrics collected and queryable | ✅ Pass | + +**Coverage**: 100% +**Critical Paths**: ✅ All covered +**Edge Cases**: ✅ Handled + +### Enterprise Mode (6 Tests) ✅ + +| Test | Description | Status | +| ------ | ------------- | -------- | +| `test-enterprise-services-running` | Harbor, Grafana, Prometheus, KMS running | ✅ Pass | +| `test-kms-ssh-key-storage` | SSH keys stored in KMS | ✅ Pass | +| `test-rbac-full-enforcement` | RBAC enforced at all levels | ✅ Pass | +| `test-audit-logging` | All operations logged | ✅ Pass | +| `test-harbor-registry` | Harbor OCI registry operational | ✅ Pass | +| `test-monitoring-stack` | Prometheus + Grafana operational | ✅ Pass | + +**Coverage**: 100% +**Critical Paths**: ✅ All covered +**Edge Cases**: ✅ Handled + +--- + +## Service Coverage + +### CoreDNS (8 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-dns-registration` | Server creation triggers DNS A record | ✅ | +| `test-dns-resolution` | DNS queries resolve correctly | ✅ | +| `test-dns-cleanup` | DNS records removed on server deletion | ✅ | +| `test-dns-update` | DNS records updated on IP change | ✅ | +| `test-dns-external-query` | External clients can query DNS | ✅ | +| `test-dns-multiple-records` | Multiple servers get unique records | ✅ | +| `test-dns-zone-transfer` | Zone transfers work (if enabled) | ✅ | +| `test-dns-caching` | DNS caching works correctly | ✅ | + +**Coverage**: 100% + +### Gitea (10 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-gitea-initialization` | Gitea initializes with default settings | ✅ | +| `test-git-clone` | Clone workspace repository | ✅ | +| `test-git-push` | Push workspace changes | ✅ | +| `test-git-pull` | Pull workspace updates | ✅ | +| `test-workspace-locking-acquire` | Acquire workspace lock via issue | ✅ | +| `test-workspace-locking-release` | Release workspace lock | ✅ | +| `test-extension-publish` | Publish extension to Gitea release | ✅ | +| `test-extension-download` | Download extension from release | ✅ | +| `test-gitea-webhooks` | Webhooks trigger on push | ✅ | +| `test-gitea-api-access` | Gitea API accessible | ✅ | + +**Coverage**: 100% + +### OCI Registry (12 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-zot-registry-running` | Zot registry accessible (solo/multi-user) | ✅ | +| `test-harbor-registry-running` | Harbor registry accessible (enterprise) | ✅ | +| `test-oci-push-kcl-package` | Push KCL package to OCI | ✅ | +| `test-oci-pull-kcl-package` | Pull KCL package from OCI | ✅ | +| `test-oci-push-extension` | Push extension artifact to OCI | ✅ | +| `test-oci-pull-extension` | Pull extension artifact from OCI | ✅ | +| `test-oci-list-artifacts` | List artifacts in namespace | ✅ | +| `test-oci-verify-manifest` | Verify OCI manifest contents | ✅ | +| `test-oci-delete-artifact` | Delete artifact from registry | ✅ | +| `test-oci-authentication` | Authentication with OCI registry | ✅ | +| `test-oci-catalog` | Catalog API works | ✅ | +| `test-oci-blob-upload` | Blob upload works | ✅ | + +**Coverage**: 100% + +### Orchestrator (10 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-orchestrator-health` | Health endpoint returns healthy | ✅ | +| `test-task-submission` | Submit task to orchestrator | ✅ | +| `test-task-status` | Query task status | ✅ | +| `test-task-completion` | Task completes successfully | ✅ | +| `test-task-failure-handling` | Failed tasks handled correctly | ✅ | +| `test-task-retry` | Tasks retry on transient failure | ✅ | +| `test-task-queue` | Task queue processes tasks in order | ✅ | +| `test-workflow-submission` | Submit workflow | ✅ | +| `test-workflow-monitoring` | Monitor workflow progress | ✅ | +| `test-orchestrator-api` | REST API endpoints work | ✅ | + +**Coverage**: 100% + +### PostgreSQL (5 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-postgres-running` | PostgreSQL accessible | ✅ | +| `test-database-creation` | Create database | ✅ | +| `test-user-creation` | Create database user | ✅ | +| `test-data-persistence` | Data persists across restarts | ✅ | +| `test-connection-pool` | Connection pooling works | ✅ | + +**Coverage**: 100% + +--- + +## Workflow Coverage + +### Extension Loading (12 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-load-taskserv-from-oci` | Load taskserv from OCI registry | ✅ | +| `test-load-provider-from-gitea` | Load provider from Gitea release | ✅ | +| `test-load-cluster-from-local` | Load cluster from local path | ✅ | +| `test-dependency-resolution` | Resolve extension dependencies | ✅ | +| `test-version-conflict-resolution` | Handle version conflicts | ✅ | +| `test-extension-caching` | Cache extension artifacts | ✅ | +| `test-extension-lazy-loading` | Extensions loaded on-demand | ✅ | +| `test-semver-resolution` | Semver version resolution | ✅ | +| `test-extension-update` | Update extension to newer version | ✅ | +| `test-extension-rollback` | Rollback extension to previous version | ✅ | +| `test-multi-source-loading` | Load from multiple sources in one workflow | ✅ | +| `test-extension-validation` | Validate extension before loading | ✅ | + +**Coverage**: 100% + +### Batch Workflows (12 Tests) ✅ + +| Test | Description | Coverage | +| ------ | ------------- | ---------- | +| `test-batch-submit` | Submit batch workflow | ✅ | +| `test-batch-status` | Query batch status | ✅ | +| `test-batch-monitor` | Monitor batch progress | ✅ | +| `test-batch-multi-server-creation` | Create multiple servers in batch | ✅ | +| `test-batch-multi-taskserv-install` | Install taskservs on multiple servers | ✅ | +| `test-batch-cluster-deployment` | Deploy complete cluster in batch | ✅ | +| `test-batch-mixed-providers` | Batch with AWS + UpCloud + local | ✅ | +| `test-batch-dependencies` | Batch operations with dependencies | ✅ | +| `test-batch-rollback` | Rollback failed batch operation | ✅ | +| `test-batch-partial-failure` | Handle partial batch failures | ✅ | +| `test-batch-parallel-execution` | Parallel execution within batch | ✅ | +| `test-batch-checkpoint-recovery` | Recovery from checkpoint after failure | ✅ | + +**Coverage**: 100% + +--- + +## Edge Cases Covered + +### Authentication & Authorization + +| Edge Case | Test Coverage | Status | +| ----------- | --------------- | -------- | +| Unauthenticated request | ✅ Rejected in multi-user mode | ✅ | +| Invalid JWT token | ✅ Rejected with 401 | ✅ | +| Expired JWT token | ✅ Rejected with 401 | ✅ | +| Insufficient permissions | ✅ Rejected with 403 | ✅ | +| Role escalation attempt | ✅ Blocked by RBAC | ✅ | + +### Resource Management + +| Edge Case | Test Coverage | Status | +| ----------- | --------------- | -------- | +| Resource exhaustion | ✅ Graceful degradation | ✅ | +| Concurrent resource access | ✅ Locking prevents conflicts | ✅ | +| Resource cleanup failure | ✅ Retry with backoff | ✅ | +| Orphaned resources | ✅ Cleanup job removes | ✅ | + +### Network Operations + +| Edge Case | Test Coverage | Status | +| ----------- | --------------- | -------- | +| Network timeout | ✅ Retry with exponential backoff | ✅ | +| DNS resolution failure | ✅ Fallback to IP address | ✅ | +| Service unavailable | ✅ Circuit breaker pattern | ✅ | +| Partial network partition | ✅ Retry and eventual consistency | ✅ | + +### Data Consistency + +| Edge Case | Test Coverage | Status | +| ----------- | --------------- | -------- | +| Concurrent writes | ✅ Last-write-wins with timestamps | ✅ | +| Split-brain scenario | ✅ Distributed lock prevents | ✅ | +| Data corruption | ✅ Checksum validation | ✅ | +| Incomplete transactions | ✅ Rollback on failure | ✅ | + +--- + +## Coverage Gaps + +### Known Limitations + +1. **Load Testing**: No tests for extreme load (1000+ concurrent requests) + - **Impact**: Medium + - **Mitigation**: Planned for v1.1.0 + +2. **Disaster Recovery**: Limited testing of backup/restore under load + - **Impact**: Low + - **Mitigation**: Manual testing procedures documented + +3. **Network Partitions**: Limited testing of split-brain scenarios + - **Impact**: Low (distributed locking mitigates) + - **Mitigation**: Planned for v1.2.0 + +4. **Security Penetration Testing**: No automated penetration tests + - **Impact**: Medium + - **Mitigation**: Annual security audit + +### Planned Enhancements + +- [ ] Chaos engineering tests (inject failures) +- [ ] Load testing with 10,000+ concurrent operations +- [ ] Extended disaster recovery scenarios +- [ ] Fuzz testing for API endpoints +- [ ] Performance regression detection + +--- + +## Future Enhancements + +### v1.1.0 (Next Release) + +- **Load Testing Suite**: 1000+ concurrent operations +- **Chaos Engineering**: Inject random failures +- **Extended Security Tests**: Penetration testing automation +- **Performance Benchmarks**: Baseline performance metrics + +### v1.2.0 (Q2 2025) + +- **Multi-Cloud Integration**: Test AWS + UpCloud + GCP simultaneously +- **Network Partition Testing**: Advanced split-brain scenarios +- **Compliance Testing**: GDPR, SOC2 compliance validation +- **Visual Regression Testing**: UI component testing + +### v2.0.0 (Future) + +- **AI-Powered Test Generation**: Generate tests from user scenarios +- **Property-Based Testing**: QuickCheck-style property testing +- **Mutation Testing**: Detect untested code paths +- **Continuous Fuzzing**: 24/7 fuzz testing + +--- + +## Test Quality Metrics + +### Code Coverage (Orchestrator Rust Code) + +| Module | Coverage | Tests | +| -------- | ---------- | ------- | +| `main.rs` | 85% | 12 | +| `config.rs` | 92% | 8 | +| `queue.rs` | 88% | 10 | +| `batch.rs` | 90% | 15 | +| `dependency.rs` | 87% | 12 | +| `rollback.rs` | 89% | 14 | +| **Average** | **88.5%** | **71** | + +### Test Reliability + +- **Flaky Tests**: 0% +- **Test Success Rate**: 99.8% +- **Average Test Duration**: 15 minutes (full suite) +- **Parallel Execution Speedup**: 4x (with 4 workers) + +### Bug Detection Rate + +- **Bugs Caught by Integration Tests**: 23/25 (92%) +- **Bugs Caught by Unit Tests**: 45/50 (90%) +- **Bugs Found in Production**: 2/75 (2.7%) + +--- + +## References + +- [Integration Testing Guide](TESTING_GUIDE.md) +- [OrbStack Setup Guide](ORBSTACK_SETUP.md) +- [Platform Architecture](/docs/architecture/) +- [CI/CD Pipeline](/.github/workflows/) + +--- + +**Maintained By**: Platform Team +**Last Updated**: 2025-10-06 +**Next Review**: 2025-11-06 \ No newline at end of file diff --git a/tests/integration/docs/testing-guide.md b/tests/integration/docs/testing-guide.md index d754e39..152b5be 100644 --- a/tests/integration/docs/testing-guide.md +++ b/tests/integration/docs/testing-guide.md @@ -1 +1,716 @@ -# Integration Testing Guide\n\n**Version**: 1.0.0\n**Last Updated**: 2025-10-06\n\nThis guide provides comprehensive documentation for the provisioning platform integration testing suite.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Test Infrastructure](#test-infrastructure)\n3. [Running Tests Locally](#running-tests-locally)\n4. [Running Tests on OrbStack](#running-tests-on-orbstack)\n5. [Writing New Tests](#writing-new-tests)\n6. [Test Organization](#test-organization)\n7. [CI/CD Integration](#cicd-integration)\n8. [Troubleshooting](#troubleshooting)\n\n---\n\n## Overview\n\nThe integration testing suite validates all four execution modes of the provisioning platform:\n\n- **Solo Mode**: Single-user, minimal services (orchestrator, CoreDNS, OCI registry)\n- **Multi-User Mode**: Multi-user support with Gitea, PostgreSQL, RBAC\n- **CI/CD Mode**: Automation mode with API server, service accounts\n- **Enterprise Mode**: Full enterprise features (Harbor, KMS, Prometheus, Grafana, ELK)\n\n### Key Features\n\n- ✅ **Comprehensive Coverage**: Tests for all 4 modes, 15+ services\n- ✅ **OrbStack Integration**: Tests deployable to OrbStack machine "provisioning"\n- ✅ **Parallel Execution**: Run independent tests in parallel for speed\n- ✅ **Automatic Cleanup**: Resources cleaned up automatically after tests\n- ✅ **Multiple Report Formats**: JUnit XML, HTML, JSON\n- ✅ **CI/CD Ready**: GitHub Actions and GitLab CI integration\n\n---\n\n## Test Infrastructure\n\n### Prerequisites\n\n1. **OrbStack Installed**:\n\n ```bash\n # Install OrbStack (macOS)\n brew install --cask orbstack\n ```\n\n2. **OrbStack Machine Named "provisioning"**:\n\n ```bash\n # Create OrbStack machine\n orb create provisioning\n\n # Verify machine is running\n orb status provisioning\n ```\n\n3. **Nushell 0.107.1+**:\n\n ```bash\n # Install Nushell\n brew install nushell\n ```\n\n4. **Docker CLI**:\n\n ```bash\n # Verify Docker is available\n docker version\n ```\n\n### Test Configuration\n\nThe test suite is configured via `provisioning/tests/integration/test_config.yaml`:\n\n```\n# OrbStack connection\norbstack:\n machine_name: "provisioning"\n connection:\n type: "docker"\n socket: "/var/run/docker.sock"\n\n# Service endpoints\nservices:\n orchestrator:\n host: "172.20.0.10"\n port: 8080\n\n coredns:\n host: "172.20.0.2"\n port: 53\n\n # ... more services\n```\n\n**Key Settings**:\n\n- `orbstack.machine_name`: Name of OrbStack machine to use\n- `services.*`: IP addresses and ports for deployed services\n- `test_execution.parallel.max_workers`: Number of parallel test workers\n- `test_execution.timeouts.*`: Timeout values for various operations\n\n---\n\n## Running Tests Locally\n\n### Quick Start\n\n1. **Setup Test Environment**:\n\n ```bash\n # Setup solo mode environment\n nu provisioning/tests/integration/setup_test_environment.nu --mode solo\n ```\n\n1. **Run Tests**:\n\n ```bash\n # Run all tests for solo mode\n nu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n # Run specific test file\n nu provisioning/tests/integration/modes/test_solo_mode.nu\n ```\n\n2. **Teardown Test Environment**:\n\n ```bash\n # Cleanup all resources\n nu provisioning/tests/integration/teardown_test_environment.nu --force\n ```\n\n### Test Runner Options\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu \n --mode # Test specific mode (solo, multiuser, cicd, enterprise)\n --filter # Filter tests by regex pattern\n --parallel # Number of parallel workers (default: 1)\n --verbose # Detailed output\n --report # Generate HTML report\n --skip-setup # Skip environment setup\n --skip-teardown # Skip environment teardown\n```\n\n**Examples**:\n\n```\n# Run all tests for all modes\nnu provisioning/tests/integration/framework/test_runner.nu\n\n# Run only solo mode tests\nnu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n# Run tests matching pattern\nnu provisioning/tests/integration/framework/test_runner.nu --filter "dns"\n\n# Run tests in parallel with 4 workers\nnu provisioning/tests/integration/framework/test_runner.nu --parallel 4\n\n# Generate HTML report\nnu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html\n\n# Run tests without cleanup (for debugging)\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown\n```\n\n---\n\n## Running Tests on OrbStack\n\n### Setup OrbStack Machine\n\n1. **Create OrbStack Machine**:\n\n ```bash\n # Create machine named "provisioning"\n orb create provisioning --cpu 4 --memory 8192 --disk 100\n\n # Verify machine is created\n orb list\n ```\n\n1. **Configure Machine**:\n\n ```bash\n # Start machine\n orb start provisioning\n\n # Verify Docker is accessible\n docker -H /var/run/docker.sock ps\n ```\n\n### Deploy Platform to OrbStack\n\nThe test setup automatically deploys platform services to OrbStack:\n\n```\n# Deploy solo mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode solo\n\n# Deploy multi-user mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode multiuser\n\n# Deploy CI/CD mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode cicd\n\n# Deploy enterprise mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode enterprise\n```\n\n**Deployed Services**:\n\n| Mode | Services |\n| ------ | ---------- |\n| Solo | Orchestrator, CoreDNS, Zot (OCI registry) |\n| Multi-User | Solo services + Gitea, PostgreSQL |\n| CI/CD | Multi-User services + API server, Prometheus |\n| Enterprise | CI/CD services + Harbor, KMS, Grafana, Elasticsearch |\n\n### Verify Deployment\n\n```\n# Check service health\nnu provisioning/tests/integration/framework/test_helpers.nu check-service-health orchestrator\n\n# View service logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# List running containers\ndocker -H /var/run/docker.sock ps\n```\n\n---\n\n## Writing New Tests\n\n### Test File Structure\n\nAll test files follow this structure:\n\n```\n# Test Description\n# Brief description of what this test validates\n\nuse std log\nuse ../framework/test_helpers.nu *\nuse ../framework/orbstack_helpers.nu *\n\n# Main test suite\nexport def main [] {\n log info "Running "\n\n let test_config = (load-test-config)\n\n mut results = []\n\n # Run all tests\n $results = ($results | append (test-case-1 $test_config))\n $results = ($results | append (test-case-2 $test_config))\n\n # Report results\n report-test-results $results\n}\n\n# Individual test case\ndef test-case-1 [test_config: record] {\n run-test "test-case-1-name" {\n log info "Testing specific functionality..."\n\n # Test logic\n let result = (some-operation)\n\n # Assertions\n assert-eq $result.status "success" "Operation should succeed"\n assert-not-empty $result.data "Result should contain data"\n\n log info "✓ Test case 1 passed"\n }\n}\n\n# Report test results\ndef report-test-results [results: list] {\n # ... reporting logic\n}\n```\n\n### Using Assertion Helpers\n\nThe test framework provides several assertion helpers:\n\n```\n# Equality assertion\nassert-eq $actual $expected "Error message if assertion fails"\n\n# Boolean assertions\nassert-true $condition "Error message"\nassert-false $condition "Error message"\n\n# Collection assertions\nassert-contains $list $item "Error message"\nassert-not-contains $list $item "Error message"\nassert-not-empty $value "Error message"\n\n# HTTP assertions\nassert-http-success $response "Error message"\n```\n\n### Using Test Fixtures\n\nCreate reusable test fixtures:\n\n```\n# Create test workspace\nlet workspace = create-test-workspace "my-test-ws" {\n provider: "local"\n environment: "test"\n}\n\n# Create test server\nlet server = create-test-server "test-server" "local" {\n cores: 4\n memory: 8192\n}\n\n# Cleanup\ncleanup-test-workspace $workspace\ndelete-test-server $server.id\n```\n\n### Using Retry Logic\n\nFor flaky operations, use retry helpers:\n\n```\n# Retry operation up to 3 times\nlet result = (with-retry --max-attempts 3 --delay 5 {\n # Operation that might fail\n http get "http://example.com/api"\n})\n\n# Wait for condition with timeout\nwait-for-condition --timeout 60 --interval 5 {\n # Condition to check\n check-service-health "orchestrator"\n} "orchestrator to be healthy"\n```\n\n### Example: Writing a New Service Integration Test\n\n```\n# Test Gitea Integration\n# Validates Gitea workspace git operations and extension publishing\n\nuse std log\nuse ../framework/test_helpers.nu *\n\ndef test-gitea-workspace-operations [test_config: record] {\n run-test "gitea-workspace-git-operations" {\n log info "Testing Gitea workspace operations..."\n\n # Create workspace\n let workspace = create-test-workspace "gitea-test" {\n provider: "local"\n }\n\n # Initialize git repo\n cd $workspace.path\n git init\n\n # Configure Gitea remote\n let gitea_url = $"http://($test_config.services.gitea.host):($test_config.services.gitea.port)"\n git remote add origin $"($gitea_url)/test-user/gitea-test.git"\n\n # Create test file\n "test content" | save test.txt\n git add test.txt\n git commit -m "Test commit"\n\n # Push to Gitea\n git push -u origin main\n\n # Verify push succeeded\n let remote_log = (git ls-remote origin)\n assert-not-empty $remote_log "Remote should have commits"\n\n log info "✓ Gitea workspace operations work"\n\n # Cleanup\n cleanup-test-workspace $workspace\n }\n}\n```\n\n---\n\n## Test Organization\n\n### Directory Structure\n\n```\nprovisioning/tests/integration/\n├── test_config.yaml # Test configuration\n├── setup_test_environment.nu # Environment setup script\n├── teardown_test_environment.nu # Cleanup script\n├── framework/ # Test framework utilities\n│ ├── test_helpers.nu # Common test helpers\n│ ├── orbstack_helpers.nu # OrbStack integration\n│ └── test_runner.nu # Test orchestration\n├── modes/ # Mode-specific tests\n│ ├── test_solo_mode.nu # Solo mode tests\n│ ├── test_multiuser_mode.nu # Multi-user mode tests\n│ ├── test_cicd_mode.nu # CI/CD mode tests\n│ └── test_enterprise_mode.nu # Enterprise mode tests\n├── services/ # Service integration tests\n│ ├── test_dns_integration.nu # CoreDNS tests\n│ ├── test_gitea_integration.nu # Gitea tests\n│ ├── test_oci_integration.nu # OCI registry tests\n│ └── test_service_orchestration.nu # Service manager tests\n├── workflows/ # Workflow tests\n│ ├── test_extension_loading.nu # Extension loading tests\n│ └── test_batch_workflows.nu # Batch workflow tests\n├── e2e/ # End-to-end tests\n│ ├── test_complete_deployment.nu # Full deployment workflow\n│ └── test_disaster_recovery.nu # Backup/restore tests\n├── performance/ # Performance tests\n│ ├── test_concurrency.nu # Concurrency tests\n│ └── test_scalability.nu # Scalability tests\n├── security/ # Security tests\n│ ├── test_rbac_enforcement.nu # RBAC tests\n│ └── test_kms_integration.nu # KMS tests\n└── docs/ # Documentation\n ├── TESTING_GUIDE.md # This guide\n ├── TEST_COVERAGE.md # Coverage report\n └── ORBSTACK_SETUP.md # OrbStack setup guide\n```\n\n### Test Naming Conventions\n\n- **Test Files**: `test__.nu`\n- **Test Functions**: `test-`\n- **Test Names**: `--`\n\n**Examples**:\n\n- File: `test_dns_integration.nu`\n- Function: `test-dns-registration`\n- Test Name: `solo-mode-dns-registration`\n\n---\n\n## CI/CD Integration\n\n### GitHub Actions\n\nCreate `.github/workflows/integration-tests.yml`:\n\n```\nname: Integration Tests\n\non:\n pull_request:\n push:\n branches: [main]\n schedule:\n - cron: '0 2 * * *' # Nightly at 2 AM\n\njobs:\n integration-tests:\n runs-on: macos-latest\n\n strategy:\n matrix:\n mode: [solo, multiuser, cicd, enterprise]\n\n steps:\n - name: Checkout code\n uses: actions/checkout@v3\n\n - name: Install OrbStack\n run: brew install --cask orbstack\n\n - name: Create OrbStack machine\n run: orb create provisioning\n\n - name: Install Nushell\n run: brew install nushell\n\n - name: Setup test environment\n run: |\n nu provisioning/tests/integration/setup_test_environment.nu \n --mode ${{ matrix.mode }}\n\n - name: Run integration tests\n run: |\n nu provisioning/tests/integration/framework/test_runner.nu \n --mode ${{ matrix.mode }} \n --report test-report.html\n\n - name: Upload test results\n if: always()\n uses: actions/upload-artifact@v3\n with:\n name: test-results-${{ matrix.mode }}\n path: |\n /tmp/provisioning-test-reports/\n test-report.html\n\n - name: Teardown test environment\n if: always()\n run: |\n nu provisioning/tests/integration/teardown_test_environment.nu --force\n```\n\n### GitLab CI\n\nCreate `.gitlab-ci.yml`:\n\n```\nstages:\n - test\n\nintegration-tests:\n stage: test\n image: ubuntu:22.04\n\n parallel:\n matrix:\n - MODE: [solo, multiuser, cicd, enterprise]\n\n before_script:\n # Install dependencies\n - apt-get update && apt-get install -y docker.io nushell\n\n script:\n # Setup test environment\n - nu provisioning/tests/integration/setup_test_environment.nu --mode $MODE\n\n # Run tests\n - nu provisioning/tests/integration/framework/test_runner.nu --mode $MODE --report test-report.html\n\n after_script:\n # Cleanup\n - nu provisioning/tests/integration/teardown_test_environment.nu --force\n\n artifacts:\n when: always\n paths:\n - /tmp/provisioning-test-reports/\n - test-report.html\n reports:\n junit: /tmp/provisioning-test-reports/junit-results.xml\n```\n\n---\n\n## Troubleshooting\n\n### Common Issues\n\n#### 1. OrbStack Machine Not Found\n\n**Error**: `OrbStack machine 'provisioning' not found`\n\n**Solution**:\n\n```\n# Create OrbStack machine\norb create provisioning\n\n# Verify creation\norb list\n```\n\n#### 2. Docker Connection Failed\n\n**Error**: `Cannot connect to Docker daemon`\n\n**Solution**:\n\n```\n# Verify OrbStack is running\norb status provisioning\n\n# Restart OrbStack\norb restart provisioning\n```\n\n#### 3. Service Health Check Timeout\n\n**Error**: `Timeout waiting for service orchestrator to be healthy`\n\n**Solution**:\n\n```\n# Check service logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# Verify service is running\ndocker -H /var/run/docker.sock ps | grep orchestrator\n\n# Increase timeout in test_config.yaml\n# test_execution.timeouts.test_timeout_seconds: 600\n```\n\n#### 4. Test Environment Cleanup Failed\n\n**Error**: `Failed to remove test workspace`\n\n**Solution**:\n\n```\n# Manual cleanup\nrm -rf /tmp/provisioning-test-workspace*\n\n# Cleanup OrbStack resources\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-cleanup\n```\n\n#### 5. DNS Resolution Failed\n\n**Error**: `DNS record should exist for server`\n\n**Solution**:\n\n```\n# Check CoreDNS logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs coredns\n\n# Verify CoreDNS is running\ndocker -H /var/run/docker.sock ps | grep coredns\n\n# Test DNS manually\ndig @172.20.0.2 test-server.local\n```\n\n### Debug Mode\n\nRun tests with verbose logging:\n\n```\n# Enable verbose output\nnu provisioning/tests/integration/framework/test_runner.nu --verbose --mode solo\n\n# Keep environment after tests for debugging\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown --mode solo\n\n# Inspect environment manually\ndocker -H /var/run/docker.sock ps\ndocker -H /var/run/docker.sock logs orchestrator\n```\n\n### Viewing Test Logs\n\n```\n# View test execution logs\ncat /tmp/provisioning-test.log\n\n# View service logs\nls /tmp/provisioning-test-reports/logs/\n\n# View HTML report\nopen /tmp/provisioning-test-reports/test-report.html\n```\n\n---\n\n## Performance Benchmarks\n\nExpected test execution times:\n\n| Test Suite | Duration (Solo) | Duration (Enterprise) |\n| ------------ | ----------------- | ------------------------ |\n| Mode Tests | 5-10 min | 15-20 min |\n| Service Tests | 3-5 min | 10-15 min |\n| Workflow Tests | 5-10 min | 15-20 min |\n| E2E Tests | 10-15 min | 30-40 min |\n| **Total** | **25-40 min** | **70-95 min** |\n\n**Parallel Execution** (4 workers):\n\n- Solo mode: ~10-15 min\n- Enterprise mode: ~25-35 min\n\n---\n\n## Best Practices\n\n1. **Idempotent Tests**: Tests should be repeatable without side effects\n2. **Isolated Tests**: Each test should be independent\n3. **Clear Assertions**: Use descriptive error messages\n4. **Cleanup**: Always cleanup resources, even on failure\n5. **Retry Flaky Operations**: Use `with-retry` for network operations\n6. **Meaningful Names**: Use descriptive test names\n7. **Fast Feedback**: Run quick tests first, slow tests later\n8. **Log Important Steps**: Log key operations for debugging\n\n---\n\n## References\n\n- [OrbStack Documentation](https://orbstack.dev/docs)\n- [Nushell Documentation](https://www.nushell.sh)\n- [Provisioning Platform Architecture](/docs/architecture/)\n- [Test Coverage Report](TEST_COVERAGE.md)\n- [OrbStack Setup Guide](ORBSTACK_SETUP.md)\n\n---\n\n**Maintained By**: Platform Team\n**Last Updated**: 2025-10-06 \ No newline at end of file +# Integration Testing Guide + +**Version**: 1.0.0 +**Last Updated**: 2025-10-06 + +This guide provides comprehensive documentation for the provisioning platform integration testing suite. + +## Table of Contents + +1. [Overview](#overview) +2. [Test Infrastructure](#test-infrastructure) +3. [Running Tests Locally](#running-tests-locally) +4. [Running Tests on OrbStack](#running-tests-on-orbstack) +5. [Writing New Tests](#writing-new-tests) +6. [Test Organization](#test-organization) +7. [CI/CD Integration](#cicd-integration) +8. [Troubleshooting](#troubleshooting) + +--- + +## Overview + +The integration testing suite validates all four execution modes of the provisioning platform: + +- **Solo Mode**: Single-user, minimal services (orchestrator, CoreDNS, OCI registry) +- **Multi-User Mode**: Multi-user support with Gitea, PostgreSQL, RBAC +- **CI/CD Mode**: Automation mode with API server, service accounts +- **Enterprise Mode**: Full enterprise features (Harbor, KMS, Prometheus, Grafana, ELK) + +### Key Features + +- ✅ **Comprehensive Coverage**: Tests for all 4 modes, 15+ services +- ✅ **OrbStack Integration**: Tests deployable to OrbStack machine "provisioning" +- ✅ **Parallel Execution**: Run independent tests in parallel for speed +- ✅ **Automatic Cleanup**: Resources cleaned up automatically after tests +- ✅ **Multiple Report Formats**: JUnit XML, HTML, JSON +- ✅ **CI/CD Ready**: GitHub Actions and GitLab CI integration + +--- + +## Test Infrastructure + +### Prerequisites + +1. **OrbStack Installed**: + + ```bash + # Install OrbStack (macOS) + brew install --cask orbstack + ``` + +2. **OrbStack Machine Named "provisioning"**: + + ```bash + # Create OrbStack machine + orb create provisioning + + # Verify machine is running + orb status provisioning + ``` + +3. **Nushell 0.107.1+**: + + ```bash + # Install Nushell + brew install nushell + ``` + +4. **Docker CLI**: + + ```bash + # Verify Docker is available + docker version + ``` + +### Test Configuration + +The test suite is configured via `provisioning/tests/integration/test_config.yaml`: + +```yaml +# OrbStack connection +orbstack: + machine_name: "provisioning" + connection: + type: "docker" + socket: "/var/run/docker.sock" + +# Service endpoints +services: + orchestrator: + host: "172.20.0.10" + port: 8080 + + coredns: + host: "172.20.0.2" + port: 53 + + # ... more services +``` + +**Key Settings**: + +- `orbstack.machine_name`: Name of OrbStack machine to use +- `services.*`: IP addresses and ports for deployed services +- `test_execution.parallel.max_workers`: Number of parallel test workers +- `test_execution.timeouts.*`: Timeout values for various operations + +--- + +## Running Tests Locally + +### Quick Start + +1. **Setup Test Environment**: + + ```bash + # Setup solo mode environment + nu provisioning/tests/integration/setup_test_environment.nu --mode solo + ``` + +1. **Run Tests**: + + ```bash + # Run all tests for solo mode + nu provisioning/tests/integration/framework/test_runner.nu --mode solo + + # Run specific test file + nu provisioning/tests/integration/modes/test_solo_mode.nu + ``` + +2. **Teardown Test Environment**: + + ```bash + # Cleanup all resources + nu provisioning/tests/integration/teardown_test_environment.nu --force + ``` + +### Test Runner Options + +```nushell +nu provisioning/tests/integration/framework/test_runner.nu + --mode # Test specific mode (solo, multiuser, cicd, enterprise) + --filter # Filter tests by regex pattern + --parallel # Number of parallel workers (default: 1) + --verbose # Detailed output + --report # Generate HTML report + --skip-setup # Skip environment setup + --skip-teardown # Skip environment teardown +``` + +**Examples**: + +```bash +# Run all tests for all modes +nu provisioning/tests/integration/framework/test_runner.nu + +# Run only solo mode tests +nu provisioning/tests/integration/framework/test_runner.nu --mode solo + +# Run tests matching pattern +nu provisioning/tests/integration/framework/test_runner.nu --filter "dns" + +# Run tests in parallel with 4 workers +nu provisioning/tests/integration/framework/test_runner.nu --parallel 4 + +# Generate HTML report +nu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html + +# Run tests without cleanup (for debugging) +nu provisioning/tests/integration/framework/test_runner.nu --skip-teardown +``` + +--- + +## Running Tests on OrbStack + +### Setup OrbStack Machine + +1. **Create OrbStack Machine**: + + ```bash + # Create machine named "provisioning" + orb create provisioning --cpu 4 --memory 8192 --disk 100 + + # Verify machine is created + orb list + ``` + +1. **Configure Machine**: + + ```bash + # Start machine + orb start provisioning + + # Verify Docker is accessible + docker -H /var/run/docker.sock ps + ``` + +### Deploy Platform to OrbStack + +The test setup automatically deploys platform services to OrbStack: + +```bash +# Deploy solo mode +nu provisioning/tests/integration/setup_test_environment.nu --mode solo + +# Deploy multi-user mode +nu provisioning/tests/integration/setup_test_environment.nu --mode multiuser + +# Deploy CI/CD mode +nu provisioning/tests/integration/setup_test_environment.nu --mode cicd + +# Deploy enterprise mode +nu provisioning/tests/integration/setup_test_environment.nu --mode enterprise +``` + +**Deployed Services**: + +| Mode | Services | +| ------ | ---------- | +| Solo | Orchestrator, CoreDNS, Zot (OCI registry) | +| Multi-User | Solo services + Gitea, PostgreSQL | +| CI/CD | Multi-User services + API server, Prometheus | +| Enterprise | CI/CD services + Harbor, KMS, Grafana, Elasticsearch | + +### Verify Deployment + +```bash +# Check service health +nu provisioning/tests/integration/framework/test_helpers.nu check-service-health orchestrator + +# View service logs +nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator + +# List running containers +docker -H /var/run/docker.sock ps +``` + +--- + +## Writing New Tests + +### Test File Structure + +All test files follow this structure: + +```bash +# Test Description +# Brief description of what this test validates + +use std log +use ../framework/test_helpers.nu * +use ../framework/orbstack_helpers.nu * + +# Main test suite +export def main [] { + log info "Running " + + let test_config = (load-test-config) + + mut results = [] + + # Run all tests + $results = ($results | append (test-case-1 $test_config)) + $results = ($results | append (test-case-2 $test_config)) + + # Report results + report-test-results $results +} + +# Individual test case +def test-case-1 [test_config: record] { + run-test "test-case-1-name" { + log info "Testing specific functionality..." + + # Test logic + let result = (some-operation) + + # Assertions + assert-eq $result.status "success" "Operation should succeed" + assert-not-empty $result.data "Result should contain data" + + log info "✓ Test case 1 passed" + } +} + +# Report test results +def report-test-results [results: list] { + # ... reporting logic +} +``` + +### Using Assertion Helpers + +The test framework provides several assertion helpers: + +```bash +# Equality assertion +assert-eq $actual $expected "Error message if assertion fails" + +# Boolean assertions +assert-true $condition "Error message" +assert-false $condition "Error message" + +# Collection assertions +assert-contains $list $item "Error message" +assert-not-contains $list $item "Error message" +assert-not-empty $value "Error message" + +# HTTP assertions +assert-http-success $response "Error message" +``` + +### Using Test Fixtures + +Create reusable test fixtures: + +```bash +# Create test workspace +let workspace = create-test-workspace "my-test-ws" { + provider: "local" + environment: "test" +} + +# Create test server +let server = create-test-server "test-server" "local" { + cores: 4 + memory: 8192 +} + +# Cleanup +cleanup-test-workspace $workspace +delete-test-server $server.id +``` + +### Using Retry Logic + +For flaky operations, use retry helpers: + +```bash +# Retry operation up to 3 times +let result = (with-retry --max-attempts 3 --delay 5 { + # Operation that might fail + http get "http://example.com/api" +}) + +# Wait for condition with timeout +wait-for-condition --timeout 60 --interval 5 { + # Condition to check + check-service-health "orchestrator" +} "orchestrator to be healthy" +``` + +### Example: Writing a New Service Integration Test + +```bash +# Test Gitea Integration +# Validates Gitea workspace git operations and extension publishing + +use std log +use ../framework/test_helpers.nu * + +def test-gitea-workspace-operations [test_config: record] { + run-test "gitea-workspace-git-operations" { + log info "Testing Gitea workspace operations..." + + # Create workspace + let workspace = create-test-workspace "gitea-test" { + provider: "local" + } + + # Initialize git repo + cd $workspace.path + git init + + # Configure Gitea remote + let gitea_url = $"http://($test_config.services.gitea.host):($test_config.services.gitea.port)" + git remote add origin $"($gitea_url)/test-user/gitea-test.git" + + # Create test file + "test content" | save test.txt + git add test.txt + git commit -m "Test commit" + + # Push to Gitea + git push -u origin main + + # Verify push succeeded + let remote_log = (git ls-remote origin) + assert-not-empty $remote_log "Remote should have commits" + + log info "✓ Gitea workspace operations work" + + # Cleanup + cleanup-test-workspace $workspace + } +} +``` + +--- + +## Test Organization + +### Directory Structure + +```bash +provisioning/tests/integration/ +├── test_config.yaml # Test configuration +├── setup_test_environment.nu # Environment setup script +├── teardown_test_environment.nu # Cleanup script +├── framework/ # Test framework utilities +│ ├── test_helpers.nu # Common test helpers +│ ├── orbstack_helpers.nu # OrbStack integration +│ └── test_runner.nu # Test orchestration +├── modes/ # Mode-specific tests +│ ├── test_solo_mode.nu # Solo mode tests +│ ├── test_multiuser_mode.nu # Multi-user mode tests +│ ├── test_cicd_mode.nu # CI/CD mode tests +│ └── test_enterprise_mode.nu # Enterprise mode tests +├── services/ # Service integration tests +│ ├── test_dns_integration.nu # CoreDNS tests +│ ├── test_gitea_integration.nu # Gitea tests +│ ├── test_oci_integration.nu # OCI registry tests +│ └── test_service_orchestration.nu # Service manager tests +├── workflows/ # Workflow tests +│ ├── test_extension_loading.nu # Extension loading tests +│ └── test_batch_workflows.nu # Batch workflow tests +├── e2e/ # End-to-end tests +│ ├── test_complete_deployment.nu # Full deployment workflow +│ └── test_disaster_recovery.nu # Backup/restore tests +├── performance/ # Performance tests +│ ├── test_concurrency.nu # Concurrency tests +│ └── test_scalability.nu # Scalability tests +├── security/ # Security tests +│ ├── test_rbac_enforcement.nu # RBAC tests +│ └── test_kms_integration.nu # KMS tests +└── docs/ # Documentation + ├── TESTING_GUIDE.md # This guide + ├── TEST_COVERAGE.md # Coverage report + └── ORBSTACK_SETUP.md # OrbStack setup guide +``` + +### Test Naming Conventions + +- **Test Files**: `test__.nu` +- **Test Functions**: `test-` +- **Test Names**: `--` + +**Examples**: + +- File: `test_dns_integration.nu` +- Function: `test-dns-registration` +- Test Name: `solo-mode-dns-registration` + +--- + +## CI/CD Integration + +### GitHub Actions + +Create `.github/workflows/integration-tests.yml`: + +```yaml +name: Integration Tests + +on: + pull_request: + push: + branches: [main] + schedule: + - cron: '0 2 * * *' # Nightly at 2 AM + +jobs: + integration-tests: + runs-on: macos-latest + + strategy: + matrix: + mode: [solo, multiuser, cicd, enterprise] + + steps: + - name: Checkout code + uses: actions/checkout@v3 + + - name: Install OrbStack + run: brew install --cask orbstack + + - name: Create OrbStack machine + run: orb create provisioning + + - name: Install Nushell + run: brew install nushell + + - name: Setup test environment + run: | + nu provisioning/tests/integration/setup_test_environment.nu + --mode ${{ matrix.mode }} + + - name: Run integration tests + run: | + nu provisioning/tests/integration/framework/test_runner.nu + --mode ${{ matrix.mode }} + --report test-report.html + + - name: Upload test results + if: always() + uses: actions/upload-artifact@v3 + with: + name: test-results-${{ matrix.mode }} + path: | + /tmp/provisioning-test-reports/ + test-report.html + + - name: Teardown test environment + if: always() + run: | + nu provisioning/tests/integration/teardown_test_environment.nu --force +``` + +### GitLab CI + +Create `.gitlab-ci.yml`: + +```yaml +stages: + - test + +integration-tests: + stage: test + image: ubuntu:22.04 + + parallel: + matrix: + - MODE: [solo, multiuser, cicd, enterprise] + + before_script: + # Install dependencies + - apt-get update && apt-get install -y docker.io nushell + + script: + # Setup test environment + - nu provisioning/tests/integration/setup_test_environment.nu --mode $MODE + + # Run tests + - nu provisioning/tests/integration/framework/test_runner.nu --mode $MODE --report test-report.html + + after_script: + # Cleanup + - nu provisioning/tests/integration/teardown_test_environment.nu --force + + artifacts: + when: always + paths: + - /tmp/provisioning-test-reports/ + - test-report.html + reports: + junit: /tmp/provisioning-test-reports/junit-results.xml +``` + +--- + +## Troubleshooting + +### Common Issues + +#### 1. OrbStack Machine Not Found + +**Error**: `OrbStack machine 'provisioning' not found` + +**Solution**: + +```bash +# Create OrbStack machine +orb create provisioning + +# Verify creation +orb list +``` + +#### 2. Docker Connection Failed + +**Error**: `Cannot connect to Docker daemon` + +**Solution**: + +```bash +# Verify OrbStack is running +orb status provisioning + +# Restart OrbStack +orb restart provisioning +``` + +#### 3. Service Health Check Timeout + +**Error**: `Timeout waiting for service orchestrator to be healthy` + +**Solution**: + +```bash +# Check service logs +nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator + +# Verify service is running +docker -H /var/run/docker.sock ps | grep orchestrator + +# Increase timeout in test_config.yaml +# test_execution.timeouts.test_timeout_seconds: 600 +``` + +#### 4. Test Environment Cleanup Failed + +**Error**: `Failed to remove test workspace` + +**Solution**: + +```bash +# Manual cleanup +rm -rf /tmp/provisioning-test-workspace* + +# Cleanup OrbStack resources +nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-cleanup +``` + +#### 5. DNS Resolution Failed + +**Error**: `DNS record should exist for server` + +**Solution**: + +```bash +# Check CoreDNS logs +nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs coredns + +# Verify CoreDNS is running +docker -H /var/run/docker.sock ps | grep coredns + +# Test DNS manually +dig @172.20.0.2 test-server.local +``` + +### Debug Mode + +Run tests with verbose logging: + +```bash +# Enable verbose output +nu provisioning/tests/integration/framework/test_runner.nu --verbose --mode solo + +# Keep environment after tests for debugging +nu provisioning/tests/integration/framework/test_runner.nu --skip-teardown --mode solo + +# Inspect environment manually +docker -H /var/run/docker.sock ps +docker -H /var/run/docker.sock logs orchestrator +``` + +### Viewing Test Logs + +```bash +# View test execution logs +cat /tmp/provisioning-test.log + +# View service logs +ls /tmp/provisioning-test-reports/logs/ + +# View HTML report +open /tmp/provisioning-test-reports/test-report.html +``` + +--- + +## Performance Benchmarks + +Expected test execution times: + +| Test Suite | Duration (Solo) | Duration (Enterprise) | +| ------------ | ----------------- | ------------------------ | +| Mode Tests | 5-10 min | 15-20 min | +| Service Tests | 3-5 min | 10-15 min | +| Workflow Tests | 5-10 min | 15-20 min | +| E2E Tests | 10-15 min | 30-40 min | +| **Total** | **25-40 min** | **70-95 min** | + +**Parallel Execution** (4 workers): + +- Solo mode: ~10-15 min +- Enterprise mode: ~25-35 min + +--- + +## Best Practices + +1. **Idempotent Tests**: Tests should be repeatable without side effects +2. **Isolated Tests**: Each test should be independent +3. **Clear Assertions**: Use descriptive error messages +4. **Cleanup**: Always cleanup resources, even on failure +5. **Retry Flaky Operations**: Use `with-retry` for network operations +6. **Meaningful Names**: Use descriptive test names +7. **Fast Feedback**: Run quick tests first, slow tests later +8. **Log Important Steps**: Log key operations for debugging + +--- + +## References + +- [OrbStack Documentation](https://orbstack.dev/docs) +- [Nushell Documentation](https://www.nushell.sh) +- [Provisioning Platform Architecture](/docs/architecture/) +- [Test Coverage Report](TEST_COVERAGE.md) +- [OrbStack Setup Guide](ORBSTACK_SETUP.md) + +--- + +**Maintained By**: Platform Team +**Last Updated**: 2025-10-06 \ No newline at end of file