# Extension Development API This document provides comprehensive guidance for developing extensions for provisioning, including providers, task services, and cluster configurations. ## Overview Provisioning supports three types of extensions: 1. **Providers**: Cloud infrastructure providers (AWS, UpCloud, Local, etc.) 2. **Task Services**: Infrastructure components (Kubernetes, Cilium, Containerd, etc.) 3. **Clusters**: Complete deployment configurations (BuildKit, CI/CD, etc.) All extensions follow a standardized structure and API for seamless integration. ## Extension Structure ### Standard Directory Layout ```bash extension-name/ ├── manifest.toml # Extension metadata ├── schemas/ # Nickel configuration files │ ├── main.ncl # Main schema │ ├── settings.ncl # Settings schema │ ├── version.ncl # Version configuration │ └── contracts.ncl # Contract definitions ├── nulib/ # Nushell library modules │ ├── mod.nu # Main module │ ├── create.nu # Creation operations │ ├── delete.nu # Deletion operations │ └── utils.nu # Utility functions ├── templates/ # Jinja2 templates │ ├── config.j2 # Configuration templates │ └── scripts/ # Script templates ├── generate/ # Code generation scripts │ └── generate.nu # Generation commands ├── README.md # Extension documentation └── metadata.toml # Extension metadata ``` ## Provider Extension API ### Provider Interface All providers must implement the following interface: #### Core Operations - `create-server(config: record) -> record` - `delete-server(server_id: string) -> null` - `list-servers() -> list` - `get-server-info(server_id: string) -> record` - `start-server(server_id: string) -> null` - `stop-server(server_id: string) -> null` - `reboot-server(server_id: string) -> null` #### Pricing and Plans - `get-pricing() -> list` - `get-plans() -> list` - `get-zones() -> list` #### SSH and Access - `get-ssh-access(server_id: string) -> record` - `configure-firewall(server_id: string, rules: list) -> null` ### Provider Development Template #### Nickel Configuration Schema Create `schemas/settings.ncl`: ```nickel # Provider settings schema { ProviderSettings = { # Authentication configuration auth | { method | "api_key" | "certificate" | "oauth" | "basic", api_key | String = null, api_secret | String = null, username | String = null, password | String = null, certificate_path | String = null, private_key_path | String = null, }, # API configuration api | { base_url | String, version | String = "v1", timeout | Number = 30, retries | Number = 3, }, # Default server configuration defaults: { plan?: str zone?: str os?: str ssh_keys?: [str] firewall_rules?: [FirewallRule] } # Provider-specific settings features: { load_balancer?: bool = false storage_encryption?: bool = true backup?: bool = true monitoring?: bool = false } } schema FirewallRule { direction: "ingress" | "egress" protocol: "tcp" | "udp" | "icmp" port?: str source?: str destination?: str action: "allow" | "deny" } schema ServerConfig { hostname: str plan: str zone: str os: str = "ubuntu-22.04" ssh_keys: [str] = [] tags?: {str: str} = {} firewall_rules?: [FirewallRule] = [] storage?: { size?: int type?: str encrypted?: bool = true } network?: { public_ip?: bool = true private_network?: str bandwidth?: int } } ``` #### Nushell Implementation Create `nulib/mod.nu`: ```nushell use std log # Provider name and version export const PROVIDER_NAME = "my-provider" export const PROVIDER_VERSION = "1.0.0" # Import sub-modules use create.nu * use delete.nu * use utils.nu * # Provider interface implementation export def "provider-info" [] -> record { { name: $PROVIDER_NAME, version: $PROVIDER_VERSION, type: "provider", interface: "API", supported_operations: [ "create-server", "delete-server", "list-servers", "get-server-info", "start-server", "stop-server" ], required_auth: ["api_key", "api_secret"], supported_os: ["ubuntu-22.04", "debian-11", "centos-8"], regions: (get-zones).name } } export def "validate-config" [config: record] -> record { mut errors = [] mut warnings = [] # Validate authentication if ($config | get -o "auth.api_key" | is-empty) { $errors = ($errors | append "Missing API key") } if ($config | get -o "auth.api_secret" | is-empty) { $errors = ($errors | append "Missing API secret") } # Validate API configuration let api_url = ($config | get -o "api.base_url") if ($api_url | is-empty) { $errors = ($errors | append "Missing API base URL") } else { try { http get $"($api_url)/health" | ignore } catch { $warnings = ($warnings | append "API endpoint not reachable") } } { valid: ($errors | is-empty), errors: $errors, warnings: $warnings } } export def "test-connection" [config: record] -> record { try { let api_url = ($config | get "api.base_url") let response = (http get $"($api_url)/account" --headers { Authorization: $"Bearer ($config | get 'auth.api_key')" }) { success: true, account_info: $response, message: "Connection successful" } } catch {|e| { success: false, error: ($e | get msg), message: "Connection failed" } } } ``` Create `nulib/create.nu`: ```nushell use std log use utils.nu * export def "create-server" [ config: record # Server configuration --check # Check mode only --wait # Wait for completion ] -> record { log info $"Creating server: ($config.hostname)" if $check { return { action: "create-server", hostname: $config.hostname, check_mode: true, would_create: true, estimated_time: "2-5 minutes" } } # Validate configuration let validation = (validate-server-config $config) if not $validation.valid { error make { msg: $"Invalid server configuration: ($validation.errors | str join ', ')" } } # Prepare API request let api_config = (get-api-config) let request_body = { hostname: $config.hostname, plan: $config.plan, zone: $config.zone, os: $config.os, ssh_keys: $config.ssh_keys, tags: $config.tags, firewall_rules: $config.firewall_rules } try { let response = (http post $"($api_config.base_url)/servers" --headers { Authorization: $"Bearer ($api_config.auth.api_key)" Content-Type: "application/json" } $request_body) let server_id = ($response | get id) log info $"Server creation initiated: ($server_id)" if $wait { let final_status = (wait-for-server-ready $server_id) { success: true, server_id: $server_id, hostname: $config.hostname, status: $final_status, ip_addresses: (get-server-ips $server_id), ssh_access: (get-ssh-access $server_id) } } else { { success: true, server_id: $server_id, hostname: $config.hostname, status: "creating", message: "Server creation in progress" } } } catch {|e| error make { msg: $"Server creation failed: ($e | get msg)" } } } def validate-server-config [config: record] -> record { mut errors = [] # Required fields if ($config | get -o hostname | is-empty) { $errors = ($errors | append "Hostname is required") } if ($config | get -o plan | is-empty) { $errors = ($errors | append "Plan is required") } if ($config | get -o zone | is-empty) { $errors = ($errors | append "Zone is required") } # Validate plan exists let available_plans = (get-plans) if not ($config.plan in ($available_plans | get name)) { $errors = ($errors | append $"Invalid plan: ($config.plan)") } # Validate zone exists let available_zones = (get-zones) if not ($config.zone in ($available_zones | get name)) { $errors = ($errors | append $"Invalid zone: ($config.zone)") } { valid: ($errors | is-empty), errors: $errors } } def wait-for-server-ready [server_id: string] -> string { mut attempts = 0 let max_attempts = 60 # 10 minutes while $attempts < $max_attempts { let server_info = (get-server-info $server_id) let status = ($server_info | get status) match $status { "running" => { return "running" }, "error" => { error make { msg: "Server creation failed" } }, _ => { log info $"Server status: ($status), waiting..." sleep 10sec $attempts = $attempts + 1 } } } error make { msg: "Server creation timeout" } } ``` ### Provider Registration Add provider metadata in `metadata.toml`: ```toml [extension] name = "my-provider" type = "provider" version = "1.0.0" description = "Custom cloud provider integration" author = "Your Name " license = "MIT" [compatibility] provisioning_version = ">=2.0.0" nushell_version = ">=0.107.0" nickel_version = ">=1.15.0" [capabilities] server_management = true load_balancer = false storage_encryption = true backup = true monitoring = false [authentication] methods = ["api_key", "certificate"] required_fields = ["api_key", "api_secret"] [regions] default = "us-east-1" available = ["us-east-1", "us-west-2", "eu-west-1"] [support] documentation = "https://docs.example.com/provider" issues = "https://github.com/example/provider/issues" ``` ## Task Service Extension API ### Task Service Interface Task services must implement: #### Core Operations - `install(config: record) -> record` - `uninstall(config: record) -> null` - `configure(config: record) -> null` - `status() -> record` - `restart() -> null` - `upgrade(version: string) -> record` #### Version Management - `get-current-version() -> string` - `get-available-versions() -> list` - `check-updates() -> record` ### Task Service Development Template #### Nickel Schema Create `schemas/version.ncl`: ```nickel # Task service version configuration { taskserv_version = { name | String = "my-service", version | String = "1.0.0", # Version source configuration source | { type | String = "github", repository | String, release_pattern | String = "v{version}", }, # Installation configuration install | { method | String = "binary", binary_name | String, binary_path | String = "/usr/local/bin", config_path | String = "/etc/my-service", data_path | String = "/var/lib/my-service", }, # Dependencies dependencies | [ { name | String, version | String = ">=1.0.0", } ], # Service configuration service | { type | String = "systemd", user | String = "my-service", group | String = "my-service", ports | [Number] = [8080, 9090], }, # Health check configuration health_check | { endpoint | String, interval | Number = 30, timeout | Number = 5, retries | Number = 3, }, } } ``` #### Nushell Implementation Create `nulib/mod.nu`: ```nushell use std log use ../../../lib_provisioning * export const SERVICE_NAME = "my-service" export const SERVICE_VERSION = "1.0.0" export def "taskserv-info" [] -> record { { name: $SERVICE_NAME, version: $SERVICE_VERSION, type: "taskserv", category: "application", description: "Custom application service", dependencies: ["containerd"], ports: [8080, 9090], config_files: ["/etc/my-service/config.yaml"], data_directories: ["/var/lib/my-service"] } } export def "install" [ config: record = {} --check # Check mode only --version: string # Specific version to install ] -> record { let install_version = if ($version | is-not-empty) { $version } else { (get-latest-version) } log info $"Installing ($SERVICE_NAME) version ($install_version)" if $check { return { action: "install", service: $SERVICE_NAME, version: $install_version, check_mode: true, would_install: true, requirements_met: (check-requirements) } } # Check system requirements let req_check = (check-requirements) if not $req_check.met { error make { msg: $"Requirements not met: ($req_check.missing | str join ', ')" } } # Download and install let binary_path = (download-binary $install_version) install-binary $binary_path create-user-and-directories generate-config $config install-systemd-service # Start service systemctl start $SERVICE_NAME systemctl enable $SERVICE_NAME # Verify installation let health = (check-health) if not $health.healthy { error make { msg: "Service failed health check after installation" } } { success: true, service: $SERVICE_NAME, version: $install_version, status: "running", health: $health } } export def "uninstall" [ --force # Force removal even if running --keep-data # Keep data directories ] -> null { log info $"Uninstalling ($SERVICE_NAME)" # Stop and disable service try { systemctl stop $SERVICE_NAME systemctl disable $SERVICE_NAME } catch { log warning "Failed to stop systemd service" } # Remove binary try { rm -f $"/usr/local/bin/($SERVICE_NAME)" } catch { log warning "Failed to remove binary" } # Remove configuration try { rm -rf $"/etc/($SERVICE_NAME)" } catch { log warning "Failed to remove configuration" } # Remove data directories (unless keeping) if not $keep_data { try { rm -rf $"/var/lib/($SERVICE_NAME)" } catch { log warning "Failed to remove data directories" } } # Remove systemd service file try { rm -f $"/etc/systemd/system/($SERVICE_NAME).service" systemctl daemon-reload } catch { log warning "Failed to remove systemd service" } log info $"($SERVICE_NAME) uninstalled successfully" } export def "status" [] -> record { let systemd_status = try { systemctl is-active $SERVICE_NAME | str trim } catch { "unknown" } let health = (check-health) let version = (get-current-version) { service: $SERVICE_NAME, version: $version, systemd_status: $systemd_status, health: $health, uptime: (get-service-uptime), memory_usage: (get-memory-usage), cpu_usage: (get-cpu-usage) } } def check-requirements [] -> record { mut missing = [] mut met = true # Check for containerd if not (which containerd | is-not-empty) { $missing = ($missing | append "containerd") $met = false } # Check for systemctl if not (which systemctl | is-not-empty) { $missing = ($missing | append "systemctl") $met = false } { met: $met, missing: $missing } } def check-health [] -> record { try { let response = (http get "http://localhost:9090/health") { healthy: true, status: ($response | get status), last_check: (date now) } } catch { { healthy: false, error: "Health endpoint not responding", last_check: (date now) } } } ``` ## Cluster Extension API ### Cluster Interface Clusters orchestrate multiple components: #### Core Operations - `create(config: record) -> record` - `delete(config: record) -> null` - `status() -> record` - `scale(replicas: int) -> record` - `upgrade(version: string) -> record` #### Component Management - `list-components() -> list` - `component-status(name: string) -> record` - `restart-component(name: string) -> null` ### Cluster Development Template #### Nickel Configuration Create `schemas/cluster.ncl`: ```nickel # Cluster configuration schema { ClusterConfig = { # Cluster metadata name | String, version | String = "1.0.0", description | String = "", # Components to deploy components | [Component], # Resource requirements resources | { min_nodes | Number = 1, cpu_per_node | String = "2", memory_per_node | String = "4Gi", storage_per_node | String = "20Gi", }, # Network configuration network | { cluster_cidr | String = "10.244.0.0/16", service_cidr | String = "10.96.0.0/12", dns_domain | String = "cluster.local", }, # Feature flags features | { monitoring | Bool = true, logging | Bool = true, ingress | Bool = false, storage | Bool = true, }, }, Component = { name | String, type | String | "taskserv" | "application" | "infrastructure", version | String = "", enabled | Bool = true, dependencies | [String] = [], config | {} = {}, resources | { cpu | String = "", memory | String = "", storage | String = "", replicas | Number = 1, } = {}, }, # Example cluster configuration buildkit_cluster = { name = "buildkit", version = "1.0.0", description = "Container build cluster with BuildKit and registry", components = [ { name = "containerd", type = "taskserv", version = "1.7.0", enabled = true, dependencies = [], }, { name = "buildkit", type = "taskserv", version = "0.12.0", enabled = true, dependencies = ["containerd"], config = { worker_count = 4, cache_size = "10Gi", registry_mirrors = ["registry:5000"], }, }, { name = "registry", type = "application", version = "2.8.0", enabled = true, dependencies = [], config = { storage_driver = "filesystem", storage_path = "/var/lib/registry", auth_enabled = false, }, resources = { cpu = "500m", memory = "1Gi", storage = "50Gi", replicas = 1, }, }, ], resources = { min_nodes = 1, cpu_per_node = "4", memory_per_node = "8Gi", storage_per_node = "100Gi", }, features = { monitoring = true, logging = true, ingress = false, storage = true, }, }, } ``` #### Nushell Implementation Create `nulib/mod.nu`: ```nushell use std log use ../../../lib_provisioning * export const CLUSTER_NAME = "my-cluster" export const CLUSTER_VERSION = "1.0.0" export def "cluster-info" [] -> record { { name: $CLUSTER_NAME, version: $CLUSTER_VERSION, type: "cluster", category: "build", description: "Custom application cluster", components: (get-cluster-components), required_resources: { min_nodes: 1, cpu_per_node: "2", memory_per_node: "4Gi", storage_per_node: "20Gi" } } } export def "create" [ config: record = {} --check # Check mode only --wait # Wait for completion ] -> record { log info $"Creating cluster: ($CLUSTER_NAME)" if $check { return { action: "create-cluster", cluster: $CLUSTER_NAME, check_mode: true, would_create: true, components: (get-cluster-components), requirements_check: (check-cluster-requirements) } } # Validate cluster requirements let req_check = (check-cluster-requirements) if not $req_check.met { error make { msg: $"Cluster requirements not met: ($req_check.issues | str join ', ')" } } # Get component deployment order let components = (get-cluster-components) let deployment_order = (resolve-component-dependencies $components) mut deployment_status = [] # Deploy components in dependency order for component in $deployment_order { log info $"Deploying component: ($component.name)" try { let result = match $component.type { "taskserv" => { taskserv create $component.name --config $component.config --wait }, "application" => { deploy-application $component }, _ => { error make { msg: $"Unknown component type: ($component.type)" } } } $deployment_status = ($deployment_status | append { component: $component.name, status: "deployed", result: $result }) } catch {|e| log error $"Failed to deploy ($component.name): ($e.msg)" $deployment_status = ($deployment_status | append { component: $component.name, status: "failed", error: $e.msg }) # Rollback on failure rollback-cluster-deployment $deployment_status error make { msg: $"Cluster deployment failed at component: ($component.name)" } } } # Configure cluster networking and integrations configure-cluster-networking $config setup-cluster-monitoring $config # Wait for all components to be ready if $wait { wait-for-cluster-ready } { success: true, cluster: $CLUSTER_NAME, components: $deployment_status, endpoints: (get-cluster-endpoints), status: "running" } } export def "delete" [ config: record = {} --force # Force deletion ] -> null { log info $"Deleting cluster: ($CLUSTER_NAME)" let components = (get-cluster-components) let deletion_order = ($components | reverse) # Delete in reverse order for component in $deletion_order { log info $"Removing component: ($component.name)" try { match $component.type { "taskserv" => { taskserv delete $component.name --force=$force }, "application" => { remove-application $component --force=$force }, _ => { log warning $"Unknown component type: ($component.type)" } } } catch {|e| log error $"Failed to remove ($component.name): ($e.msg)" if not $force { error make { msg: $"Component removal failed: ($component.name)" } } } } # Clean up cluster-level resources cleanup-cluster-networking cleanup-cluster-monitoring cleanup-cluster-storage log info $"Cluster ($CLUSTER_NAME) deleted successfully" } def get-cluster-components [] -> list { [ { name: "containerd", type: "taskserv", version: "1.7.0", dependencies: [] }, { name: "my-service", type: "taskserv", version: "1.0.0", dependencies: ["containerd"] }, { name: "registry", type: "application", version: "2.8.0", dependencies: [] } ] } def resolve-component-dependencies [components: list] -> list { # Topological sort of components based on dependencies mut sorted = [] mut remaining = $components while ($remaining | length) > 0 { let no_deps = ($remaining | where {|comp| ($comp.dependencies | all {|dep| $dep in ($sorted | get name) }) }) if ($no_deps | length) == 0 { error make { msg: "Circular dependency detected in cluster components" } } $sorted = ($sorted | append $no_deps) $remaining = ($remaining | where {|comp| not ($comp.name in ($no_deps | get name)) }) } $sorted } ``` ## Extension Registration and Discovery ### Extension Registry Extensions are registered in the system through: 1. **Directory Structure**: Placed in appropriate directories (providers/, taskservs/, cluster/) 2. **Metadata Files**: `metadata.toml` with extension information 3. **Schema Files**: `schemas/` directory with Nickel schema files ### Registration API #### `register-extension(path: string, type: string) -> record` Registers a new extension with the system. **Parameters:** - `path`: Path to extension directory - `type`: Extension type (provider, taskserv, cluster) #### `unregister-extension(name: string, type: string) -> null` Removes extension from the registry. #### `list-registered-extensions(type?: string) -> list` Lists all registered extensions, optionally filtered by type. ### Extension Validation #### Validation Rules 1. **Structure Validation**: Required files and directories exist 2. **Schema Validation**: Nickel schemas are valid 3. **Interface Validation**: Required functions are implemented 4. **Dependency Validation**: Dependencies are available 5. **Version Validation**: Version constraints are met #### `validate-extension(path: string, type: string) -> record` Validates extension structure and implementation. ## Testing Extensions ### Test Framework Extensions should include comprehensive tests: #### Unit Tests Create `tests/unit_tests.nu`: ```nushell use std testing export def test_provider_config_validation [] { let config = { auth: { api_key: "test-key", api_secret: "test-secret" }, api: { base_url: "https://api.test.com" } } let result = (validate-config $config) assert ($result.valid == true) assert ($result.errors | is-empty) } export def test_server_creation_check_mode [] { let config = { hostname: "test-server", plan: "1xCPU-1 GB", zone: "test-zone" } let result = (create-server $config --check) assert ($result.check_mode == true) assert ($result.would_create == true) } ``` #### Integration Tests Create `tests/integration_tests.nu`: ```nushell use std testing export def test_full_server_lifecycle [] { # Test server creation let create_config = { hostname: "integration-test", plan: "1xCPU-1 GB", zone: "test-zone" } let server = (create-server $create_config --wait) assert ($server.success == true) let server_id = $server.server_id # Test server info retrieval let info = (get-server-info $server_id) assert ($info.hostname == "integration-test") assert ($info.status == "running") # Test server deletion delete-server $server_id # Verify deletion let final_info = try { get-server-info $server_id } catch { null } assert ($final_info == null) } ``` ### Running Tests ```bash # Run unit tests nu tests/unit_tests.nu # Run integration tests nu tests/integration_tests.nu # Run all tests nu tests/run_all_tests.nu ``` ## Documentation Requirements ### Extension Documentation Each extension must include: 1. **README.md**: Overview, installation, and usage 2. **API.md**: Detailed API documentation 3. **EXAMPLES.md**: Usage examples and tutorials 4. **CHANGELOG.md**: Version history and changes ### API Documentation Template ```bash # Extension Name API ## Overview Brief description of the extension and its purpose. ## Installation Steps to install and configure the extension. ## Configuration Configuration schema and options. ## API Reference Detailed API documentation with examples. ## Examples Common usage patterns and examples. ## Troubleshooting Common issues and solutions. ``` ## Best Practices ### Development Guidelines 1. **Follow Naming Conventions**: Use consistent naming for functions and variables 2. **Error Handling**: Implement comprehensive error handling and recovery 3. **Logging**: Use structured logging for debugging and monitoring 4. **Configuration Validation**: Validate all inputs and configurations 5. **Documentation**: Document all public APIs and configurations 6. **Testing**: Include comprehensive unit and integration tests 7. **Versioning**: Follow semantic versioning principles 8. **Security**: Implement secure credential handling and API calls ### Performance Considerations 1. **Caching**: Cache expensive operations and API calls 2. **Parallel Processing**: Use parallel execution where possible 3. **Resource Management**: Clean up resources properly 4. **Batch Operations**: Batch API calls when possible 5. **Health Monitoring**: Implement health checks and monitoring ### Security Best Practices 1. **Credential Management**: Store credentials securely 2. **Input Validation**: Validate and sanitize all inputs 3. **Access Control**: Implement proper access controls 4. **Audit Logging**: Log all security-relevant operations 5. **Encryption**: Encrypt sensitive data in transit and at rest This extension development API provides a comprehensive framework for building robust, scalable, and maintainable extensions for provisioning.