provisioning/docs/src/architecture/design-principles.md

Design Principles

Overview

Provisioning is built on a foundation of architectural principles that guide design decisions, ensure system quality, and maintain consistency across the codebase. These principles have evolved from real-world experience and represent lessons learned from complex infrastructure automation challenges.

Core Architectural Principles

1. Project Architecture Principles (PAP) Compliance

Principle: Fully agnostic and configuration-driven, not hardcoded. Use abstraction layers dynamically loaded from configurations.

Rationale: Infrastructure as Code (IaC) systems must be flexible enough to adapt to any environment without code changes. Hardcoded values defeat the purpose of IaC and create maintenance burdens.

Implementation Guidelines:

• Never patch the system with hardcoded fallbacks when configuration parsing fails
• All behavior must be configurable through the hierarchical configuration system
• Use abstraction layers that are dynamically loaded from configuration
• Validate configuration fully before execution, fail fast on invalid config

Anti-Patterns (Anti-PAP):

• Hardcoded provider endpoints or credentials
• Environment-specific logic in code
• Fallback to default values when configuration is missing
• Mixed configuration and implementation logic

Example:

# ✅ PAP Compliant - Configuration-driven
[providers.aws]
regions = ["us-west-2", "us-east-1"]
instance_types = ["t3.micro", "t3.small"]
api_endpoint = "https://ec2.amazonaws.com"

# ❌ Anti-PAP - Hardcoded fallback in code
if config.providers.aws.regions.is_empty() {
    regions = vec!["us-west-2"]; // Hardcoded fallback
}

2. Hybrid Architecture Optimization

Principle: Use each language for what it does best - Rust for coordination, Nushell for business logic.

Rationale: Different languages have different strengths. Rust excels at performance-critical coordination tasks, while Nushell excels at configuration management and domain-specific operations.

Implementation Guidelines:

• Rust handles orchestration, state management, and performance-critical paths
• Nushell handles provider operations, configuration processing, and CLI interfaces
• Clear boundaries between language responsibilities
• Structured data exchange (JSON) between languages
• Preserve existing domain expertise in Nushell

Language Responsibility Matrix:

Rust Layer:
├── Workflow orchestration and coordination
├── REST API servers and HTTP endpoints
├── State persistence and checkpoint management
├── Parallel processing and batch operations
├── Error recovery and rollback logic
└── Performance-critical data processing

Nushell Layer:
├── Provider implementations (AWS, UpCloud, local)
├── Task service management and configuration
├── Nickel configuration processing and validation
├── Template generation and Infrastructure as Code
├── CLI user interfaces and interactive tools
└── Domain-specific business logic

3. Configuration-First Architecture

Principle: All system behavior is determined by configuration, with clear hierarchical precedence and validation.

Rationale: True Infrastructure as Code requires that all behavior be configurable without code changes. Configuration hierarchy provides flexibility while maintaining predictability.

Configuration Hierarchy (precedence order):

1. Runtime Parameters (highest precedence)
2. Environment Configuration
3. Infrastructure Configuration
4. User Configuration
5. System Defaults (lowest precedence)

Implementation Guidelines:

• Complete configuration validation before execution
• Variable interpolation for dynamic values
• Schema-based validation using Nickel
• Configuration immutability during execution
• Comprehensive error reporting for configuration issues

4. Domain-Driven Structure

Principle: Organize code by business domains and functional boundaries, not by technical concerns.

Rationale: Domain-driven organization scales better, reduces coupling, and enables focused development by domain experts.

Domain Organization:

├── core/           # Core system and library functions
├── platform/       # High-performance coordination layer
├── provisioning/   # Main business logic with providers and services
├── control-center/ # Web-based management interface
├── tools/          # Development and utility tools
└── extensions/     # Plugin and extension framework

Domain Responsibilities:

• Each domain has clear ownership and boundaries
• Cross-domain communication through well-defined interfaces
• Domain-specific testing and validation strategies
• Independent evolution and versioning within architectural guidelines

5. Isolation and Modularity

Principle: Components are isolated, modular, and independently deployable with clear interface contracts.

Rationale: Isolation enables independent development, testing, and deployment. Clear interfaces prevent tight coupling and enable system evolution.

Implementation Guidelines:

• User workspace isolation from system installation
• Extension sandboxing and security boundaries
• Provider abstraction with standardized interfaces
• Service modularity with dependency management
• Clear API contracts between components

Quality Attribute Principles

6. Reliability Through Recovery

Principle: Build comprehensive error recovery and rollback capabilities into every operation.

Rationale: Infrastructure operations can fail at any point. Systems must be able to recover gracefully and maintain consistent state.

Implementation Guidelines:

• Checkpoint-based recovery for long-running workflows
• Comprehensive rollback capabilities for all operations
• Transactional semantics where possible
• State validation and consistency checks
• Detailed audit trails for debugging and recovery

Recovery Strategies:

Operation Level:
├── Atomic operations with rollback
├── Retry logic with exponential backoff
├── Circuit breakers for external dependencies
└── Graceful degradation on partial failures

Workflow Level:
├── Checkpoint-based recovery
├── Dependency-aware rollback
├── State consistency validation
└── Resume from failure points

System Level:
├── Health monitoring and alerting
├── Automatic recovery procedures
├── Data backup and restoration
└── Disaster recovery capabilities

7. Performance Through Parallelism

Principle: Design for parallel execution and efficient resource utilization while maintaining correctness.

Rationale: Infrastructure operations often involve multiple independent resources that can be processed in parallel for significant performance gains.

Implementation Guidelines:

• Configurable parallelism limits to prevent resource exhaustion
• Dependency-aware parallel execution
• Resource pooling and connection management
• Efficient data structures and algorithms
• Memory-conscious processing for large datasets

8. Security Through Isolation

Principle: Implement security through isolation boundaries, least privilege, and comprehensive validation.

Rationale: Infrastructure systems handle sensitive data and powerful operations. Security must be built in at the architectural level.

Security Implementation:

Authentication & Authorization:
├── API authentication for external access
├── Role-based access control for operations
├── Permission validation before execution
└── Audit logging for all security events

Data Protection:
├── Encrypted secrets management (SOPS/Age)
├── Secure configuration file handling
├── Network communication encryption
└── Sensitive data sanitization in logs

Isolation Boundaries:
├── User workspace isolation
├── Extension sandboxing
├── Provider credential isolation
└── Process and network isolation

Development Methodology Principles

9. Configuration-Driven Testing

Principle: Tests should be configuration-driven and validate both happy path and error conditions.

Rationale: Infrastructure systems must work across diverse environments and configurations. Tests must validate the configuration-driven nature of the system.

Testing Strategy:

Unit Testing:
├── Configuration validation tests
├── Individual component tests
├── Error condition tests
└── Performance benchmark tests

Integration Testing:
├── Multi-provider workflow tests
├── Configuration hierarchy tests
├── Error recovery tests
└── End-to-end scenario tests

System Testing:
├── Full deployment tests
├── Upgrade and migration tests
├── Performance and scalability tests
└── Security and isolation tests

Error Handling Principles

11. Fail Fast, Recover Gracefully

Principle: Validate early and fail fast on errors, but provide comprehensive recovery mechanisms.

Rationale: Early validation prevents complex error states, while graceful recovery maintains system reliability.

Implementation Guidelines:

• Complete configuration validation before execution
• Input validation at system boundaries
• Clear error messages without internal stack traces (except in DEBUG mode)
• Comprehensive error categorization and handling
• Recovery procedures for all error categories

Error Categories:

Configuration Errors:
├── Invalid configuration syntax
├── Missing required configuration
├── Configuration conflicts
└── Schema validation failures

Runtime Errors:
├── Provider API failures
├── Network connectivity issues
├── Resource availability problems
└── Permission and authentication errors

System Errors:
├── File system access problems
├── Memory and resource exhaustion
├── Process communication failures
└── External dependency failures

12. Observable Operations

Principle: All operations must be observable through comprehensive logging, metrics, and monitoring.

Rationale: Infrastructure operations must be debuggable and monitorable in production environments.

Observability Implementation:

Logging:
├── Structured JSON logging
├── Configurable log levels
├── Context-aware log messages
└── Audit trail for all operations

Metrics:
├── Operation performance metrics
├── Resource utilization metrics
├── Error rate and type metrics
└── Business logic metrics

Monitoring:
├── Health check endpoints
├── Real-time status reporting
├── Workflow progress tracking
└── Alert integration capabilities

Evolution and Maintenance Principles

13. Backward Compatibility

Principle: Maintain backward compatibility for configuration, APIs, and user interfaces.

Rationale: Infrastructure systems are long-lived and must support existing configurations and workflows during evolution.

Compatibility Guidelines:

• Semantic versioning for all interfaces
• Configuration migration tools and procedures
• Deprecation warnings and migration guides
• API versioning for external interfaces
• Comprehensive upgrade testing

14. Documentation-Driven Development

Principle: Architecture decisions, APIs, and operational procedures must be thoroughly documented.

Rationale: Infrastructure systems are complex and require clear documentation for operation, maintenance, and evolution.

Documentation Requirements:

• Architecture Decision Records (ADRs) for major decisions
• API documentation with examples
• Operational runbooks and procedures
• Configuration guides and examples
• Troubleshooting guides and common issues

15. Technical Debt Management

Principle: Actively manage technical debt through regular assessment and systematic improvement.

Rationale: Infrastructure systems accumulate complexity over time. Proactive debt management prevents system degradation.

Debt Management Strategy:

Assessment:
├── Regular code quality reviews
├── Performance profiling and optimization
├── Security audit and updates
└── Dependency management and updates

Improvement:
├── Refactoring for clarity and maintainability
├── Performance optimization based on metrics
├── Security enhancement and hardening
└── Test coverage improvement and validation

Trade-off Management

16. Explicit Trade-off Documentation

Principle: All architectural trade-offs must be explicitly documented with rationale and alternatives considered.

Rationale: Understanding trade-offs enables informed decision making and future evolution of the system.

Trade-off Categories:

Performance vs. Maintainability:
├── Rust coordination layer for performance
├── Nushell business logic for maintainability
├── Caching strategies for speed vs. consistency
└── Parallel processing vs. resource usage

Flexibility vs. Complexity:
├── Configuration-driven architecture vs. simplicity
├── Extension framework vs. core system complexity
├── Multi-provider support vs. specialization
└── Hierarchical configuration vs. simple key-value

Security vs. Usability:
├── Workspace isolation vs. convenience
├── Extension sandboxing vs. functionality
├── Authentication requirements vs. ease of use
└── Audit logging vs. performance overhead

Conclusion

These design principles form the foundation of provisioning's architecture. They guide decision making, ensure quality, and provide a framework for system evolution. Adherence to these principles has enabled the development of a sophisticated, reliable, and maintainable infrastructure automation platform.

The principles are living guidelines that evolve with the system while maintaining core architectural integrity. They serve as both implementation guidance and evaluation criteria for new features and modifications.

Success in applying these principles is measured by:

• System reliability and error recovery capabilities
• Development efficiency and maintainability
• Configuration flexibility and user experience
• Performance and scalability characteristics
• Security and isolation effectiveness

These principles represent the distilled wisdom from building and operating complex infrastructure automation systems at scale.