prvng_extensions/taskservs/infrastructure/polkadot/solochain/README.md

# Polkadot Solochain Task Service

## Overview

The Polkadot Solochain task service provides a complete installation and configuration of a standalone [Polkadot](https://polkadot.network/) blockchain using the Polkadot SDK. A solochain operates independently of the Polkadot relay chain while leveraging the same runtime technology, making it perfect for private networks, testnets, and independent blockchain projects.

## Features

### Core Capabilities
- **Independent Blockchain** - Complete standalone blockchain using Polkadot SDK
- **PVM Support** - Full Polkadot Virtual Machine integration with WASM execution
- **Custom Runtime** - Modular pallet system with configurable features
- **Consensus Mechanisms** - Aura (block authoring) + GRANDPA (finality)
- **Development & Production** - Suitable for both environments

### Runtime Features
- **Substrate Framework** - Built on Substrate with modern pallets
- **WASM Runtime** - WebAssembly runtime with native fallback
- **Modular Pallets** - Configurable system including balances, timestamp, sudo
- **Transaction Pool** - Configurable transaction management
- **Block Production** - Customizable block time and limits

### Network Configuration
- **P2P Networking** - Configurable peer-to-peer networking
- **Bootnodes** - Support for bootstrap nodes
- **Reserved Nodes** - Trusted node connections
- **Chain Specification** - Automated chain spec generation

### Consensus & Validation
- **Aura Consensus** - Authority-based block authoring
- **ELVES Consensus** - Ethereum-Like Validation Execution System for enhanced compatibility
- **GRANDPA Finality** - Byzantine fault-tolerant finality gadget
- **Hybrid Consensus** - Support for multiple consensus mechanisms simultaneously
- **Session Key Management** - Automated key generation and rotation
- **Validator Support** - Production validator configuration

### Development Features
- **Alice/Bob Keys** - Built-in development accounts
- **Telemetry** - Optional network telemetry reporting
- **RPC Services** - WebSocket and HTTP endpoints
- **Archive Mode** - Complete historical data retention

## Configuration

### Basic Development Solochain
```kcl
solochain: PolkadotSolochain = {
    name: "dev-solochain"
    version: "1.5.0"
    run_user: {
        name: "polkadot"
        home: "/home/polkadot"
    }
    chain_id: "dev"
    mode: "development"
    base_path: "/var/lib/polkadot-solochain"
    ports: {
        p2p_port: 30333
        ws_port: 9944
        http_port: 9933
    }
    consensus: {
        type: "aura"  # Options: "aura", "elves", "hybrid"
        aura_duration: 6000
        grandpa_interval: 4
    }
    alice_validator: true
}
```

### Production Solochain
```kcl
solochain: PolkadotSolochain = {
    name: "production-solochain"
    version: "1.5.0"
    run_user: {
        name: "polkadot"
        group: "polkadot"
        home: "/opt/polkadot"
    }
    chain_id: "mychain"
    mode: "production"
    base_path: "/var/lib/polkadot-solochain"
    build_path: "/opt/polkadot/solochain-build"
    ports: {
        p2p_port: 30333
        ws_port: 9944
        http_port: 9933
        prometheus_port: 9615
    }
    consensus: {
        type: "aura"  # Options: "aura", "elves", "hybrid"
        aura_duration: 12000
        grandpa_interval: 8
    }
    runtime: {
        pallets: [
            "pallet-balances",
            "pallet-timestamp",
            "pallet-transaction-payment",
            "pallet-sudo"
        ]
        block_time: 12
        block_length_max: 5242880
        block_weights: {
            base_block: 10000000
            max_block: 2000000000000
        }
    }
    bootnodes: [
        "/ip4/198.51.100.1/tcp/30333/p2p/12D3KooW...",
        "/ip4/198.51.100.2/tcp/30333/p2p/12D3KooW..."
    ]
    session_keys: {
        aura_key: "//Alice"
        grandpa_key: "//Alice"
    }
    telemetry_enabled: true
    log_level: "info"
}
```

### Multi-Validator Network
```kcl
solochain: PolkadotSolochain = {
    name: "multi-validator-chain"
    # ... base configuration
    mode: "production"
    consensus: {
        aura_duration: 6000
        grandpa_interval: 4
        authorities: [
            {
                name: "validator-1"
                aura_key: "0x1234..."
                grandpa_key: "0x5678..."
            },
            {
                name: "validator-2"
                aura_key: "0xabcd..."
                grandpa_key: "0xefgh..."
            },
            {
                name: "validator-3"
                aura_key: "0x9876..."
                grandpa_key: "0x5432..."
            }
        ]
    }
    reserved_nodes: [
        "/ip4/10.0.1.1/tcp/30333/p2p/12D3KooW...",
        "/ip4/10.0.1.2/tcp/30333/p2p/12D3KooW...",
        "/ip4/10.0.1.3/tcp/30333/p2p/12D3KooW..."
    ]
    max_peers: 25
    archive_mode: false
}
```

### ELVES Consensus Configuration
```kcl
solochain: PolkadotSolochain = {
    name: "elves-consensus-chain"
    # ... base configuration
    mode: "production"
    consensus: {
        type: "elves"
        elves_config: {
            block_time: 12000
            epoch_duration: 2400  # blocks per epoch
            validators_per_epoch: 21
            proposal_timeout: 3000
            prevote_timeout: 3000
            precommit_timeout: 3000
            commit_timeout: 1000
        }
        finality: {
            type: "grandpa"
            grandpa_interval: 8
        }
    }
    ethereum_compatibility: {
        enabled: true
        chain_id: 1337
        gas_limit_multiplier: 1.0
        gas_price_multiplier: 1.0
        evm_config: {
            create_contract_limit: null
            call_stack_limit: 1024
            public_key_recovery: true
        }
    }
    runtime: {
        pallets: [
            "pallet-evm",
            "pallet-ethereum",
            "pallet-dynamic-fee",
            "pallet-base-fee",
            "pallet-hotfix-sufficients",
            "pallet-balances",
            "pallet-timestamp",
            "pallet-elves-consensus"
        ]
        ethereum_block_time: 12
        ethereum_gas_limit: 15000000
    }
}
```

### Hybrid Consensus Configuration (Aura + ELVES)
```kcl
solochain: PolkadotSolochain = {
    name: "hybrid-consensus-chain"
    # ... base configuration
    consensus: {
        type: "hybrid"
        primary_consensus: "aura"
        secondary_consensus: "elves"
        aura_duration: 6000
        elves_config: {
            fallback_enabled: true
            activation_threshold: 5  # blocks without aura consensus
            deactivation_threshold: 10  # consecutive elves blocks
        }
        transition_rules: {
            aura_to_elves: {
                trigger: "network_partition"
                min_validators: 3
            }
            elves_to_aura: {
                trigger: "network_recovery"
                consensus_timeout: 30000
            }
        }
    }
    runtime: {
        pallets: [
            "pallet-aura",
            "pallet-elves-consensus",
            "pallet-consensus-manager",
            "pallet-balances",
            "pallet-timestamp",
            "pallet-grandpa"
        ]
    }
}
```

### PVM Runtime Configuration
```kcl
solochain: PolkadotSolochain = {
    # ... base configuration
    pvm: {
        enabled: true
        runtime_config: {
            wasm_method: "Compiled"
            default_heap_pages: 64
            max_runtime_instances: 8
            runtime_cache_size: 2
        }
        execution_strategies: {
            syncing: "NativeElseWasm"
            importing: "NativeElseWasm"
            block_construction: "Native"
            offchain_worker: "Native"
            other: "Native"
        }
    }
    runtime: {
        pallets: [
            "frame-system",
            "pallet-balances",
            "pallet-timestamp",
            "pallet-aura",
            "pallet-grandpa",
            "pallet-transaction-payment",
            "pallet-sudo",
            "pallet-utility",
            "pallet-multisig"
        ]
        custom_types: {
            "CustomStruct": {
                "field1": "u32",
                "field2": "Vec<u8>"
            }
        }
    }
}
```

## Usage

### Deploy Solochain
```bash
./core/nulib/provisioning taskserv create polkadot-solochain --infra <infrastructure-name>
```

### List Available Task Services
```bash
./core/nulib/provisioning taskserv list
```

### SSH to Solochain Server
```bash
./core/nulib/provisioning server ssh <solochain-server>
```

### Service Management
```bash
# Check solochain status
systemctl status polkadot-solochain

# Start/stop solochain
systemctl start polkadot-solochain
systemctl stop polkadot-solochain
systemctl restart polkadot-solochain

# View solochain logs
journalctl -u polkadot-solochain -f

# Check build status
systemctl status polkadot-solochain-build
```

### Chain Operations
```bash
# Generate new session keys
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \
  http://localhost:9933/

# Check chain info
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \
  http://localhost:9933/

# Get latest block
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getBlock", "params":[]}' \
  http://localhost:9933/

# Check consensus mechanism (ELVES-specific)
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "elves_getConsensusState", "params":[]}' \
  http://localhost:9933/

# Get ELVES epoch information
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "elves_getCurrentEpoch", "params":[]}' \
  http://localhost:9933/

# Check validator set for ELVES consensus
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "elves_getValidators", "params":[]}' \
  http://localhost:9933/
```

### Development Operations
```bash
# Connect with Polkadot.js Apps
# Navigate to: https://polkadot.js.org/apps/?rpc=ws://localhost:9944

# Purge chain data (development)
sudo -u polkadot /opt/polkadot/target/release/solochain-node purge-chain \
  --chain /var/lib/polkadot-solochain/chain-spec.json \
  --base-path /var/lib/polkadot-solochain

# Export chain state
sudo -u polkadot /opt/polkadot/target/release/solochain-node export-state \
  --chain /var/lib/polkadot-solochain/chain-spec.json \
  --base-path /var/lib/polkadot-solochain
```

### Key Management
```bash
# Generate new validator keys
sudo -u polkadot /opt/polkadot/scripts/generate-keys.sh

# Inspect session keys
sudo -u polkadot /opt/polkadot/target/release/solochain-node key inspect \
  --scheme sr25519 \
  --key-type aura

# Insert key into keystore
sudo -u polkadot /opt/polkadot/target/release/solochain-node key insert \
  --key-type aura \
  --scheme sr25519 \
  --suri "//Alice" \
  --base-path /var/lib/polkadot-solochain
```

## Architecture

### System Architecture
```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   dApps &       │────│  Solochain Node  │────│   Blockchain    │
│   Frontends     │    │                  │    │   Network       │
│                 │    │ • Runtime (WASM) │    │                 │
│ • Polkadot.js   │────│ • Consensus      │────│ • Validators    │
│ • Custom UIs    │    │ • P2P Network    │    │ • Full Nodes    │
│ • Wallets       │    │ • RPC Services   │    │ • Archive Nodes │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

### Runtime Architecture
```
┌─────────────────────────────────────────────────────────────┐
│                    Solochain Runtime                        │
├─────────────────────────────────────────────────────────────┤
│  System Pallets  │  Consensus           │  Utility Pallets       │
│                  │                      │                        │
│ • frame-system   │ • pallet-aura        │ • pallet-balances     │
│ • pallet-timestamp│ • pallet-elves      │ • pallet-sudo        │
│ • pallet-evm     │ • pallet-grandpa     │ • pallet-utility      │
│ • pallet-ethereum│ • consensus-manager  │ • pallet-dynamic-fee   │
├─────────────────────────────────────────────────────────────┤
│                    Substrate Framework                      │
├─────────────────────────────────────────────────────────────┤
│              Polkadot Virtual Machine (PVM)                │
└─────────────────────────────────────────────────────────────┘
```

### Network Ports
- **P2P Port (30333)** - Peer-to-peer blockchain network
- **WebSocket Port (9944)** - Real-time RPC and subscriptions  
- **HTTP Port (9933)** - Standard RPC API calls
- **Prometheus Port (9615)** - Metrics and monitoring

### File Structure
```
/var/lib/polkadot-solochain/  # Main data directory
├── chains/                   # Chain-specific data
│   └── mychain/             # Custom chain data
├── keystore/                # Validator keys
├── chain-spec.json          # Chain specification
└── node.key                 # Node identity

/opt/polkadot/               # Build directory
├── solochain-template/      # Source code
├── target/release/          # Compiled binaries
├── scripts/                 # Management scripts
└── runtime/                 # Runtime configuration

/etc/polkadot-solochain/     # Configuration
├── solochain.conf          # Main configuration
└── runtime.toml            # Runtime settings
```

## Supported Operating Systems

- Ubuntu 20.04+ / Debian 11+
- CentOS 8+ / RHEL 8+ / Fedora 35+

## System Requirements

### Development Environment
- **RAM**: 4GB (8GB recommended)
- **Storage**: 50GB SSD (for build artifacts and chain data)
- **CPU**: 4 cores (compilation requirements)
- **Network**: Standard internet connection

### Production Single Node
- **RAM**: 8GB (16GB recommended)
- **Storage**: 100GB SSD (200GB+ recommended)
- **CPU**: 4 cores (8 cores recommended)
- **Network**: Stable, low-latency connection

### Production Multi-Validator Network
- **RAM**: 16GB+ per validator
- **Storage**: 200GB+ SSD per node
- **CPU**: 8+ cores per validator
- **Network**: Dedicated network with low latency between validators

### Build Requirements
- **RAM**: 8GB+ (for Rust compilation)
- **Storage**: 10GB+ free space for build artifacts
- **CPU**: Multi-core for parallel compilation
- **Time**: 30-60 minutes for initial build

## Troubleshooting

### Build Issues
```bash
# Check build logs
journalctl -u polkadot-solochain-build -f

# Manual build
cd /opt/polkadot/solochain-template
sudo -u polkadot cargo build --release

# Clean build
sudo -u polkadot cargo clean
sudo -u polkadot cargo build --release

# Check Rust version
rustc --version
cargo --version
```

### Runtime Issues
```bash
# Check runtime compilation
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "state_getRuntimeVersion", "params":[]}' \
  http://localhost:9933/

# Validate chain spec
sudo -u polkadot /opt/polkadot/target/release/solochain-node \
  chain-spec-builder validate \
  --chain /var/lib/polkadot-solochain/chain-spec.json

# Check pallet configuration
journalctl -u polkadot-solochain | grep -i pallet
```

### Consensus Issues
```bash
# Check validator status
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "author_hasSessionKeys", "params":["0x..."]}' \
  http://localhost:9933/

# Monitor block production
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "chain_subscribeFinalizedHeads", "params":[]}' \
  ws://localhost:9944/

# Check session keys
ls -la /var/lib/polkadot-solochain/keystore/

# ELVES consensus troubleshooting
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "elves_getConsensusHealth", "params":[]}' \
  http://localhost:9933/

# Check ELVES validator participation
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "elves_getValidatorParticipation", "params":[]}' \
  http://localhost:9933/

# Monitor ELVES consensus transitions (hybrid mode)
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "elves_getConsensusTransitions", "params":[]}' \
  http://localhost:9933/
```

### Network Issues
```bash
# Check connected peers
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "system_peers", "params":[]}' \
  http://localhost:9933/

# Test network connectivity
telnet target-node-ip 30333

# Check node identity
sudo -u polkadot /opt/polkadot/target/release/solochain-node key inspect-node-key \
  --file /var/lib/polkadot-solochain/node.key
```

### Performance Issues
```bash
# Check system resources
htop
df -h /var/lib/polkadot-solochain
iostat -x 1

# Monitor chain metrics
curl http://localhost:9615/metrics

# Check block times
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \
  http://localhost:9933/
```

## Security Considerations

### Validator Security
- **Key Management** - Secure storage of session keys
- **Network Isolation** - Isolate validators from public internet
- **Access Control** - Limit administrative access
- **Monitoring** - Comprehensive validator monitoring

### Node Security
- **User Isolation** - Run as dedicated system user
- **File Permissions** - Secure keystore and configuration files
- **System Updates** - Keep OS and dependencies updated
- **Firewall Rules** - Limit network access to necessary ports

### Runtime Security
- **Code Review** - Review all custom pallets and runtime code
- **Testing** - Comprehensive testing of runtime logic
- **Upgrades** - Secure runtime upgrade procedures
- **Audit** - Regular security audits of custom code

## Performance Optimization

### Build Optimization
- **Parallel Builds** - Use multiple CPU cores for compilation
- **Target CPU** - Optimize for target deployment hardware
- **Link-Time Optimization** - Enable LTO for better performance
- **Profile-Guided Optimization** - Use PGO for hot paths

### Runtime Optimization
- **Execution Strategy** - Use native execution where possible
- **Cache Configuration** - Optimize database and state caches
- **Block Parameters** - Tune block time and weight limits
- **Pallet Selection** - Include only necessary pallets

### System Optimization
- **Storage Performance** - Use high-IOPS NVMe storage
- **Memory Allocation** - Configure appropriate heap sizes
- **Network Tuning** - Optimize TCP settings for validator networks
- **CPU Affinity** - Pin processes to specific CPU cores

## Development Workflow

### Adding Custom Pallets
```bash
# 1. Add pallet to Cargo.toml
vim /opt/polkadot/solochain-template/runtime/Cargo.toml

# 2. Configure pallet in runtime
vim /opt/polkadot/solochain-template/runtime/src/lib.rs

# 3. Rebuild runtime
cd /opt/polkadot/solochain-template
cargo build --release

# 4. Restart node with new runtime
systemctl restart polkadot-solochain
```

### Chain Spec Management
```bash
# Generate new chain spec
sudo -u polkadot /opt/polkadot/target/release/solochain-node build-spec \
  --disable-default-bootnode > /tmp/plain-spec.json

# Convert to raw format
sudo -u polkadot /opt/polkadot/target/release/solochain-node build-spec \
  --chain /tmp/plain-spec.json --raw > /var/lib/polkadot-solochain/chain-spec.json

# Update genesis configuration
vim /tmp/plain-spec.json  # Edit genesis config
# Then convert to raw format
```

### Testing and Debugging
```bash
# Run unit tests
cd /opt/polkadot/solochain-template
cargo test

# Run integration tests
cargo test --features runtime-benchmarks

# Enable debug logging
systemctl edit polkadot-solochain
# Add: Environment="RUST_LOG=debug"

# Runtime debugging
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "state_call", "params":["Core_version", "0x"]}' \
  http://localhost:9933/
```

## Resources

- **Substrate Documentation**: [docs.substrate.io](https://docs.substrate.io)
- **Polkadot SDK**: [github.com/paritytech/polkadot-sdk](https://github.com/paritytech/polkadot-sdk)
- **Substrate Node Template**: [github.com/substrate-developer-hub/substrate-node-template](https://github.com/substrate-developer-hub/substrate-node-template)
- **Polkadot.js Documentation**: [polkadot.js.org/docs](https://polkadot.js.org/docs)
- **Substrate Stack Exchange**: [substrate.stackexchange.com](https://substrate.stackexchange.com)