596 lines
14 KiB
Markdown
596 lines
14 KiB
Markdown
# Doc-Lifecycle-Manager Integration Guide
|
|
|
|
## Overview
|
|
|
|
**doc-lifecycle-manager** (external project) provides complete documentation lifecycle management for VAPORA, including classification, consolidation, semantic search, real-time updates, and enterprise security features.
|
|
|
|
**Project Location**: External project (doc-lifecycle-manager)
|
|
**Status**: ✅ **Enterprise-Ready**
|
|
**Tests**: 155/155 passing | Zero unsafe code
|
|
|
|
---
|
|
|
|
## What is doc-lifecycle-manager
|
|
|
|
A comprehensive Rust-based system that handles documentation throughout its entire lifecycle:
|
|
|
|
### Core Capabilities (Phases 1-3)
|
|
- **Automatic Classification**: Categorizes docs (vision, design, specs, ADRs, guides, testing, archive)
|
|
- **Duplicate Detection**: Finds similar documents with TF-IDF analysis
|
|
- **Semantic RAG Indexing**: Vector embeddings for semantic search
|
|
- **mdBook Generation**: Auto-generates documentation websites
|
|
|
|
### Enterprise Features (Phases 4-7)
|
|
- **GraphQL API**: Semantic document queries with pagination
|
|
- **Real-Time Events**: WebSocket streaming of doc updates
|
|
- **Distributed Tracing**: OpenTelemetry with W3C Trace Context
|
|
- **Security**: mTLS with automatic certificate rotation
|
|
- **Performance**: Comprehensive benchmarking with percentiles
|
|
- **Persistence**: SurrealDB backend (feature-gated)
|
|
|
|
---
|
|
|
|
## Integration Architecture
|
|
|
|
### Data Flow in VAPORA
|
|
|
|
```
|
|
Frontend/Agents
|
|
↓
|
|
┌─────────────────────────────────┐
|
|
│ VAPORA API Layer (Axum) │
|
|
│ ├─ REST endpoints │
|
|
│ └─ WebSocket gateway │
|
|
└─────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────┐
|
|
│ doc-lifecycle-manager Services │
|
|
│ │
|
|
│ ├─ GraphQL Resolver │
|
|
│ ├─ WebSocket Manager │
|
|
│ ├─ Document Classifier │
|
|
│ ├─ RAG Indexer │
|
|
│ └─ mTLS Auth Manager │
|
|
└─────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────┐
|
|
│ Data Layer │
|
|
│ ├─ SurrealDB (vectors) │
|
|
│ ├─ NATS JetStream (events) │
|
|
│ └─ Redis (cache) │
|
|
└─────────────────────────────────┘
|
|
```
|
|
|
|
### Component Integration Points
|
|
|
|
**1. Documenter Agent ↔ doc-lifecycle-manager**
|
|
```rust
|
|
use vapora_doc_lifecycle::prelude::*;
|
|
|
|
// On task completion
|
|
async fn on_task_completed(task_id: &str) {
|
|
let config = PluginConfig::default();
|
|
let mut docs = DocumenterIntegration::new(config)?;
|
|
docs.on_task_completed(task_id).await?;
|
|
}
|
|
```
|
|
|
|
**2. Frontend ↔ GraphQL API**
|
|
```graphql
|
|
{
|
|
documentSearch(query: {
|
|
text_query: "authentication"
|
|
limit: 10
|
|
}) {
|
|
results { id title relevance_score }
|
|
}
|
|
}
|
|
```
|
|
|
|
**3. Frontend ↔ WebSocket Events**
|
|
```javascript
|
|
const ws = new WebSocket("ws://vapora/doc-events");
|
|
ws.onmessage = (event) => {
|
|
const { event_type, payload } = JSON.parse(event.data);
|
|
// Update UI on document_indexed, document_updated, etc.
|
|
};
|
|
```
|
|
|
|
**4. Agent-to-Agent ↔ NATS JetStream**
|
|
```
|
|
Task Completed Event
|
|
→ Documenter Agent (NATS)
|
|
→ Classify + Index
|
|
→ Broadcast DocumentIndexed Event
|
|
→ All Agents notified
|
|
```
|
|
|
|
---
|
|
|
|
## Feature Set by Phase
|
|
|
|
### Phase 1: Foundation & Core Library ✅
|
|
- Error handling and configuration
|
|
- Core abstractions and types
|
|
|
|
### Phase 2: Extended Implementation ✅
|
|
- Document Classifier (7 types)
|
|
- Consolidator (TF-IDF)
|
|
- RAG Indexer (markdown-aware)
|
|
- MDBook Generator
|
|
|
|
### Phase 3: CLI & Automation ✅
|
|
- 4 command handlers
|
|
- 62+ Just recipes
|
|
- 5 NuShell scripts
|
|
|
|
### Phase 4: VAPORA Deep Integration ✅
|
|
- NATS JetStream events
|
|
- Vector store trait
|
|
- Plugin system
|
|
- Agent coordination
|
|
|
|
### Phase 5: Production Hardening ✅
|
|
- Real NATS integration
|
|
- DocServer RBAC (4 roles, 3 visibility levels)
|
|
- Root Files Keeper (auto-update README, CHANGELOG)
|
|
- Kubernetes manifests (7 YAML files)
|
|
|
|
### Phase 6: Multi-Agent VAPORA ✅
|
|
- Agent registry with health checking
|
|
- CI/CD pipeline (GitHub Actions)
|
|
- Prometheus monitoring rules
|
|
- Comprehensive documentation
|
|
|
|
### Phase 7: Advanced Features ✅
|
|
- **SurrealDB Backend**: Persistent vector store
|
|
- **OpenTelemetry**: W3C Trace Context support
|
|
- **GraphQL API**: Query builder with semantic search
|
|
- **WebSocket Events**: Real-time subscriptions
|
|
- **mTLS Auth**: Certificate rotation
|
|
- **Benchmarking**: P95/P99 metrics
|
|
|
|
---
|
|
|
|
## How to Use in VAPORA
|
|
|
|
### 1. Basic Integration (Documenter Agent)
|
|
|
|
```rust
|
|
// In vapora-backend/documenter_agent.rs
|
|
|
|
use vapora_doc_lifecycle::prelude::*;
|
|
|
|
impl DocumenterAgent {
|
|
async fn process_task(&self, task: Task) -> Result<()> {
|
|
let config = PluginConfig::default();
|
|
let mut integration = DocumenterIntegration::new(config)?;
|
|
|
|
// Automatically classifies, indexes, and generates docs
|
|
integration.on_task_completed(&task.id).await?;
|
|
|
|
Ok(())
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. GraphQL Queries (Frontend/Agents)
|
|
|
|
```graphql
|
|
# Search for documentation
|
|
query SearchDocs($query: String!) {
|
|
documentSearch(query: {
|
|
text_query: $query
|
|
limit: 10
|
|
visibility: "Public"
|
|
}) {
|
|
results {
|
|
id
|
|
title
|
|
path
|
|
relevance_score
|
|
preview
|
|
}
|
|
total_count
|
|
has_more
|
|
}
|
|
}
|
|
|
|
# Get specific document
|
|
query GetDoc($id: ID!) {
|
|
document(id: $id) {
|
|
id
|
|
title
|
|
content
|
|
metadata {
|
|
created_at
|
|
updated_at
|
|
owner_id
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3. Real-Time Updates (Frontend)
|
|
|
|
```javascript
|
|
// Connect to doc-lifecycle WebSocket
|
|
const docWs = new WebSocket('ws://vapora-api/doc-lifecycle/events');
|
|
|
|
// Subscribe to document changes
|
|
docWs.onopen = () => {
|
|
docWs.send(JSON.stringify({
|
|
type: 'subscribe',
|
|
event_types: ['document_indexed', 'document_updated', 'search_index_rebuilt'],
|
|
min_priority: 5
|
|
}));
|
|
};
|
|
|
|
// Handle updates
|
|
docWs.onmessage = (event) => {
|
|
const message = JSON.parse(event.data);
|
|
|
|
if (message.event_type === 'document_indexed') {
|
|
console.log('New doc indexed:', message.payload);
|
|
// Refresh documentation view
|
|
}
|
|
};
|
|
```
|
|
|
|
### 4. Distributed Tracing
|
|
|
|
All operations are automatically traced:
|
|
|
|
```
|
|
GET /api/documents?search=auth
|
|
trace_id: 0af7651916cd43dd8448eb211c80319c
|
|
span_id: b7ad6b7169203331
|
|
|
|
├─ graphql_resolver [15ms]
|
|
│ ├─ rbac_check [2ms]
|
|
│ └─ semantic_search [12ms]
|
|
└─ response [1ms]
|
|
```
|
|
|
|
### 5. mTLS Security
|
|
|
|
Service-to-service communication is secured:
|
|
|
|
```yaml
|
|
# Kubernetes secret for certs
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: doc-lifecycle-certs
|
|
data:
|
|
server.crt: <base64>
|
|
server.key: <base64>
|
|
ca.crt: <base64>
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment in VAPORA
|
|
|
|
### Kubernetes Manifests Provided
|
|
|
|
```
|
|
kubernetes/
|
|
├── namespace.yaml # Create doc-lifecycle namespace
|
|
├── configmap.yaml # Configuration
|
|
├── deployment.yaml # Main service (2 replicas)
|
|
├── statefulset-nats.yaml # NATS JetStream (3 replicas)
|
|
├── statefulset-surreal.yaml # SurrealDB (1 replica)
|
|
├── service.yaml # Internal services
|
|
├── rbac.yaml # RBAC configuration
|
|
└── prometheus-rules.yaml # Monitoring rules
|
|
```
|
|
|
|
### Quick Deploy
|
|
|
|
```bash
|
|
# Deploy to VAPORA cluster
|
|
kubectl apply -f /Tools/doc-lifecycle-manager/kubernetes/
|
|
|
|
# Verify
|
|
kubectl get pods -n doc-lifecycle
|
|
kubectl get svc -n doc-lifecycle
|
|
```
|
|
|
|
### Configuration via ConfigMap
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: doc-lifecycle-config
|
|
namespace: doc-lifecycle
|
|
data:
|
|
config.json: |
|
|
{
|
|
"mode": "full",
|
|
"classification": {
|
|
"auto_classify": true,
|
|
"confidence_threshold": 0.8
|
|
},
|
|
"rag": {
|
|
"enable_embeddings": true,
|
|
"max_chunk_size": 512
|
|
},
|
|
"nats": {
|
|
"server": "nats://nats:4222",
|
|
"jetstream_enabled": true
|
|
},
|
|
"otel": {
|
|
"enabled": true,
|
|
"jaeger_endpoint": "http://jaeger:14268"
|
|
},
|
|
"mtls": {
|
|
"enabled": true,
|
|
"rotation_days": 30
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## VAPORA Agent Integration
|
|
|
|
### Documenter Agent
|
|
|
|
```rust
|
|
// Processes documentation tasks
|
|
pub struct DocumenterAgent {
|
|
integration: DocumenterIntegration,
|
|
nats: NatsEventHandler,
|
|
}
|
|
|
|
impl DocumenterAgent {
|
|
pub async fn handle_task(&self, task: Task) -> Result<()> {
|
|
// 1. Classify document
|
|
self.integration.on_task_completed(&task.id).await?;
|
|
|
|
// 2. Broadcast via NATS
|
|
let event = DocsUpdatedEvent {
|
|
task_id: task.id,
|
|
doc_count: 5,
|
|
};
|
|
self.nats.publish_docs_updated(event).await?;
|
|
|
|
Ok(())
|
|
}
|
|
}
|
|
```
|
|
|
|
### Developer Agent (Uses Search)
|
|
|
|
```rust
|
|
// Searches for relevant documentation
|
|
pub struct DeveloperAgent;
|
|
|
|
impl DeveloperAgent {
|
|
pub async fn find_relevant_docs(&self, task: Task) -> Result<Vec<DocumentResult>> {
|
|
// GraphQL query for semantic search
|
|
let query = DocumentQuery {
|
|
text_query: Some(task.description),
|
|
limit: Some(5),
|
|
visibility: Some("Internal".to_string()),
|
|
..Default::default()
|
|
};
|
|
|
|
// Execute search
|
|
resolver.resolve_document_search(query, user).await
|
|
}
|
|
}
|
|
```
|
|
|
|
### CodeReviewer Agent (Uses Context)
|
|
|
|
```rust
|
|
// Uses documentation as context for reviews
|
|
pub struct CodeReviewerAgent;
|
|
|
|
impl CodeReviewerAgent {
|
|
pub async fn review_with_context(&self, code: &str) -> Result<Review> {
|
|
// Search for related documentation
|
|
let docs = semantic_search(code_summary).await?;
|
|
|
|
// Use docs as context in review
|
|
let review = llm_client
|
|
.review_code(code, &docs.to_context_string())
|
|
.await?;
|
|
|
|
Ok(review)
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Performance & Scaling
|
|
|
|
### Expected Performance
|
|
|
|
| Operation | Latency | Throughput |
|
|
|-----------|---------|-----------|
|
|
| Classify doc | <10ms | 1000 docs/sec |
|
|
| GraphQL query | <200ms | 50 queries/sec |
|
|
| WebSocket broadcast | <20ms | 1000 events/sec |
|
|
| Semantic search | <100ms | 50 searches/sec |
|
|
| mTLS validation | <5ms | N/A |
|
|
|
|
### Resource Requirements
|
|
|
|
**Deployment Resources**:
|
|
- CPU: 2-4 cores (main service)
|
|
- Memory: 512MB-2GB
|
|
- Storage: 50GB (SurrealDB + vectors)
|
|
|
|
**NATS Requirements**:
|
|
- CPU: 1-2 cores
|
|
- Memory: 256MB-1GB
|
|
- Persistent volume: 20GB
|
|
|
|
---
|
|
|
|
## Monitoring & Observability
|
|
|
|
### Prometheus Metrics
|
|
|
|
```promql
|
|
# Error rate
|
|
rate(doc_lifecycle_errors_total[5m])
|
|
|
|
# Latency
|
|
histogram_quantile(0.99, doc_lifecycle_request_duration_seconds)
|
|
|
|
# Service availability
|
|
up{job="doc-lifecycle"}
|
|
```
|
|
|
|
### Distributed Tracing
|
|
|
|
Traces are sent to Jaeger in W3C format:
|
|
|
|
```
|
|
Trace: 0af7651916cd43dd8448eb211c80319c
|
|
├─ Span: graphql_resolver
|
|
│ ├─ Span: rbac_check
|
|
│ └─ Span: semantic_search
|
|
└─ Span: response
|
|
```
|
|
|
|
### Health Checks
|
|
|
|
```bash
|
|
# Liveness probe
|
|
curl http://doc-lifecycle:8080/health/live
|
|
|
|
# Readiness probe
|
|
curl http://doc-lifecycle:8080/health/ready
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration Reference
|
|
|
|
### Environment Variables
|
|
|
|
```bash
|
|
# Core
|
|
DOC_LIFECYCLE_MODE=full # minimal|standard|full
|
|
DOC_LIFECYCLE_ENABLED=true
|
|
|
|
# Classification
|
|
CLASSIFIER_AUTO_CLASSIFY=true
|
|
CLASSIFIER_CONFIDENCE_THRESHOLD=0.8
|
|
|
|
# RAG/Search
|
|
RAG_ENABLE_EMBEDDINGS=true
|
|
RAG_MAX_CHUNK_SIZE=512
|
|
RAG_CHUNK_OVERLAP=50
|
|
|
|
# NATS
|
|
NATS_SERVER_URL=nats://nats:4222
|
|
NATS_JETSTREAM_ENABLED=true
|
|
|
|
# SurrealDB (optional)
|
|
SURREAL_DB_URL=ws://surrealdb:8000
|
|
SURREAL_NAMESPACE=vapora
|
|
SURREAL_DATABASE=documents
|
|
|
|
# OpenTelemetry
|
|
OTEL_ENABLED=true
|
|
OTEL_JAEGER_ENDPOINT=http://jaeger:14268
|
|
OTEL_SERVICE_NAME=vapora-doc-lifecycle
|
|
|
|
# mTLS
|
|
MTLS_ENABLED=true
|
|
MTLS_SERVER_CERT=/etc/vapora/certs/server.crt
|
|
MTLS_SERVER_KEY=/etc/vapora/certs/server.key
|
|
MTLS_CA_CERT=/etc/vapora/certs/ca.crt
|
|
MTLS_ROTATION_DAYS=30
|
|
```
|
|
|
|
---
|
|
|
|
## Integration Checklist
|
|
|
|
### Immediate (Ready Now)
|
|
- [x] Core features (Phases 1-3)
|
|
- [x] VAPORA integration (Phase 4)
|
|
- [x] Production hardening (Phase 5)
|
|
- [x] Multi-agent support (Phase 6)
|
|
- [x] Enterprise features (Phase 7)
|
|
- [x] Kubernetes deployment
|
|
- [x] GraphQL API
|
|
- [x] WebSocket events
|
|
- [x] Distributed tracing
|
|
- [x] mTLS security
|
|
|
|
### Planned (Phase 8)
|
|
- [ ] Jaeger exporter
|
|
- [ ] SurrealDB live testing
|
|
- [ ] Load testing
|
|
- [ ] Performance tuning
|
|
- [ ] Production deployment guide
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
**1. NATS Connection Failed**
|
|
```bash
|
|
# Check NATS service
|
|
kubectl get svc -n doc-lifecycle
|
|
kubectl logs -n doc-lifecycle deployment/nats
|
|
```
|
|
|
|
**2. GraphQL Query Timeout**
|
|
```bash
|
|
# Check semantic search performance
|
|
# Query execution should be < 200ms
|
|
# Check RAG index size
|
|
```
|
|
|
|
**3. WebSocket Disconnection**
|
|
```bash
|
|
# Verify WebSocket port is open
|
|
# Check subscription history size
|
|
# Monitor event broadcast latency
|
|
```
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
**Documentation Files**:
|
|
- `/Tools/doc-lifecycle-manager/PHASE_7_COMPLETION.md` - Phase 7 details
|
|
- `/Tools/doc-lifecycle-manager/PHASES_COMPLETION.md` - All phases overview
|
|
- `/Tools/doc-lifecycle-manager/INTEGRATION_WITH_VAPORA.md` - Integration guide
|
|
- `/Tools/doc-lifecycle-manager/kubernetes/README.md` - K8s deployment
|
|
|
|
**Source Code**:
|
|
- `crates/vapora-doc-lifecycle/src/lib.rs` - Main library
|
|
- `crates/vapora-doc-lifecycle/src/graphql_api.rs` - GraphQL resolver
|
|
- `crates/vapora-doc-lifecycle/src/websocket_events.rs` - WebSocket manager
|
|
- `crates/vapora-doc-lifecycle/src/mtls_auth.rs` - Security
|
|
|
|
---
|
|
|
|
## Support
|
|
|
|
For questions or issues:
|
|
1. Check documentation in `/Tools/doc-lifecycle-manager/`
|
|
2. Review test cases for usage examples
|
|
3. Check Kubernetes logs: `kubectl logs -n doc-lifecycle <pod>`
|
|
4. Monitor with Prometheus/Grafana
|
|
|
|
---
|
|
|
|
**Status**: ✅ Ready for Production Deployment
|
|
**Last Updated**: 2025-11-10
|
|
**Maintainer**: VAPORA Team
|