provisioning/provisioning-platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
provisioning-platform/prov-ecosystem/crates/observability/INTEGRATION.md

660 lines
14 KiB
Markdown
Raw Blame History

# Observability Integration Guide
This guide covers integrating the observability crate with other prov-ecosystem components and external systems.
## Table of Contents
1. [Integration with Other Crates](#integration-with-other-crates)
2. [Docker Integration](#docker-integration)
3. [Kubernetes Integration](#kubernetes-integration)
4. [Grafana Setup](#grafana-setup)
5. [OpenTelemetry Collector](#opentelemetry-collector)
6. [Init Systems](#init-systems)
7. [Cloud Providers](#cloud-providers)
## Integration with Other Crates
### Backup Module
Monitor backup operations and integrate with observability:
```rust
use observability::metrics;
use backup::client::BackupClient;
async fn monitor_backup() -> Result<(), Box<dyn std::error::Error>> {
let backup_client = BackupClient::from_config(config)?;
let start_time = std::time::Instant::now();
match backup_client.backup().await {
Ok(result) => {
let duration = start_time.elapsed().as_secs_f64();
observability::metrics::counter!("backup_success_total").increment();
observability::metrics::gauge!("backup_data_bytes").set(result.data_size as f64);
observability::metrics::histogram!("backup_duration_seconds").record(duration);
tracing::info!(
duration_secs = duration,
data_bytes = result.data_size,
"Backup completed successfully"
);
}
Err(e) => {
observability::metrics::counter!("backup_failure_total").increment();
tracing::error!(error = %e, "Backup failed");
}
}
Ok(())
}
```
### Init-Servs Module
Integrate service management with observability:
```rust
use observability::health::checks::{HealthCheckRegistry, HealthStatus};
use init_servs::ServiceManager;
fn setup_service_health_checks(
manager: &ServiceManager,
registry: &HealthCheckRegistry,
) -> Result<(), Box<dyn std::error::Error>> {
let manager_clone = manager.clone();
registry.register("services_running", move || {
match manager_clone.list_services() {
Ok(services) => {
let running = services.iter().filter(|s| s.is_running).count();
if running == services.len() {
HealthStatus::Healthy
} else {
HealthStatus::Degraded
}
}
Err(_) => HealthStatus::Unhealthy,
}
});
Ok(())
}
```
### Runtime Module
Monitor container runtime metrics:
```rust
use observability::metrics;
use runtime::ContainerRuntime;
async fn monitor_containers() -> Result<(), Box<dyn std::error::Error>> {
let runtime = ContainerRuntime::new()?;
for container in runtime.list_containers()? {
observability::metrics::gauge!(
"container_memory_bytes"
).set(container.memory_usage as f64);
observability::metrics::gauge!(
"container_cpu_shares"
).set(container.cpu_shares as f64);
}
Ok(())
}
```
### Encryption Module
Track encryption operations:
```rust
use observability::metrics;
use encrypt::EncryptionClient;
async fn encrypt_with_monitoring(
client: &EncryptionClient,
data: &[u8],
) -> Result<Vec<u8>, Box<dyn std::error::Error>> {
let start = std::time::Instant::now();
let encrypted = client.encrypt(data).await?;
let duration = start.elapsed().as_secs_f64();
observability::metrics::histogram!("encryption_duration_seconds").record(duration);
observability::metrics::counter!("encryption_operations_total").increment();
Ok(encrypted)
}
```
## Docker Integration
### Dockerfile with Health Checks
```dockerfile
FROM rust:1.75
WORKDIR /app
COPY . .
RUN cargo build --release
# Health check using observability endpoints
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -f http://localhost:8080/healthz || exit 1
EXPOSE 8080 9090
CMD ["./target/release/my-app"]
```
### Docker Compose
```yaml
version: '3.8'
services:
app:
build: .
ports:
- "8080:8080" # Health checks
- "9090:9090" # Prometheus metrics
environment:
RUST_LOG: info
OBS_ENVIRONMENT: docker
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/healthz"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
prometheus:
image: prom/prometheus:latest
ports:
- "9091:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- ./dashboards:/etc/grafana/provisioning/dashboards
```
### Prometheus Configuration
```yaml
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'app'
static_configs:
- targets: ['localhost:9090']
scrape_interval: 10s
scrape_timeout: 5s
```
## Kubernetes Integration
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
containers:
- name: my-app
image: my-app:latest
ports:
- containerPort: 8080
name: health
- containerPort: 9090
name: metrics
env:
- name: OBS_ENVIRONMENT
value: "kubernetes"
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# Health checks
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
startupProbe:
httpGet:
path: /startup
port: 8080
initialDelaySeconds: 0
periodSeconds: 2
timeoutSeconds: 3
failureThreshold: 30
# Resource limits
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
```
### ServiceMonitor for Prometheus Operator
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
interval: 30s
path: /metrics
```
### Configuration via ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: my-app-observability
namespace: default
data:
observability.toml: |
[context]
service_name = "my-app"
service_version = "1.0.0"
environment = "kubernetes"
namespace = "default"
[logging]
level = "info"
format = "json"
include_spans = false
outputs = ["stdout"]
[metrics]
enabled = true
export_interval_secs = 60
[health]
enabled = true
port = 8080
liveness_path = "/healthz"
readiness_path = "/ready"
startup_path = "/startup"
---
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: my-app
image: my-app:latest
volumeMounts:
- name: config
mountPath: /etc/my-app
volumes:
- name: config
configMap:
name: my-app-observability
```
## Grafana Setup
### Adding Prometheus Data Source
```json
{
"name": "Prometheus",
"type": "prometheus",
"url": "http://prometheus:9090",
"access": "proxy",
"isDefault": true
}
```
### Provisioning Dashboards
Create `provisioning/dashboards/provider.yaml`:
```yaml
apiVersion: 1
providers:
- name: 'Observability Dashboards'
orgId: 1
folder: 'Services'
type: file
disableDeletion: false
editable: true
options:
path: /etc/grafana/provisioning/dashboards
```
Copy dashboard JSON files to `/etc/grafana/provisioning/dashboards/`:
```bash
cp dashboards/*.json /etc/grafana/provisioning/dashboards/
```
### Alerting Rules
Create `rules.yml`:
```yaml
groups:
- name: app_alerts
interval: 30s
rules:
- alert: HighErrorRate
expr: rate(http_errors_total[5m]) > 0.05
for: 5m
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: ServiceDown
expr: up == 0
for: 1m
annotations:
summary: "Service is down"
description: "{{ $labels.job }} on {{ $labels.instance }} is down"
- alert: HighLatency
expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
for: 5m
annotations:
summary: "High latency detected"
description: "p95 latency is {{ $value }}s"
```
## OpenTelemetry Collector
### Docker Compose with OTEL Collector
```yaml
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
ports:
- "4317:4317" # gRPC
- "4318:4318" # HTTP
- "8888:8888" # Prometheus metrics
volumes:
- ./otel-config.yaml:/etc/otel/config.yaml
command: ["--config=/etc/otel/config.yaml"]
```
### OpenTelemetry Collector Configuration
`otel-config.yaml`:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['localhost:8888']
processors:
batch:
send_batch_size: 1024
timeout: 10s
memory_limiter:
check_interval: 1s
limit_mib: 1024
exporters:
prometheus:
endpoint: "0.0.0.0:8888"
jaeger:
endpoint: "jaeger:14250"
otlp:
client:
endpoint: "datadog-agent:4317"
service:
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [memory_limiter, batch]
exporters: [prometheus, otlp]
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger, otlp]
```
## Init Systems
### Systemd Service with Health Checks
```ini
[Unit]
Description=My Application
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/my-app
ExecReload=/bin/kill -HUP $MAINPID
Environment="RUST_LOG=info"
Environment="OBS_ENVIRONMENT=systemd"
Environment="OBS_CONFIG=/etc/my-app/observability.toml"
# Health check via systemd
ExecStartPost=/usr/bin/curl -f http://localhost:8080/healthz
# Restart policy
Restart=on-failure
RestartSec=10
# Resource limits
MemoryMax=512M
CPUQuota=50%
[Install]
WantedBy=multi-user.target
```
### Monitoring with journalctl
```bash
# View recent logs
journalctl -u my-app -n 100 -f
# Parse JSON structured logs
journalctl -u my-app -o json | jq '.MESSAGE | fromjson?'
```
## Cloud Providers
### AWS CloudWatch Integration
Use CloudWatch exporter or send logs to CloudWatch:
```rust
// Configuration for CloudWatch
[logging]
outputs = ["stdout", "cloudwatch"]
[logging.cloudwatch]
region = "us-east-1"
log_group = "/aws/services/my-app"
log_stream = "production"
```
### Google Cloud Monitoring
Configure for Google Cloud logging:
```rust
[logging]
format = "json"
outputs = ["stdout"]
[context]
environment = "gcp"
attributes = { "gcp_project" = "my-project" }
```
### Azure Monitor
Send metrics to Azure Monitor:
```toml
[metrics.otlp]
endpoint = "https://<workspace-id>.ods.opinsights.azure.com/api/logs"
protocol = "http"
headers = { "Authorization" = "Bearer <token>" }
```
## Performance Tuning
### For High-Throughput Scenarios
```toml
[logging]
# Reduce overhead
level = "warn"
include_spans = false
[metrics]
export_interval_secs = 120
[metrics.prometheus]
# Histogram buckets optimized for typical latencies
histogram_buckets = [0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
[health]
# Disable if not needed in high-throughput
enabled = false
```
### For Development
```toml
[logging]
level = "debug"
format = "pretty"
include_spans = true
include_target = true
[metrics]
export_interval_secs = 10
[health]
enabled = true
port = 8080
```
## Troubleshooting
### Metrics Not Appearing
1. Check Prometheus targets at `http://localhost:9091/targets`
2. Verify metrics endpoint: `curl http://localhost:9090/metrics`
3. Check logs for initialization errors
### Health Checks Timing Out
1. Verify port binding: `curl http://localhost:8080/healthz`
2. Check firewall rules
3. Review probe configuration in Kubernetes
### High Memory Usage
1. Reduce histogram buckets
2. Decrease log verbosity
3. Increase batch timeout for OTLP
### Logs Not Appearing
1. Check `RUST_LOG` environment variable
2. Verify configured outputs (stdout/file)
3. Check file permissions for file output
## Best Practices
1. **Development**: Use pretty format with debug logging
2. **Staging**: Use JSON format with info logging
3. **Production**: Use JSON format with warn logging
4. **Health Checks**: Always implement for K8s deployments
5. **Metrics**: Use consistent naming conventions
6. **Dashboards**: Create dashboards for critical metrics
7. **Alerts**: Set up alerts for error rates and latency
Reference in a new issue View git blame Copy permalink