jester/venture
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9d7d8d7be7
venture/technical-stack.md

13 KiB
Raw Blame History

Technical Stack & Architecture

Overview

This document outlines the technical infrastructure, tools, and architecture for all three phases of the venture.

Phase 1: Consulting (Customer Hardware)

Deployment Environment

Location: Customer premises
Hardware: Customer-provided or recommended purchase

  • Raspberry Pi 4 (8GB): $75-100
  • OR repurposed industrial PC (free if available)
  • OR Intel NUC ($300-800)

Software Stack

Operating System:

  • Ubuntu Server 24.04 LTS (free)
  • Debian 12 (alternative, free)

Container Platform:

  • Docker (free, easier for customer maintenance)
  • OR LXC (free, lower overhead)

MQTT Broker:

  • Eclipse Mosquitto (free, open source)
  • Configuration: Local network only, authenticated users
  • Port: 1883 (or 8883 for TLS)

Time-Series Database:

  • InfluxDB 2.x OSS (free, open source)
  • Alternative: TimescaleDB (PostgreSQL extension, free)
  • Retention: 30-90 days default

Visualization:

  • Grafana OSS (free, open source)
  • Dashboards: Production line overview, OEE tracking, downtime analysis
  • Alerts: Email/SMS via SMTP or webhook

PLC Integration:

  • Node-RED (free, visual programming)
    • node-red-contrib-modbus
    • node-red-contrib-opcua
    • node-red-contrib-s7
  • Alternative: Python scripts
    • pymodbus (Modbus TCP/RTU)
    • opcua-client (OPC UA)
    • python-snap7 (Siemens S7)

Network Architecture

Production Floor:
├── PLCs (Allen-Bradley, Siemens, etc.)
│   └── Connected via: Ethernet, Serial, or OPC UA Server
├── Edge Gateway (Raspberry Pi / Industrial PC)
│   ├── Mosquitto MQTT broker
│   ├── InfluxDB time-series database
│   ├── Grafana visualization
│   └── Node-RED PLC integration
└── Local Network
    ├── Operators access via web browser (http://edge-gateway:3000)
    └── Managers access via web browser or mobile

Security:

  • Isolated VLAN (recommended)
  • Firewall rules (only necessary ports)
  • HTTPS/TLS for Grafana (Let's Encrypt)
  • VPN for remote access (WireGuard or OpenVPN)

Protocols Supported

OT Protocols:

  • Modbus TCP/RTU
  • OPC UA
  • Siemens S7 (via Snap7)
  • EtherNet/IP (Allen-Bradley)
  • BACnet (building automation)
  • Profinet (via OPC UA gateway)

IT Protocols:

  • MQTT (primary message bus)
  • HTTP/REST APIs
  • HTTPS for dashboards
  • SMTP for email alerts

Phase 2: Edge Monitoring Platform (GTHost Multi-Tenant)

Infrastructure

GTHost Dedicated Server #1:

Configuration:
├── CPU: 8 cores (Intel Xeon or AMD EPYC)
├── RAM: 32GB
├── Storage: 1TB NVMe SSD
├── Network: 1Gbps unmetered
├── Location: Choose closest to majority of customers
└── Cost: $100-150/month

Operating System:

  • Ubuntu Server 24.04 LTS
  • Automated updates (unattended-upgrades)
  • Fail2ban for security
  • UFW firewall configured

Multi-Tenant Architecture

Container Platform:

  • LXC (Linux Containers)
    • Lightweight vs Docker
    • Better for long-running services
    • Kernel-level isolation
    • Proven from ZeroLagHub experience

Container Template:

Customer Container (LXC):
├── Ubuntu 24.04 minimal
├── Mosquitto MQTT broker (isolated)
├── InfluxDB (isolated database)
├── Grafana (customer-specific dashboards)
├── Node-RED (optional, for advanced workflows)
├── Backup agent (automated daily)
└── Resource limits (CPU, RAM, disk)

Resource Allocation per Customer:

  • CPU: 1-2 cores (burstable)
  • RAM: 2-4GB
  • Disk: 50-100GB
  • Network: Shared 1Gbps

Capacity Planning:

  • 8-10 basic customers per server
  • 5-8 customers if heavy data volume
  • Monitor: CPU, RAM, disk I/O, network

Networking & Security

Network Architecture:

Internet
    ↓
Caddy Reverse Proxy (TLS termination)
    ↓
LXC Bridge (internal network)
    ├── Customer 1 Container (192.168.100.10)
    ├── Customer 2 Container (192.168.100.11)
    ├── Customer 3 Container (192.168.100.12)
    └── ...

Subdomain Structure:

  • customer1.yourdomain.com → Grafana dashboard
  • customer2.yourdomain.com → Grafana dashboard
  • mqtt.yourdomain.com → MQTT broker (port per customer)

Security Features:

  • TLS/SSL via Let's Encrypt (automated)
  • Firewall (UFW) - only necessary ports
  • Fail2ban - brute force protection
  • Container isolation (LXC namespaces)
  • VPN access for edge devices (WireGuard)
  • Backup encryption (GPG)

Data Flow

Customer Site:
    PLC → Node-RED → MQTT (local edge device)
                        ↓
                    (Over VPN or direct connection)
                        ↓
GTHost Server:
    MQTT Broker → InfluxDB → Grafana
                        ↓
                   Alert Engine → Email/SMS

Backup & Disaster Recovery

Backup Strategy:

  • Automated daily backups (3am UTC)
  • Retention: 7 daily, 4 weekly, 12 monthly
  • Storage: GTHost server + offsite (Wasabi/Backblaze B2)
  • Encrypted with GPG
  • Automated restore testing (monthly)

Disaster Recovery:

  • RTO (Recovery Time Objective): 4 hours
  • RPO (Recovery Point Objective): 24 hours
  • Documented restoration procedure
  • Annual DR test

Monitoring & Alerting

Server Monitoring:

  • Prometheus + Grafana (internal)
  • Alerts: CPU >80%, RAM >80%, Disk >85%
  • UptimeRobot (external monitoring)
  • PagerDuty or similar (if needed)

Customer Monitoring:

  • Per-container resource usage
  • MQTT connection status
  • InfluxDB query performance
  • Grafana dashboard access logs

Phase 3: GPU-Powered AI Platform

Infrastructure

GTHost Dedicated Server #2 (AI/Premium Tier):

Configuration:
├── CPU: 16 cores (Intel Xeon or AMD EPYC)
├── RAM: 64GB
├── GPU: NVIDIA Tesla P4 8GB (or T1000 8GB)
├── Storage: 2TB NVMe SSD
├── Network: 1Gbps unmetered
└── Cost: $350-450/month

Why Tesla P4:

  • Optimized for AI inference (not training)
  • 8GB VRAM sufficient for production models
  • Low power consumption (75W)
  • Good performance/cost ratio

AI/ML Stack

ML Frameworks:

  • TensorFlow Lite (optimized inference)
  • PyTorch (model development, optional)
  • ONNX Runtime (cross-framework inference)
  • Scikit-learn (traditional ML)

GPU Acceleration:

  • CUDA 12.x
  • cuDNN (deep learning primitives)
  • TensorRT (inference optimization)

Model Serving:

  • FastAPI (REST API for predictions)
  • Triton Inference Server (optional, for heavy workloads)
  • Redis (result caching)

AI Features Architecture

1. Predictive Maintenance:

Sensor Data → Feature Engineering → Model Inference → Alert
   (MQTT)      (Python script)      (TensorFlow)     (Email/SMS)

Models:

  • Anomaly detection (vibration, temperature patterns)
  • Failure prediction (time-to-failure models)
  • Remaining Useful Life (RUL) estimation

2. Computer Vision Quality Inspection:

Camera → Image Capture → Preprocessing → Model Inference → Classification
 (HTTP)    (Python)       (OpenCV)       (TensorFlow)      (Pass/Fail)

Models:

  • Object detection (YOLOv8, faster RCNN)
  • Defect classification (CNN)
  • OCR (text recognition on parts)

Container Architecture (Phase 3)

Premium Customer Container:

├── Basic monitoring stack (MQTT, InfluxDB, Grafana)
├── ML inference service (FastAPI + TensorFlow)
├── Feature engineering pipeline
├── Model registry (versioned models)
├── Result database (predictions, alerts)
└── GPU access (controlled, per-customer limits)

Resource Allocation (Premium):

  • CPU: 4-8 cores
  • RAM: 16-32GB
  • GPU: Shared (time-sliced or MIG partitioning)
  • Disk: 200-500GB

Model Development Workflow

Development (Offline):

  1. Collect customer data (4-8 weeks)
  2. Feature engineering and labeling
  3. Model training (local GPU or cloud)
  4. Model validation (accuracy, false positives)
  5. Export to ONNX or TensorFlow Lite

Deployment:

  1. Upload model to server
  2. A/B test against baseline
  3. Monitor inference latency and accuracy
  4. Gradual rollout to production
  5. Continuous monitoring

Data Pipeline (AI Features)

Customer PLCs/Cameras
    ↓
Edge Device (optional preprocessing)
    ↓
MQTT → Feature Store (InfluxDB + PostgreSQL)
    ↓
ML Inference Service (GPU-accelerated)
    ↓
Prediction Results → InfluxDB
    ↓
Grafana Dashboard + Alerts

Development & Deployment Tools

Local Development

Workstation Setup:

  • Ubuntu 22.04 or macOS
  • Docker Desktop (for testing containers)
  • VS Code with extensions:
    • Python
    • Docker
    • YAML
    • Grafana dashboards

Testing Environment:

  • Local LXC or Docker setup
  • Simulated PLC data (Node-RED)
  • Small InfluxDB + Grafana instance

CI/CD Pipeline

Source Control:

  • Git (self-hosted Gitea or GitHub)
  • Branches: main, development, customer-specific

Automation (Future):

  • GitHub Actions or Gitea Actions
  • Automated testing on push
  • Deployment scripts (Ansible)

Deployment Process (Manual Initially):

  1. Test in local environment
  2. Deploy to staging container
  3. Validate with test data
  4. Deploy to production
  5. Monitor for issues

Technology Decisions & Rationale

Why LXC over Docker?

Advantages:

  • Lower overhead (runs closer to bare metal)
  • Better for long-running services (MQTT, databases)
  • Simpler networking (bridge vs overlay)
  • Proven from ZeroLagHub experience
  • Less complexity than Kubernetes

Disadvantages:

  • Less popular than Docker (smaller community)
  • Fewer pre-built images
  • Manual setup required

Decision: Use LXC for multi-tenant platform, Docker for customer edge deployments (easier for them to maintain).

Why InfluxDB over Prometheus?

Advantages:

  • Purpose-built for time-series data
  • Better query language (Flux/InfluxQL)
  • Native downsampling and retention policies
  • Better Grafana integration for industrial data
  • Can handle high-frequency data (1-10 second resolution)

Disadvantages:

  • More complex than Prometheus
  • Heavier resource usage

Decision: InfluxDB for customer data, Prometheus for internal monitoring.

Why Grafana over Custom Dashboard?

Advantages:

  • Industry standard
  • Excellent out-of-box visualizations
  • Plugin ecosystem
  • Customer familiarity (many have seen it)
  • Lower development time

Disadvantages:

  • Not as customizable as custom solution
  • Licensing considerations (AGPL for self-hosted)

Decision: Grafana for Phase 1-2, consider custom dashboard in Phase 3 if needed.

Why MQTT over HTTP?

Advantages:

  • Purpose-built for IoT (lightweight)
  • Pub/sub model (flexible)
  • Quality of Service levels (QoS 0, 1, 2)
  • Better for unreliable networks
  • Lower bandwidth overhead

Disadvantages:

  • One more service to manage
  • Not as universally understood as HTTP

Decision: MQTT for OT data collection, HTTP/REST for management APIs.

Scaling Plan

Server Capacity Thresholds

Add Server #2 When:

  • Server #1 >70% CPU average
  • OR >80% RAM average
  • OR >10 customers on Server #1

Add Server #3 When:

  • Combined >70% capacity
  • OR >20 total customers
  • OR geographic distribution (West Coast + East Coast servers)

Database Scaling

InfluxDB Scaling:

  • Start: Single node per customer container
  • Scale: Consider InfluxDB clustering (Enterprise) if needed
  • Alternative: TimescaleDB for SQL-familiar customers

Backup Scaling:

  • Start: Daily backups to local disk
  • Scale: Offsite backup to object storage (S3-compatible)
  • Future: Real-time replication to hot standby

Security Best Practices

Server Hardening

  • Disable root login (SSH key only)
  • Fail2ban configured
  • UFW firewall (only necessary ports)
  • Automated security updates
  • Regular security audits (quarterly)

Application Security

  • TLS/SSL everywhere (Let's Encrypt)
  • Strong passwords (generated, stored in 1Password)
  • API keys rotated (quarterly)
  • Container isolation verified
  • Database encryption at rest

Compliance Considerations

  • GDPR (if EU customers): Data residency, right to deletion
  • HIPAA (if medical devices): BAA required, encryption
  • ISO 27001 (future): Information security management

Tools & Subscriptions

Required (Paid)

Tool Purpose Cost/Month
GTHost Server #1 Infrastructure $100-150
Domain + DNS yourdomain.com $1-2
Email (G Suite or similar) Professional email $6-12

Total: $107-164/month

Optional (Free/Paid)

Tool Purpose Cost/Month
1Password Password management $0 (personal)
Wasabi Offsite backups $6/TB
UptimeRobot External monitoring $0 (free tier)
Stripe Payment processing 2.9% + $0.30
Twilio SMS alerts Pay-as-you-go

Documentation Strategy

Internal Documentation

  • Runbooks (how to deploy, backup, restore)
  • Architecture diagrams (network, data flow)
  • Troubleshooting guides
  • Security incident response plan

Customer Documentation

  • User guide (how to access dashboards)
  • FAQ (common questions)
  • Alert configuration guide
  • Troubleshooting (basic)

Format: Markdown in Git repository (easy to version, search)

Last Updated: December 2025