diff --git a/SCRATCH/handover-mar29-2026.md b/SCRATCH/handover-mar29-2026.md deleted file mode 100644 index 76ce6e3..0000000 --- a/SCRATCH/handover-mar29-2026.md +++ /dev/null @@ -1,224 +0,0 @@ -# Handover — March 29, 2026 - -## Context -This is a fresh chat handover. Read this file first, then read: -- `OPEN_THREADS.md` — current platform status and pre-launch checklist -- `PROJECT_CONTEXT.md` — infrastructure, stack, naming conventions -- `SCRATCH/migration-new-host.md` — migration checklist (in progress) -- `SCRATCH/pricing-structure.md` — pricing decisions - ---- - -## What's Happening Right Now - -A new dedicated server has been purchased from GTHost Detroit and is active. -The platform is being migrated from the old Denver server to the new Detroit server. -Migration is in progress as of Mar 29 — OPNsense routers and DNS are up on new host. - -**New server specs:** -- Supermicro 2029TP-HTR -- Intel Xeon Gold 6152 — 22c/44t, 2.1-3.7GHz -- 192GB DDR4 -- 2x1.92TB SSD -- Proxmox 9 — already installed by GTHost -- $99/month Detroit (vs old $103/month Denver) - -**Old server (still running, do not cancel yet):** -- Denver, Silver 4116 12c/24t, 192GB, 2x1.92TB SSD, $103/month -- Cancel AFTER migration is confirmed working - ---- - -## Migration Approach - -**NOT a PBS restore of everything** — deliberate fresh rebuild: -- Fresh LXC/VM installs for all services -- Copy project folders for API and portal (not restore) -- Copy jars/plugins for Velocity -- Export/import OPNsense config (not VM restore) -- rsync artifacts content -- Only restore PBS where config is too complex to rebuild - -**Why fresh rebuild:** -- Eliminates accumulated cruft (Tailscale orphans, stale configs, old packages) -- Proxmox 9 native LXC templates from the start -- Clean cgroup v2 resource management -- Only bring over what is intentionally chosen - ---- - -## Standard LXC/VM Specs (new host) - -**All LXC containers use Ubuntu 24.04** — standardized across the board. - -| Service type | CPU | RAM | Disk | -|---|---|---|---| -| Proxy (Caddy) | 1 core | 512MB | 8GB | -| DNS (Technitium) | 1 core | 512MB | 8GB | -| Velocity | 2 cores | 2GB | 16GB | -| Monitor | 2 cores | 2GB | 16GB | -| Artifacts | 1 core | 512MB | 32GB+ | -| API/Portal VMs | 2 cores | 4GB | 32GB | -| OPNsense VMs | 4 cores | 8GB | 64GB | - ---- - -## PBS Status (CRITICAL — Do This First) - -- Old backups only go to November 2025 — disk was full -- Disk has been resized to 1.1TB -- ✅ PBS datastore recreated fresh Mar 29 -- ✅ Fresh backup of all VMs/containers completed Mar 29 ~08:34 - ---- - -## Current VM/LXC Inventory (old host) - -### DO MIGRATE — New ID assignments: - -| Old ID | Name | New ID | Type | How | -|--------|-------|--------|------|-----| -| 105 | zlh-router (core OPNsense) | 9001 | VM | Fresh 26.1 + config import | -| 1006 | zlh-zpack-router (game/dev OPNsense) | 9002 | VM | Fresh 26.1 + config import | -| 1001 | zlh-dns | 9010 | LXC | Fresh + Technitium export/import | -| 1002 | zlh-proxy (Caddy core) | 9011 | LXC | Fresh + config | -| 1004 | zlh-zpack-proxy (Caddy game/dev) | 9012 | LXC | Fresh + config | -| 9000 | zlh-connect (Twingate) | 9013 | LXC | Fresh install | -| 1003 | zlh-artifacts (Caddy file server) | 9014 | LXC | Fresh Caddy + rsync content | -| 300 | zlh-velocity | 9015 | LXC | Fresh + copy jar + plugin | -| 104 | zlh-monitor (Prometheus/Grafana) | 9016 | LXC | Fresh install | -| 2001 | zlh-back (PBS) | 9017 | VM | Fresh + PBS config | -| 1005 | zpac-api | 9020 | VM | Fresh Ubuntu VM + copy project folder | -| 1100 | zpac-portal | 9021 | VM | Fresh Ubuntu VM + copy project folder | -| 4000 | aimeesites | 9030 | LXC | Migrate | - -### DO NOT MIGRATE (legacy/unused): -- 100 (zlh-panel) — old Pterodactyl -- 101 (zlh-wings) — old Pterodactyl -- 102 (zlh-portal) — old Pterodactyl -- 103 (zlh-api) — old Pterodactyl -- 1000 (zlh-router) — original Pterodactyl router -- 810/890 (zlh-base templates) -- 5000 (pup) — replaced by Twingate -- 1007 (zlh-bastion) — on hold -- 2000 (zlh-ctl) — no Ansible in stack - -### Dev/test containers (not production, can recreate if needed): -- 6050 (zpack-dev-velocity) — used to develop Velocity plugin -- 6051 (zpack-agent-dev) — used to develop agent - -### Active game/dev containers (will be reprovisioned by platform): -- 5117 (mc-neoforge-5117) -- 5119 (mc-forge-5119) -- 5120 (mc-fabric-5120) -- 6071 (dev-6071) - ---- - -## New Host ID Scheme - -| Range | Purpose | -|-------|---------| -| 9000s | Core infrastructure (routers, DNS, proxy, monitoring, PBS, API, portal) | -| 5000+ | Game server containers (provisioned by platform) | -| 6000+ | Dev containers (provisioned by platform) | - ---- - -## Key Service Notes - -**zlh-artifacts (9014):** -- Runs Caddy as a file server -- Hosts all runtime binaries (Node, Python, Go, Java, .NET) -- Hosts Minecraft server jars -- Hosts code-server binary -- API pulls from it during provisioning — CRITICAL SERVICE -- Migration: fresh Caddy install + rsync entire content tree from old server - -**OPNsense routers:** -- Two routers: core (105→9001) and zpack/game/dev (1006→9002) -- Installing OPNsense 26.1 fresh (upgrade from 25.7.10) -- Import 25.7.10 config — upgrade path is supported -- **Install os-isc-dhcp plugin BEFORE importing config** — ISC DHCP no longer built-in in 26.1 -- Firewall rules migration assistant available but not urgent — do after everything is stable -- Interface reassignment after config import may be needed -- GTHost MAC binding required for WAN public IPs — check GTHost panel for registered MACs - -**zpac-api / zpac-portal:** -- Both kept as VMs (not LXC) — active development, always been VMs -- Node 22.21.0 (already current LTS — no upgrade needed) -- Next.js 16.1.1 (current — no upgrade needed) -- Copy project folder + npm install on new VM - -**zlh-velocity:** -- Velocity 3.5.0-SNAPSHOT (latest) -- Has a custom dynamic game server routing plugin built by the team -- Fresh LXC + copy Velocity jar + copy plugin jar -- Check current version on old server before migrating - -**zlh-proxy / zlh-zpack-proxy:** -- Both now use Caddy (not Traefik) — switched during migration -- Original Caddyfile saved at SCRATCH/caddy/Caddyfile-old -- New Caddyfile for 9011 at SCRATCH/caddy/Caddyfile-new (update IPs after API/portal are up) - ---- - -## Architecture Reminders - -- Everything internal to Proxmox except Velocity TCP port (Minecraft players) -- Portal is only public-facing web surface -- API runs on private IP — portal calls it internally -- Minecraft player traffic proxied through Velocity VM -- Twingate for admin remote access -- WireGuard on OPNsense as fallback admin access -- Agent is sole filesystem authority — API never duplicates filesystem logic -- Portal never calls agents directly — all traffic through API -- Upload transport: raw http.request piping only, never fetch() -- VMs 100, 101, 102, 103, 1000 are legacy — do not touch - ---- - -## Network Notes - -- All services run on private IPs internally -- No hardcoded public IP dependencies in application code -- DNS cutover (Cloudflare A/SRV records for Velocity) is the only external change needed at cutover -- New host has different physical NIC names — ens6f0/ens6f1 (vs eno1/eno2 on old host) -- vmbr0–vmbr6 configured and working on new host -- GTHost MAC binding: OPNsense WAN virtual NIC MAC must match what GTHost has registered for the public IPs - ---- - -## Proxmox API Setup (for migration scripting) - -When ready to script VM/LXC creation: -1. Create user `zlh-automation@pve` in Proxmox -2. Create role with VM.Allocate, VM.Config.*, Datastore.AllocateSpace, Sys.Console -3. Assign role to user at path `/` -4. Create API token — save it, only shown once -5. Use `Authorization: PVEAPIToken=zlh-automation@pve!migration=` header - ---- - -## Platform Status (from OPEN_THREADS.md) - -**Pre-launch blockers:** -1. Billing / Stripe integration -2. Game server world backup / restore -3. User onboarding flow -4. Password reset flow — verify wired up -5. Usage limits / quota enforcement -6. Email notifications -7. Upload testing -8. OPNsense audit - -**Portal copy — DONE:** -Landing, features, FAQ, about, pricing all rewritten and updated by Codex. -Pricing: Vanilla $8/mo, Modded $20/mo, Heavy $35/mo — Minecraft only launch. - ---- - -## Source of Truth - -`git.zerolaghub.com/jester/zlh-grind` — always read this before making decisions. -`git.zerolaghub.com/jester/knowledge-base` — older docs, mostly stale (Dec 2025), leave as historical reference.