jester/zlh-grind
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity
30122ea27e
zlh-grind/SCRATCH/handover-mar29-2026.md

7.5 KiB
Raw Blame History

Handover — March 29, 2026

Context

This is a fresh chat handover. Read this file first, then read:

  • OPEN_THREADS.md — current platform status and pre-launch checklist
  • PROJECT_CONTEXT.md — infrastructure, stack, naming conventions
  • SCRATCH/migration-new-host.md — migration checklist (in progress)
  • SCRATCH/pricing-structure.md — pricing decisions

What's Happening Right Now

A new dedicated server has been purchased from GTHost Detroit and is active. The platform is being migrated from the old Denver server to the new Detroit server. The user is preparing to begin the migration today.

New server specs:

  • Supermicro 2029TP-HTR
  • Intel Xeon Gold 6152 — 22c/44t, 2.1-3.7GHz
  • 192GB DDR4
  • 2x1.92TB SSD
  • Proxmox 9 — already installed by GTHost
  • $99/month Detroit (vs old $103/month Denver)

Old server (still running, do not cancel yet):

  • Denver, Silver 4116 12c/24t, 192GB, 2x1.92TB SSD, $103/month
  • Cancel AFTER migration is confirmed working

Migration Approach

NOT a PBS restore of everything — deliberate fresh rebuild:

  • Fresh LXC/VM installs for all services
  • Copy project folders for API and portal (not restore)
  • Copy jars/plugins for Velocity
  • Export/import OPNsense config (not VM restore)
  • rsync artifacts content
  • Only restore PBS where config is too complex to rebuild

Why fresh rebuild:

  • Eliminates accumulated cruft (Tailscale orphans, stale configs, old packages)
  • Proxmox 9 native LXC templates from the start
  • Clean cgroup v2 resource management
  • Only bring over what is intentionally chosen

PBS Status (CRITICAL — Do This First)

  • Old backups only go to November 2025 — disk was full
  • Disk has been resized to 1.1TB
  • Plan: delete old datastore, recreate fresh, run full backup of everything
  • DO NOT start migration until fresh backups are confirmed green

Fresh backup priority order:

  1. OPNsense core router (105)
  2. OPNsense zpack router (1006)
  3. zlh-dns (1001)
  4. zpack-api (1005)
  5. zpack-portal (1100)
  6. zlh-artifacts (1003)
  7. zlh-velocity (300)
  8. Everything else

Current VM/LXC Inventory (old host)

DO MIGRATE — New ID assignments:

Old ID Name New ID Type How
105 zlh-router (core OPNsense) 9001 VM Fresh 26.1 + config import
1006 zlh-zpack-router (game/dev OPNsense) 9002 VM Fresh 26.1 + config import
1001 zlh-dns 9010 LXC Fresh + Technitium export/import
1002 zlh-proxy (Traefik core) 9011 LXC Fresh + config
1004 zlh-zpack-proxy (Traefik game/dev) 9012 LXC Fresh + config
9000 zlh-connect (Twingate) 9013 LXC Fresh install
1003 zlh-artifacts (Caddy file server) 9014 LXC Fresh Caddy + rsync content
300 zlh-velocity 9015 LXC Fresh + copy jar + plugin
104 zlh-monitor (Prometheus/Grafana) 9016 LXC Fresh install
2001 zlh-back (PBS) 9017 VM Fresh + PBS config
1005 zpack-api 9020 VM Fresh Ubuntu VM + copy project folder
1100 zpack-portal 9021 VM Fresh Ubuntu VM + copy project folder
4000 aimeesites 9030 LXC Migrate

DO NOT MIGRATE (legacy/unused):

  • 100 (zlh-panel) — old Pterodactyl
  • 101 (zlh-wings) — old Pterodactyl
  • 102 (zlh-portal) — old Pterodactyl
  • 103 (zlh-api) — old Pterodactyl
  • 1000 (zlh-router) — original Pterodactyl router
  • 810/890 (zlh-base templates)
  • 5000 (pup) — replaced by Twingate
  • 1007 (zlh-bastion) — on hold
  • 2000 (zlh-ctl) — no Ansible in stack

Dev/test containers (not production, can recreate if needed):

  • 6050 (zpack-dev-velocity) — used to develop Velocity plugin
  • 6051 (zpack-agent-dev) — used to develop agent

Active game/dev containers (will be reprovisioned by platform):

  • 5117 (mc-neoforge-5117)
  • 5119 (mc-forge-5119)
  • 5120 (mc-fabric-5120)
  • 6071 (dev-6071)

New Host ID Scheme

Range Purpose
9000s Core infrastructure (routers, DNS, proxy, monitoring, PBS, API, portal)
5000+ Game server containers (provisioned by platform)
6000+ Dev containers (provisioned by platform)

Key Service Notes

zlh-artifacts (9014):

  • Runs Caddy as a file server
  • Hosts all runtime binaries (Node, Python, Go, Java, .NET)
  • Hosts Minecraft server jars
  • Hosts code-server binary
  • API pulls from it during provisioning — CRITICAL SERVICE
  • Migration: fresh Caddy install + rsync entire content tree from old server

OPNsense routers:

  • Two routers: core (105→9001) and zpack/game/dev (1006→9002)
  • Installing OPNsense 26.1 fresh (upgrade from 25.7.10)
  • Import 25.7.10 config — upgrade path is supported
  • DHCP plugin auto-installs during upgrade
  • Firewall rules migration assistant available but not urgent
  • Interface reassignment after config import may be needed
  • GTHost MAC binding required for WAN public IPs — check GTHost panel for registered MACs

zpack-api / zpack-portal:

  • Both kept as VMs (not LXC) — active development, always been VMs
  • Node 22.21.0 (already current LTS — no upgrade needed)
  • Next.js 16.1.1 (current — no upgrade needed)
  • Copy project folder + npm install on new VM

zlh-velocity:

  • Velocity 3.5.0-SNAPSHOT (latest)
  • Has a custom dynamic game server routing plugin built by the team
  • Fresh LXC + copy Velocity jar + copy plugin jar
  • Check current version on old server before migrating

Architecture Reminders

  • Everything internal to Proxmox except Velocity TCP port (Minecraft players)
  • Portal is only public-facing web surface
  • API runs on private IP — portal calls it internally
  • Minecraft player traffic proxied through Velocity VM
  • Twingate for admin remote access
  • WireGuard on OPNsense as fallback admin access
  • Agent is sole filesystem authority — API never duplicates filesystem logic
  • Portal never calls agents directly — all traffic through API
  • Upload transport: raw http.request piping only, never fetch()
  • VMs 100, 101, 102, 103, 1000 are legacy — do not touch

Network Notes

  • All services run on private IPs internally
  • No hardcoded public IP dependencies in application code
  • DNS cutover (Cloudflare A/SRV records for Velocity) is the only external change needed at cutover
  • New host will have different physical NIC names — check with ip link show before configuring bridges
  • Mirror the vmbr layout from current host exactly
  • GTHost MAC binding: OPNsense WAN virtual NIC MAC must match what GTHost has registered for the public IPs

Proxmox API Setup (for migration scripting)

When ready to script VM/LXC creation:

  1. Create user zlh-automation@pve in Proxmox
  2. Create role with VM.Allocate, VM.Config.*, Datastore.AllocateSpace, Sys.Console
  3. Assign role to user at path /
  4. Create API token — save it, only shown once
  5. Use Authorization: PVEAPIToken=zlh-automation@pve!migration=<secret> header

Platform Status (from OPEN_THREADS.md)

Pre-launch blockers:

  1. Billing / Stripe integration
  2. Game server world backup / restore
  3. User onboarding flow
  4. Password reset flow — verify wired up
  5. Usage limits / quota enforcement
  6. Email notifications
  7. Upload testing
  8. OPNsense audit

Portal copy — DONE: Landing, features, FAQ, about, pricing all rewritten and updated by Codex. Pricing: Vanilla $8/mo, Modded $20/mo, Heavy $35/mo — Minecraft only launch.

Source of Truth

git.zerolaghub.com/jester/zlh-grind — always read this before making decisions. git.zerolaghub.com/jester/knowledge-base — older docs, mostly stale (Dec 2025), leave as historical reference.