# Open Threads — zlh-grind

This file tracks active but unfinished work.

Keep it short.

---

## Agent (zlh-agent)

### Dev Runtime System
Completed:
- catalog validation implemented
- runtime installs artifact-backed
- install guard implemented
- all installs now fetch from artifact server

Outstanding:
- runtime install verification improvements
- catalog hash validation
- runtime removal / upgrade handling
- runtime update process for dev containers

### Dev Environment
Completed:
- dev user creation
- workspace root `/home/dev/workspace`
- console runs as dev user
- `HOME`, `USER`, `LOGNAME`, `TERM` env vars set correctly

Outstanding:
- PATH normalization
- shell profile consistency
- runtime PATH injection

### Code Server Addon
Status: Installed, running, browser-verified end-to-end.

Outstanding:
- code-server memory baseline
- decide whether default dev RAM should increase or become tier-based
- k6 IDE session load test

### Game Server Supervision
Completed:
- crash recovery with backoff
- crash observability and classification

### Fabric Readiness Gating
Status: startup rehydrate path fixed in plugin and happy-path validated live.

Still outstanding:
- negative-path validation: verify a `ready=false` backend is skipped on Velocity startup rehydrate
- confirm whether any remaining agent-side registration path can surface a backend before readiness probe success
- see `SCRATCH/session-stabilization-fabric-findings.md`

### Agent Future Work
1. Structured logging (slog) for Loki
2. Dev container `provisioningComplete` state in `/status`
3. Graceful shutdown verification
4. Process reattachment on agent restart
5. SSH server install in dev container provisioning pipeline

---

## Dev IDE Access

### Browser IDE
Status: fully working, browser-verified, zero-install.

Remaining:
- confirm "Open IDE" button in portal uses hosted URL in production path
- reduce legacy `/__ide/:id` compatibility paths once portal button confirmed
- simplify and harden `devProxy`

### Local Dev Access — SSH via CF Tunnel
Current state:
- tunnel created and connected to bastion VM
- Zero Trust free plan active
- SSH hostname mapping not yet configured
- bastion SSH proxy jump config not yet done
- dev container SSH server not yet verified
- portal SSH config snippet not yet built

---

## API (zpack-api)

Completed:
- dev provisioning payload
- runtime/version fields
- enable_code_server flag
- status endpoint
- IDE token generation + hosted URL
- bootstrap IDE route
- live tunnel proxy
- host-based routing for hosted IDE
- hosted flow browser-verified end-to-end
- backend lifecycle hardened
- duplicate server creation fixed
- console routing corrected
- control plane switched to IP-based service communication
- Velocity rehydration uses DB + Redis instead of Proxmox live state

Outstanding:
- Billing / Stripe integration
- password reset flow verification
- usage limits / quota enforcement
- simplify and harden host-native `devProxy`
- dev runtime catalog endpoint for portal
- Headscale auth key generation
- service discovery migration for remaining hot-path `internal.zlh` references

---

## Portal (zpack-portal)

Completed:
- dev runtime dropdown
- dotnet runtime support
- enable code-server checkbox
- dev file browser support
- site copy rewrite
- pricing page updated

Outstanding:
- confirm "Open IDE" button fully uses hosted URL flow
- SSH config snippet for power users
- user onboarding flow
- email notifications
- remove `testdaemon` binary from repo root

---

## Game Servers

Outstanding:
- game server world backup / restore
- game server subdomain / player connection method verification

---

## Velocity / ZpackVelocityBridge

Completed:
- startup rehydrate now requires `ready == true`
- stale default `zpack-api.internal.zlh` fallback removed
- explicit `ZPACK_REHYDRATE_ENDPOINT` env wiring validated
- happy-path player routing validated live after restart

Outstanding:
- ZpackCommands not registered (`/zpack status`, `/zpack list`, `/zpack reload`)
- negative-path readiness validation still pending for a `ready=false` backend
- see `SCRATCH/velocity-plugin.md`

---

## Pre-Launch Checklist

Outstanding before launch:
- Billing / Stripe integration
- game server world backup / restore
- user onboarding flow
- password reset flow verification
- usage limits / quota enforcement
- game server subdomain verification
- email notifications
- upload testing
- billing endpoints
- stress testing: k6 IDE + Minecraft bot + code-server memory baseline
- OPNsense audit
- Fabric readiness gating full validation
- service discovery migration
- provisioning validation
- remove `testdaemon` from `zpack-portal`

---

## Platform

Future work:
- CF Tunnel SSH completion
- artifact version promotion
- runtime rollback support
- Cloudflare R2 for large artifact/mod file delivery
- admin panel
- referral / dev pipeline reward system
- uptime history
- revisit DDoS mitigation later if needed

---

## Closed Threads

- PTY console (dev + game)
- mod lifecycle
- upload pipeline
- runtime artifact installs
- dev container filesystem model
- code-server artifact fix
- API status endpoint for frontend agent-state consumption
- game server crash recovery with backoff
- crash observability
- code-server lifecycle endpoints
- code-server process detection
- dev IDE proxy
- hosted wildcard Traefik → API → container IDE flow
- per-container dev IDE edge publish/unpublish removed from API
- wildcard TLS cert `*.zerolaghub.dev`
- browser IDE fully loading at `dev-<vmid>.zerolaghub.dev`
- CF Tunnel created and connected to bastion VM
- portal copy rewrite
- DDoS investigation
- hosting provider decision: GTHost Detroit
- migration to Detroit complete
- system stabilization after migration
- IP-based control plane
- Velocity startup rehydrate fixed and validated on happy path