docs: replace with runtime-aligned upload architecture (direct writes, metadata provenance, API streaming)
This commit is contained in:
parent
2c8a9de41c
commit
7d7e2378f5
@ -1,84 +1,181 @@
|
||||
# ZeroLagHub – File System & File Browser Strategy
|
||||
# Filesystem and File Browser Architecture
|
||||
|
||||
**Date:** 2026-02-28
|
||||
**Status:** Planning — not yet implemented
|
||||
**Next Action:** Stub file endpoints in existing agent, prove end-to-end, extract later
|
||||
**Updated:** 2026-03-01
|
||||
**Status:** Active — reflects implemented upload model
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
## Overview
|
||||
|
||||
This document captures the architectural decisions and UX direction for file system access in ZeroLagHub game and dev containers. It is a planning document, not an implementation spec.
|
||||
ZeroLagHub uses a strict, runtime-aligned filesystem model for game containers.
|
||||
The file browser reflects the real runtime layout and does not use staging directories, shadow folders, or symlink-based indirection.
|
||||
|
||||
Uploads are written directly into the runtime directory tree with strict policy enforcement and provenance tracking.
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
## Runtime Root
|
||||
|
||||
### Game Server Owners (non-technical)
|
||||
- Edit `server.properties` and config files
|
||||
- Upload mod `.jar` files to `/mods`
|
||||
- Restore deleted mods from `/mods-removed`
|
||||
- Download log files for debugging
|
||||
|
||||
### Developers / BYOS (technical)
|
||||
- Full shell access
|
||||
- File transfer (upload/download)
|
||||
- SFTP access to dev container filesystem
|
||||
|
||||
These are two distinct personas with different needs and different solutions.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Decision
|
||||
|
||||
### Game Server File Access
|
||||
Handled via **agent file endpoints + portal file browser UI**.
|
||||
|
||||
No SSH required. The agent exposes REST file management endpoints. The API proxies them behind auth + ownership enforcement (same pattern as all other agent endpoints). The portal renders a file browser panel.
|
||||
|
||||
### Dev Container File Access
|
||||
Handled via **WebSSH2 + SFTP**, proxied through the API.
|
||||
|
||||
Developers get a real SSH2 session with SFTP channel. No direct container access from the browser — API proxy maintains the security boundary (DEC-008).
|
||||
|
||||
---
|
||||
|
||||
## Agent File Endpoints (Planned)
|
||||
Each game container has a single runtime root:
|
||||
|
||||
```
|
||||
GET /game/files?path= — list directory
|
||||
GET /game/files/download?path= — download file
|
||||
POST /game/files/upload?path= — upload file
|
||||
DELETE /game/files?path= — delete file
|
||||
PATCH /game/files — rename file
|
||||
/opt/zlh/minecraft/<runtime>/<world>/
|
||||
```
|
||||
|
||||
Mirrored in API under:
|
||||
```
|
||||
/api/game/servers/:id/files/*
|
||||
```
|
||||
All file operations are resolved relative to this root.
|
||||
|
||||
### Security Requirements
|
||||
- Hard-rooted to `serverRoot` — no path traversal outside container root
|
||||
- HTTPS only
|
||||
- Auth + ownership enforced at API layer
|
||||
- Upload size limits enforced
|
||||
- No execution of uploaded files
|
||||
The agent is the only component allowed to perform filesystem mutations.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Sequencing
|
||||
## Hidden Internal Files
|
||||
|
||||
**Do not split the agent yet.**
|
||||
The following files are internal and never exposed through the file API:
|
||||
|
||||
Land file endpoints in the existing agent first. Prove the feature end-to-end. Once the surface area is clear and stable, extract if needed.
|
||||
- `.zlh-shadow`
|
||||
- `.zlh_metadata.json`
|
||||
|
||||
SFTP is the exception — if SFTP access for dev containers is implemented, it warrants its own separate process from day one due to SSH server complexity. It does not belong in the main game agent.
|
||||
These are filtered centrally inside the agent `internal/files` package, not in route handlers.
|
||||
|
||||
### Phased approach
|
||||
1. File endpoints in existing agent (stub → prove → harden)
|
||||
2. Portal file browser UI wired to API proxy
|
||||
3. SFTP as separate agent process for dev containers (separate binary, separate port, separate systemd unit)
|
||||
---
|
||||
|
||||
## Upload Model
|
||||
|
||||
### No Staging
|
||||
|
||||
Uploads are written directly into their final runtime location.
|
||||
|
||||
There is:
|
||||
- ❌ No temporary upload folder
|
||||
- ❌ No symlink-based deployment
|
||||
- ❌ No background relocation job
|
||||
|
||||
Uploads are atomic:
|
||||
|
||||
1. Write to temp file in same directory
|
||||
2. `os.Rename()` to final path
|
||||
|
||||
---
|
||||
|
||||
## Upload Allowlist
|
||||
|
||||
Uploads are restricted to:
|
||||
|
||||
| Type | Path | Extension |
|
||||
|------|------|-----------|
|
||||
| Mods | `mods/<file>.jar` | `.jar` |
|
||||
| Datapacks | `world/datapacks/<file>.zip` | `.zip` |
|
||||
|
||||
All other upload paths are rejected.
|
||||
|
||||
Parent directory must already exist. No directory creation is performed.
|
||||
|
||||
---
|
||||
|
||||
## Security Constraints
|
||||
|
||||
Upload is rejected if:
|
||||
|
||||
- Path traversal (`../`)
|
||||
- Absolute path
|
||||
- Control characters
|
||||
- Target exists as directory
|
||||
- Target is a symlink
|
||||
- Target resolves outside runtime root
|
||||
- Path not allowlisted
|
||||
- Parent directory missing
|
||||
|
||||
---
|
||||
|
||||
## Provenance Metadata
|
||||
|
||||
Uploads create or update `.zlh_metadata.json`, stored at runtime root.
|
||||
|
||||
**Example:**
|
||||
|
||||
```json
|
||||
{
|
||||
"mods/sodium.jar": {
|
||||
"source": "user",
|
||||
"uploaded_at": "2026-03-01T22:37:01Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### stat Response Extension
|
||||
|
||||
File stat now includes:
|
||||
|
||||
```
|
||||
"source": "user" | null
|
||||
```
|
||||
|
||||
Only `"user"` is currently written by the agent. No curated inference is performed.
|
||||
|
||||
---
|
||||
|
||||
## File Browser Responsibilities
|
||||
|
||||
### Portal (Frontend)
|
||||
- Displays files
|
||||
- Displays `source: "user"` badge
|
||||
- Enforces extension pre-validation
|
||||
- Sends multipart upload
|
||||
|
||||
### API
|
||||
- Auth + ownership enforcement
|
||||
- Streams upload to agent
|
||||
- Does **NOT** inspect file contents
|
||||
- Does **NOT** re-implement upload policy
|
||||
|
||||
### Agent
|
||||
- Enforces filesystem rules
|
||||
- Writes metadata
|
||||
- Performs atomic writes
|
||||
- Returns authoritative status
|
||||
|
||||
---
|
||||
|
||||
## Why No Symlinks
|
||||
|
||||
Symlink-based deployment was rejected because:
|
||||
|
||||
- It complicates mod loading behavior
|
||||
- Breaks compatibility with server-side mod loaders
|
||||
- Introduces unexpected runtime indirection
|
||||
- Makes provenance ambiguous
|
||||
|
||||
Direct writes + metadata is simpler and safer.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Browser → API → Agent → Filesystem
|
||||
```
|
||||
|
||||
The API uses raw `http.request` piping to stream uploads. It does not buffer large files in memory.
|
||||
|
||||
---
|
||||
|
||||
## API Transport Considerations
|
||||
|
||||
The API acts strictly as a streaming proxy.
|
||||
|
||||
Implementation uses raw `http.request` piping:
|
||||
|
||||
```js
|
||||
req.pipe(proxyReq)
|
||||
proxyRes.pipe(res)
|
||||
```
|
||||
|
||||
This avoids:
|
||||
- Fetch streaming incompatibilities
|
||||
- Duplex locking issues
|
||||
- Multipart buffering problems
|
||||
|
||||
Upload timeout for this route should be substantially larger than normal file routes.
|
||||
|
||||
---
|
||||
|
||||
@ -91,16 +188,16 @@ Split-pane panel — directory tree left, file detail/actions right. Slides in a
|
||||
Breadcrumb-based. Flat navigation within each directory rather than deep tree expansion. Click folder → replace view. Not expand-in-place.
|
||||
|
||||
### File Listing
|
||||
Columns: name, size, modified date, type icon. For `.jar` files: status badge (enabled / disabled / removed).
|
||||
Columns: name, size, modified date, type icon, source badge (`user` for user-uploaded files). For `.jar` files: enabled / disabled / removed status.
|
||||
|
||||
### Actions
|
||||
Context menu or per-row three-dot menu. Actions: download, delete, rename. For `.jar` files: enable / disable toggle. Drag-to-upload supported, file picker fallback.
|
||||
|
||||
### In-Browser Editor
|
||||
Plain textarea or Monaco for text files (`server.properties`, `.json`, `.txt`). Binary files get download link only. Not required for launch.
|
||||
Plain textarea or Monaco for text files (`server.properties`, `.json`, `.toml`, `.yml`, `.txt`). Binary files get download link only. Not required for launch.
|
||||
|
||||
### `mods-removed` Surface
|
||||
"Recently removed" section or toggle to show `/mods-removed` alongside active mods. This makes soft delete visible and gives users a restore path without knowing the underlying filesystem layout.
|
||||
"Recently removed" section or toggle to show `/mods-removed` alongside active mods. Makes soft delete visible and gives users a restore path without knowing the underlying filesystem layout.
|
||||
|
||||
### What to Avoid
|
||||
- Deep expand/collapse tree for mod directory (use flat list + filter)
|
||||
@ -115,7 +212,15 @@ Plain textarea or Monaco for text files (`server.properties`, `.json`, `.txt`).
|
||||
|
||||
The agent already runs an HTTP server. Serving static file browser assets from the agent directly keeps per-container footprint minimal. No additional process, no config management, no extra memory overhead.
|
||||
|
||||
Caddy or Nginx per container would make sense if you needed per-container SSL termination or direct browser access without the API proxy. ZLH's architecture routes everything through the API proxy (`zlh-proxy` handles SSL at the edge), so a local web server adds a layer without adding capability.
|
||||
`zlh-proxy` handles SSL at the edge. A local web server per container adds a layer without adding capability.
|
||||
|
||||
---
|
||||
|
||||
## Dev Container File Access
|
||||
|
||||
Handled via **WebSSH2 + SFTP**, proxied through the API.
|
||||
|
||||
Developers get a real SSH2 session with SFTP channel. No direct container access from the browser — API proxy maintains the security boundary (DEC-008). SFTP warrants its own separate process from day one due to SSH server complexity. It does not belong in the main game agent.
|
||||
|
||||
---
|
||||
|
||||
@ -129,14 +234,20 @@ The file agent (when extracted) should follow the same binary resilience pattern
|
||||
- Systemd watchdog flips `current` back to previous on health check failure
|
||||
- No dependency on artifact server for rollback — local fallback only
|
||||
|
||||
This keeps the file service self-healing without operator intervention, consistent with ZLH's overall design goal.
|
||||
---
|
||||
|
||||
## Design Philosophy
|
||||
|
||||
The runtime directory is the source of truth.
|
||||
|
||||
No abstraction layer should diverge from Minecraft's real file structure.
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- `docs/architecture/mod-deployment-safety.md` — mod lifecycle and rollback model
|
||||
- `docs/architecture/mod-deployment-safety.md` — mod lifecycle, upload safety, rollback model
|
||||
- `docs/architecture/dev-to-game-artifact-pipeline.md` — dev container promotion pipeline
|
||||
- `docs/reference/minecraft-file-locations.md` — known files and directories by loader
|
||||
- `OPEN_THREADS.md` — file browser listed as next major feature
|
||||
- `Frontend/TerminalView_Component.md` (knowledge-base) — terminal implementation reference
|
||||
- WebSSH2: `https://github.com/billchurch/webssh2` — SFTP + SSH2 for dev containers
|
||||
|
||||
Loading…
Reference in New Issue
Block a user