# Monitoring — Current State

## Launch readiness

**Status: PARTIAL FAIL for launch monitoring readiness.**

Core monitoring services are running. Game/dev dynamic discovery empty-state is now clean and stale `game-dev-alloy` VMIDs `5205` and `6084` have disappeared from Prometheus after a successful empty sync.

Launch-debug readiness is still blocked by public monitoring endpoint exposure, remaining stale/static down targets, dashboards not actually installed in Grafana, missing API health scrape, incomplete lifecycle telemetry, and missing centralized logs.

## Recently verified progress

Container discovery is clean:

- `/sd/exporters -> []`
- `/monitoring/game-dev -> containers: []`
- monitor-side `ALLOW_EMPTY="true"` added in `/etc/zlh-monitor/game-dev-discovery.env`
- one-shot sync succeeded: wrote `0 targets` to `/etc/zlh-monitor/prometheus/file_sd/game-dev-alloy.json`
- Prometheus refreshed and stale `game-dev-alloy` targets `5205` and `6084` are gone

This resolves the previous dynamic discovery empty-state/stale `game-dev-alloy` blocker.

## Running services observed from prior audit

- `prometheus.service` — active/running
- `grafana-server.service` — active/running
- `alloy.service` — active/running
- `prometheus-node-exporter.service` — active/running

`zlh-monitor-game-dev-discovery.service` was previously failed, but the latest manual sync succeeded after enabling `ALLOW_EMPTY="true"`. Verify timer/service health after the next scheduled run.

## Exposure state

Observed listening ports from prior audit:

- `*:9090` Prometheus
- `*:3000` Grafana
- `*:9100` node_exporter
- `127.0.0.1:12345` Alloy

Host firewall posture from prior audit:

- UFW inactive
- nftables input/forward/output policy accept

This remains a launch blocker because Prometheus, Grafana, and node_exporter are reachable on all interfaces without host firewall restriction.

## Prometheus state

- Config path: `/etc/prometheus/prometheus.yml`
- Global scrape interval: `15s`
- Prometheus self-scrape: `5s`
- Dynamic discovery refresh: `30s`
- `/sd/exporters` is configured with bearer auth and works with token
- unauthenticated `/sd/exporters` and `/monitoring/game-dev` return `401 unauthorized`

Token risk: the bearer token was previously observed stored directly in Prometheus YAML, backup configs, and `/etc/zlh-monitor/game-dev-discovery.env`. Tokens were not observed leaking through labels or scrape URLs in the audit, but storage should still be tightened.

## Targets/discovery state

Resolved:

- `game-dev-alloy` VMID `5205` is gone
- `game-dev-alloy` VMID `6084` is gone
- empty discovery writes a valid empty file_sd file instead of failing

Remaining problem targets to remove/fix:

- `alloy-dev-6076`, `10.100.0.14:12345`, hardcoded/down
- `cadvisor`, `10.60.0.11:8081`, down/no route
- `zerolaghub-nodes`, `10.60.0.19:9100`, down/connection refused

## API monitoring state

API host metrics exist via `integrations/unix`:

- `vmid=9020`
- `role=API`
- `instance=zlh-api`

Observed API app health endpoint `/health` previously returned `404 Cannot GET /health`; no API health scrape was found.

No centralized API logs were found on the monitoring host.

## Agent/container monitoring state

Current dynamic game/dev discovery is empty and clean.

Expected next validation is lifecycle smoke testing:

- create a game and/or dev container
- confirm target appears
- delete it
- confirm target disappears now that empty discovery works

Labels previously observed include:

- `vmid`
- `ctype` or `container_type`
- `game`
- `variant`
- `hostname` / `name`
- `job`
- `instance`

Missing or incomplete for lifecycle debugging:

- `ready` / `connectable`
- `operationInProgress`
- `operationType`
- backup/restore state
- code-server state

## Grafana state

Dashboard JSON files exist under `/etc/zlh-monitor/grafana/dashboards`:

- `core-routers.json`
- `core-service-detail.json`
- `core-services-overview.json`
- `dev-containers.json`
- `game-containers.json`

However, Grafana DB previously showed no dashboards installed. Datasource exists:

- `prometheus -> http://localhost:9090`

Dashboard JSON must be provisioned/visible in Grafana before launch monitoring is considered ready.

## Logs state

No Loki, Promtail, or log-ingesting Alloy pipeline was found in the prior audit.

Only journald/local logs for monitoring components were available on the monitoring host. This remains a launch-debug gap for API, Agent, provisioning, backup/restore, delete/teardown, Velocity/DNS/edge actions, and discovery sync.

## Security state

Working controls from prior audit:

- Grafana anonymous access is disabled; unauthenticated `/api/search` returned `401`
- discovery endpoints require token; unauthenticated `/sd/exporters` and `/monitoring/game-dev` returned `401`
- Prometheus labels/scrape URLs checked did not expose the bearer token

Launch-blocking risk:

- Prometheus, Grafana, and node_exporter bind publicly on the host network
- host firewall policy does not restrict access