Clarify that cloudflared is required on both client and bastion — add install instructions

network/cf-tunnel-ssh.md

@@ -3,8 +3,44 @@
 ## Overview
 
 Cloudflare Tunnel provides SSH access to dev containers for developers using
-local VS Code or terminal. No VPN client required. Uses the same hostname as
+local VS Code or terminal. No VPN required. Uses the same hostname as the
-the browser IDE.
+browser IDE.
+
+---
+
+## How It Works — Two Sides
+
+This requires `cloudflared` on **both** the server (bastion) and the client
+(developer machine). They do completely different things:
+
+**Bastion (server side):**
+`cloudflared` runs as a persistent service maintaining an outbound tunnel to
+Cloudflare's edge. This is already set up.
+
+**Developer machine (client side):**
+`cloudflared` is a lightweight binary used as an SSH ProxyCommand. It
+intercepts the SSH connection and routes it through Cloudflare instead of
+going directly to the IP. Without this, SSH bypasses Cloudflare entirely,
+hits the raw IP on port 22, and gets rejected.
+
+The two sides meet at Cloudflare — no inbound ports needed on either:
+
+```
+Developer machine
+  cloudflared (ProxyCommand) → Cloudflare edge ← cloudflared (tunnel)
+                                                      ↓
+                                               Bastion VM
+                                                      ↓
+                                            Dev container
+```
+
+**Critical:** If SSH connects directly to the IP instead of through
+cloudflared, you will see:
+```
+kex_exchange_identification: Connection closed by remote host
+```
+This means the ProxyCommand is not being invoked — SSH is bypassing
+Cloudflare entirely.
 
 ---
 
@@ -12,7 +48,7 @@ the browser IDE.
 
 ```
 Developer laptop
-  ↓ ssh dev-6070.zerolaghub.dev
+  ↓ ssh dev-6070.zerolaghub.dev (via cloudflared ProxyCommand)
 Cloudflare edge
   ↓ CF Tunnel (persistent, runs on bastion)
 Bastion VM (private IP, no public exposure)
@@ -30,17 +66,53 @@ HTTPS and SSH share the same hostname. Cloudflare routes them separately:
 
 - ✅ CF Tunnel created and connected to bastion VM
 - ✅ Cloudflare Zero Trust free plan active (covers up to 50 users)
-- ⏳ Tunnel configuration not fully complete
+- ⏳ Tunnel SSH hostname mapping not yet configured in Zero Trust dashboard
-- ⏳ SSH hostname mapping not yet configured
+- ⏳ Bastion SSH proxy jump config not yet done
-- ⏳ Bastion SSH proxy jump not yet configured
+- ⏳ Dev container SSH server not yet verified
-- ⏳ Dev containers SSH not yet verified reachable from bastion
 - ⏳ Portal SSH config snippet not yet built
 
 ---
 
 ## Remaining Steps
 
-### 1. Configure tunnel ingress in Cloudflare Zero Trust dashboard
+### 1. Install cloudflared on developer machines
+
+This is required on every developer's machine — not optional.
+
+```bash
+# macOS
+brew install cloudflare/cloudflare/cloudflared
+
+# Linux
+curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 \
+  -o /usr/local/bin/cloudflared && chmod +x /usr/local/bin/cloudflared
+
+# Windows
+# Download from https://github.com/cloudflare/cloudflared/releases
+
+# Termux (Android)
+pkg install cloudflared
+```
+
+### 2. Developer SSH config (one-time)
+
+Developer adds this to `~/.ssh/config`:
+
+```
+Host *.zerolaghub.dev
+    ProxyCommand cloudflared access ssh --hostname %h
+    User dev
+```
+
+This is what makes `ssh dev-6070.zerolaghub.dev` route through Cloudflare
+instead of going directly to the IP. Without this config, SSH will fail.
+
+To confirm it's working, run with `-v` and look for:
+```
+Executing proxy command: cloudflared access ssh --hostname dev-6070.zerolaghub.dev
+```
+
+### 3. Configure tunnel ingress in Cloudflare Zero Trust dashboard
 
 In Zero Trust → Networks → Tunnels → your tunnel → Configure:
 
@@ -50,65 +122,32 @@ Add a public hostname:
 - Service type: `SSH`
 - URL: `localhost:22`
 
-This routes SSH traffic arriving at Cloudflare for `*.zerolaghub.dev` through
+### 4. Configure bastion SSH for proxy jump
-the tunnel to port 22 on the bastion.
 
-### 2. Configure bastion SSH for proxy jump
+On the bastion, edit `/etc/ssh/sshd_config`:
-
-On the bastion, edit `/etc/ssh/sshd_config` to allow the bastion to act as
-a jump host into dev containers:
 
 ```
-# Allow TCP forwarding for proxy jump
 AllowTcpForwarding yes
-PermitOpen any
-```
-
-Or scope it to the container subnet only:
-```
 PermitOpen 10.100.0.0/24:22
 ```
 
 Restart SSH: `systemctl restart sshd`
 
-### 3. Ensure dev containers have SSH running
+### 5. Ensure dev containers have SSH running
 
-Each dev container needs an SSH server running and accessible from the bastion:
+Each dev container needs openssh-server installed. Add to agent dev
+provisioning pipeline:
 
 ```bash
-# In dev container
 apt-get install -y openssh-server
 systemctl enable ssh
 systemctl start ssh
 ```
 
-The agent should handle this as part of dev container provisioning — add to
+### 6. VS Code Remote SSH
-agent's dev provisioning pipeline.
 
-### 4. Developer one-time setup
+After SSH config is in place, developer opens VS Code → Remote Explorer →
-
+Add new host → `ssh dev-6070.zerolaghub.dev`. VS Code handles the rest.
-Developer adds this to `~/.ssh/config` (portal will surface this as a
-copyable snippet):
-
-```
-Host *.zerolaghub.dev
-    ProxyCommand cloudflared access ssh --hostname %h
-    User dev
-```
-
-After that:
-```bash
-ssh dev-6070.zerolaghub.dev
-```
-
-### 5. VS Code Remote SSH
-
-Developer opens VS Code → Remote Explorer → Add new host:
-```
-ssh dev-6070.zerolaghub.dev
-```
-
-VS Code handles the rest — connects through the tunnel, mounts the workspace.
 
 ---
 
@@ -117,9 +156,8 @@ VS Code handles the rest — connects through the tunnel, mounts the workspace.
 - Bastion has no public IP — all SSH access goes through CF Tunnel only
 - CF Tunnel is outbound-only from bastion to Cloudflare — no inbound ports open
 - Zero Trust free tier: up to 50 users, core access control included
-- Developer must have `cloudflared` installed locally for the ProxyCommand
+- `cloudflared` on the client is a lightweight binary, not a service
-- code-server runs `--auth none` — SSH access is separate and enforced at
+- code-server runs `--auth none` — SSH is enforced separately at OS level
-  the OS level by the SSH server in the container
 
 ---
 
@@ -130,13 +168,15 @@ Portal dev container page should show:
 ```
 SSH Access
 ──────────
-Copy this to your ~/.ssh/config:
+1. Install cloudflared: https://developers.cloudflare.com/cloudflared/install
+
+2. Add to ~/.ssh/config:
 
 Host *.zerolaghub.dev
     ProxyCommand cloudflared access ssh --hostname %h
     User dev
 
-Then connect with:
+3. Connect:
 ssh dev-6070.zerolaghub.dev
 ```
 
@@ -148,7 +188,8 @@ One-time setup. Works for all dev containers after that.
 
 | Issue | Likely Cause |
 |-------|--------------|
-| `cloudflared: command not found` | Developer needs to install cloudflared client |
+| `kex_exchange_identification: Connection closed` | ProxyCommand not in SSH config — SSH going direct to IP |
+| `cloudflared: command not found` | cloudflared not installed on client machine |
 | `Connection refused` | SSH not running in container or bastion jump not configured |
 | `Permission denied` | SSH key not added to container's `dev` user |
-| Tunnel not connecting | `cloudflared` service not running on bastion — check `systemctl status cloudflared` |
+| Tunnel not connecting | `cloudflared` service not running on bastion — `systemctl status cloudflared` |