ttrpg-initiative-tracker/docs/REWORK_PLAN.md

# Initiative Tracker — Rework Plan

Status: **APPROVED — executing**
Owner: draistrick (fork → `keen99/ttrpg-initiative-tracker`, private)
Upstream: `code.draft13.com/robert/ttrpg-initiative-tracker` (friend's Gitea)

---

## Goals

1. **Replace Firebase with self-hosted backend.** Browser cannot own a DB file (sandbox). Cross-device (DM + tablet + player view) requires a real backend. Backend is the foundation, built first.
2. **Automated test ecosystem as the baseline.** Lock current behavior before changing it. Skip bug must become provably impossible to reintroduce.
3. **Remain mergeable upstream.** Default behavior (Firebase) preserved behind flag. Upstream `main` stays clean. Friend keeps Firebase path.
4. **Self-hostable in local Docker** (in-house network). Public exposure = future, only after auth + multiuser safety.

## Non-Goals (this plan)

- Changing user-visible functionality beyond the documented bug fixes (skip, manual turn override).
- Ripping Firebase. Kept as default adapter upstream.
- Public/multiuser deployment. Deferred.
- Rewriting the entire 2935-line `App.js`. Only extract what testability demands.

---

## Problem Statement

### Why Firebase is wrong here (for this fork)
- Requires Google account + network for a single-user tabletop tool.
- Realtime value (DM view ↔ player display) is real but solvable locally.
- API key baked into client bundle (CRA `REACT_APP_*` at build); security depends entirely on console rules not in repo.
- Vendor lock + quota; `onSnapshot` on collections burns reads.
- Friend keeps it; we fork off it.

### Why a backend is mandatory
Browser sandbox cannot write the filesystem. No sqlite file, no `/data/db.sqlite`, nothing. Browser JS is blocked from disk by design. Therefore cross-device storage (DM ↔ tablet ↔ player view) requires a separate Node process owning the DB file and serving the browser over HTTP/WebSocket. There is no browser-only path. **The backend is step one, not deferred.**

### Known bug: initiative skips / lost state
Two failure classes observed:

1. **Race / data loss.** Every turn mutation = client reads snapshot → computes → writes whole doc back. Two interleaved actions → last-write-wins → state lost → skip. Firebase gives eventual snapshots, no transactions. Even single-user bites via optimistic UI vs server round-trip.
2. **Logic drift.** `turnOrderIds` array vs `participants` array vs `isActive` filter drift across mid-combat add/remove/toggle. `currentIndex === -1` fallback path is fragile. No invariant enforced. No way for DM to manually say "this participant's turn now."

### Undo is fragile
Current undo = stale snapshot write-back. Interleaved undos = data loss. Suspected already bitten during live game.

---

## Architecture

### Stack (locked)
- **Node.js** runtime
- **Express** web framework
- **ws** WebSocket lib (realtime push, replaces `onSnapshot`)
- **better-sqlite3** SQLite driver (synchronous, simple, fast)
- **SQLite** DB (single file, docker volume, trivial backup)
- **Jest** test runner (already in CRA deps)

Postgres deferred until public multiuser exposure is real. SQLite schema ports easily if that day comes.

### Backend design
- Owns SQLite file. Only writer.
- Holds authoritative state. Turn logic (initiative order, next-turn, add/remove mid-combat) runs server-side inside SQLite transaction.
- Client sends **action** (e.g. `NEXT_TURN`, not the resulting state). Server computes result, persists, broadcasts diff.
- Kills last-write-wins races by construction.
- WS broadcast on every state change → all connected clients (DM view, player display, tablet) update instantly.

### Three storage impls, one interface (frontend)

The storage interface is the test seam and the upstream-compat layer.

| Impl | When used | Automated-tested? |
|---|---|---|
| `firebase.js` | default (`STORAGE=firebase`) — upstream path | No — requires live Firebase project |
| `ws.js` | `STORAGE=ws` — our fork, talks to backend | Yes — against running backend |
| `memory.js` | test-only, in-process | Yes — fast, deterministic |

**Frontend interface contract** (all three implement):
- `getDoc(path)`, `setDoc(path, data, opts)`, `updateDoc(path, patch)`
- `deleteDoc(path)`, `batch(ops)`
- `subscribeDoc(path, cb)` / `subscribeCollection(path, cb)` → real-time push

Firebase impl: existing `onSnapshot` + SDK calls, moved verbatim behind interface (M2).
WS impl: thin client; dispatches **actions** to backend, receives **state updates** via WS subscribe (M2).
Memory impl: in-memory Map + EventEmitter, for tests (M3).

### Repo layout (npm workspaces)

```
/
  package.json              # workspaces root
  src/                      # React frontend (existing, refactored behind storage interface)
    storage/
      index.js              # factory: pick impl from STORAGE env
      firebase.js           # extracted from current App.js (verbatim)
      ws.js                 # NEW — talks to backend
      memory.js             # NEW — test only
      types.js              # interface contract (JSDoc)
  server/                   # NEW
    index.js                # Express + ws bootstrap, generic KV REST
    db.js                   # better-sqlite3, docs table (KV), broadcast
    ws-contract.test.js     # adapter vs live backend (Layer 2 test)
  shared/                   # pure logic, no I/O, importable by client + server + tests
    turn.js                 # turn logic (single source; server imports, tests import)
    types.js
  shared.test/              # turn logic unit tests (characterization + desired)
    turn.characterization.test.js
    turn.desired.test.js
  docker-compose.yml        # NEW — M5
  docs/
    REWORK_PLAN.md          # this file
```

### Auth
- **Now:** `AUTH_MODE=none`. App gated by nginx HTTP basic auth (reuse friend's existing pattern). In-house only. Risk acceptable: someone sees your initiative counter.
- **Future:** `AUTH_MODE=token` — real login, real users. Only if/when publicly exposed. Not built this plan.

---

## Milestones

Each milestone = independently mergeable PR upstream (unless marked ❌).

| M | Does | Tests? |
|---|---|---|
| 0 | repo, branch, remotes | no |
| 1 | build backend (Node+Express+ws+better-sqlite3) | unit tests as built |
| 2 | frontend WS adapter — app runs vs backend, cross-device works | yes |
| 3 | characterization tests lock current behavior (skip bug included) | yes |
| 4 | skip fix + manual override, regression-protected | yes |
| 5 | docker compose in-house | smoke |
| 6 | undo rework (tx events) | unit |
| 7 | playwright multi-window e2e (deferred) | e2e |

### Milestone 0 — Repo + branch setup ✅
- Fresh branch off `main` (not `dsr-rework`). Name: `rework-backend`.
- `upstream` remote = friend's Gitea (read-only fetch).
- Push origin = `keen99/ttrpg-initiative-tracker` (private).
- npm workspaces root config.
- Commit this plan.
- **Exit criteria:** clean branch, plan committed, remotes set. ✅ DONE (commit ad7979d, then plan-restored).
- **Upstream-PRable:** n/a (fork infra)

### Milestone 1 — Build backend
- `server/`: Express + ws + better-sqlite3.
- Schema mirrors current Firestore doc tree (campaigns, encounters subcoll, activeDisplay, logs).
- Turn logic (initiative order, next-turn, add/remove mid-combat) ported from `App.js` into `server/turn.js` (pure function, server-authoritative). Port verbatim — bugs included for now.
- Actions dispatched to backend; server computes result, persists in SQLite tx, broadcasts via WS.
- Unit tests as built: turn logic unit tests (characterization capturing current behavior), plus basic API/WS smoke.
- **Exit criteria:** backend boots, serves state over WS, persists to SQLite, unit tests green.
- **Upstream-PRable:** ❌ divergence (friend stays Firebase).

### Milestone 2 — Frontend WS adapter
- Define `storage/types.js` interface.
- Move all ~30 Firestore call sites from `App.js` into `storage/firebase.js` behind interface (verbatim).
- Implement `storage/ws.js` per interface, talking to backend. Dispatches actions, subscribes to WS.
- Implement `storage/memory.js` for frontend unit tests.
- `storage/index.js` factory: `STORAGE` env → pick impl. Default `firebase` (upstream unchanged).
- App runs against backend with `STORAGE=ws`.
- Cross-device verified manually: DM view + player display + tablet.
- **Exit criteria:** app runs fully against local backend, no Firebase. Multi-device sync works.
- **Upstream-PRable:** ⚠️ partial. Storage interface + firebase extract = ✅. WS impl = ❌.

### Milestone 3 — Characterization tests lock current behavior
- Lock current behavior end-to-end via integration tests against running backend (turn logic now server-side).
- Capture the skip bug as a characterization test (whatever current does = locked, bugs included).
- Cover: START, NEXT_TURN, PAUSE, RESUME, ADD_PARTICIPANT, REMOVE_PARTICIPANT, TOGGLE_ACTIVE, REORDER, APPLY_DAMAGE/HEAL, DEATH_SAVE, END.
- Iterate until confident: baseline solid, regressions impossible to silently slip.
- **Exit criteria:** characterization suite green against backend. Baseline locked.
- **Upstream-PRable:** ✅ if kept storage-agnostic (tests target turn logic shape).

### Milestone 4 — Skip fix + manual turn override
- Write desired-behavior tests (red):
  - Never-skip invariant: after `NEXT_TURN`, current participant is always a valid active participant, or encounter cleanly ends.
  - Mid-combat add enters turn order correctly.
  - Remove mid-combat doesn't skip next.
  - Pause/resume preserves order.
- Fix turn logic until red tests go green. Skip bug dies.
- Add new action: `JUMP_TURN_TO(participantId)`. DM clicks participant → cursor jumps → that participant's turn now → future `NEXT_TURN` continues from there. UI button label: "Make This Turn".
- Regression-protected by M3 characterization + new desired tests.
- **Exit criteria:** skip bug gone + provably cannot regress. Manual override works.
- **Upstream-PRable:** ✅ logic fix + new feature, both beneficial.

### Milestone 5 — Docker compose
- `docker-compose.yml`:
  - `backend` service (Node + sqlite volume)
  - `nginx` service (static frontend + reverse proxy + http basic auth)
- Profiles: `firebase` (frontend only, current behavior) vs `backend` (full stack).
- **Exit criteria:** `docker compose up` runs full stack in-house.
- **Upstream-PRable:** ❌ divergence.

### Milestone 6 — Undo rework
- Events table: every mutating action writes `(type, payload, undo_payload, undone, ts)`.
- Undo = apply `undo_payload` in same SQLite tx, flip `undone`. Transactional, no stale clobber.
- Replaces current fragile `/logs` snapshot-write undo.
- Migration: keep old undo working for existing entries until cleared; new format for new entries.
- **Exit criteria:** undo works transactionally; interleaved undos don't corrupt.
- **Upstream-PRable:** ⚠️ partial. Turn-logic-level undo = ✅. Backend events table = ❌.

### Milestone 7 — Playwright E2E (deferred)
- Multi-window E2E: DM view + display + player view in separate browser contexts against running backend.
- Verify realtime sync end-to-end.
- **Only build if sync regresses or we deviate significantly.** Turn-logic unit + backend integration tests cover most regression risk cheaper.
- **Exit criteria:** e2e green for core combat flow across 3 windows.
- **Upstream-PRable:** ✅ if test infra shared.

### Milestone 8 — (Future) Public exposure
- Real auth (`AUTH_MODE=token`).
- Rate limiting, CSRF, hardening.
- Postgres migration if load warrants.
- Only if we decide to expose publicly + multiuser.

---

## Testing strategy

### Layers
1. **Turn logic unit tests** (Jest, pure functions) — every turn transition, skip invariants, manual override. Built in M1 (characterization), extended in M4 (desired). Cheap, essential.
2. **Backend integration tests** (Jest) — spin server on random port, assert WS pushes + SQLite persists + transactional correctness. (M1+)
3. **Frontend adapter contract tests** (Jest, `memory`) — impl parity against interface. (M2)
4. **Playwright multi-window E2E** — deferred. Only realtime sync glue turn logic can't reach. (M7)

### Two-pass on turn logic (M1 → M4)
1. **Characterization** (M1/M3) — capture current behavior exactly (bugs included). Locks extraction/port as provably identical. Lets later fix be provable.
2. **Desired-behavior (red)** (M4) — write what *should* happen. Fail today. Fix → green. Bug dies, stays dead.

### Manual smoke via config flags
- `STORAGE=firebase` → current behavior (friend's path, upstream default).
- `STORAGE=ws` → our path, local backend.
- docker-compose profiles mirror the above.

### Accepted test gap
- Firebase adapter untested (requires live project). Accepted cost.
- Mitigated by: interface contract; if firebase impl drifts, integration smoke only.

---

## Mergability upstream

| Milestone | Upstream-PRable? | Why |
|---|---|---|
| 0 repo setup | n/a | fork infra |
| 1 backend | ❌ | divergence (friend stays Firebase) |
| 2 WS adapter | ⚠️ partial | interface + firebase extract ✅, WS ❌ |
| 3 characterization tests | ✅ | if storage-agnostic |
| 4 skip fix + manual override | ✅ | logic fix + beneficial feature |
| 5 docker compose | ❌ | divergence |
| 6 undo rework | ⚠️ partial | turn-logic-level ✅, events table ❌ |
| 7 playwright | ✅ | if test infra shared |

Default `STORAGE=firebase` + `AUTH_MODE=none` (unset) = upstream sees literally zero change.

---

## Risks

- **CRA + workspaces friction.** Create React App may resist monorepo layout. Mitigation: keep `src/` as CRA root, `server/` + `shared/` as separate workspaces imported via alias. Eject/craco only if forced.
- **Turn logic port correctness.** Current logic tangled; verbatim port risks subtle drift. Mitigation: characterization tests in M1/M3 lock behavior before any fix.
- **Firebase drift untested.** Mitigation: interface contract; friend's path his to maintain.
- **Undo history migration.** Existing log entries use old snapshot format. Mitigation: keep old undo working until cleared, new format for new entries.
- **WS reconnect/state-sync edge cases.** Transient drop mid-combat. Mitigation: client requests full state resync on (re)connect; server is source of truth.

---

## Decisions (locked)

1. **Branch:** `rework-backend` off `main`.
2. **Manual turn override:** action `JUMP_TURN_TO(participantId)`. UI button "Make This Turn".
3. **npm workspaces** for `server/` + `shared/` alongside CRA `src/`. Fallback alias if CRA fights.

---

## Next action

M0 ✅ DONE.
M1 kickoff: scaffold `server/` workspace, set up better-sqlite3 + Express + ws, port turn logic from `App.js` into `server/turn.js`, write first unit tests.