Task #632: realtime order delivery for recipients on iPad PWA

Bug: order creator (services manager) saw new orders instantly with
sound, but other users holding orders.receive — even with the app
foregrounded on the Orders screen — got nothing until they force-quit
and reopened the PWA, at which point the orders appeared. Root cause:
Socket.IO silently drops behind Tailscale/NAT on iPad PWA and the
client neither detected it nor refetched missed state on reconnect.
The creator was unaffected because their POST mutation updates their
queries locally without depending on the realtime channel.

Client (artifacts/tx-os/src/hooks/use-notifications-socket.ts):
- Tighten io() options: reconnection {Delay 500, DelayMax 3000,
  Attempts Infinity, timeout 10s}.
- Track `wasDisconnected` flag; on `disconnect` flip it and log the
  reason (skipping the clean "io client disconnect" path).
- On `connect`, if previously disconnected, invalidate every realtime-
  driven query key: notifications, home stats, my/incoming orders,
  notes, note-folders, exec meetings (list/alert-state/notifications),
  apps, /me, roles, permissions. Warmup window is reset BEFORE the
  refetch so any flushed notification_created events post-reconnect
  don't chime.
- Add a `visibilitychange` listener: on foreground return, if the
  socket is disconnected, force `socket.connect()`; if it's "connected"
  but possibly half-open (silent drop), round-trip a 3s-timeout
  `client_health_probe` ack — on timeout, `disconnect().connect()`.
- `connect_error` logger for field debugging.

Server (artifacts/api-server/src/index.ts):
- Tighten Socket.IO pingInterval=10s, pingTimeout=5s (was default
  25s/20s) so dead-connection detection cycle drops from ~45s to ~15s.
- Add `client_health_probe` handler that acks immediately — pairs
  with the client-side half-open probe.

Deviations from plan: skipped the optional UI connection indicator
(point 5) — not necessary to fix the reported bug; can ship later if
users still feel uncertain about connection state.

Architect approved with one minor caveat: severe (>3s) transient
latency on foreground could trigger a one-off socket cycle. Acceptable
tradeoff and explicitly documented in the comment.

Pre-existing TS errors in api-server/src/routes/push.ts are unrelated
to this task and not touched by this diff.

artifacts/api-server/src/index.ts

@@ -23,6 +23,15 @@ const httpServer = createServer(app);
 
 export const io = new SocketIOServer(httpServer, {
   path: "/api/socket.io",
+  // #632: Detect dead connections faster. Defaults are 25s/20s which
+  // means a silently-dropped WebSocket (Tailscale NAT timeout, iPad
+  // PWA suspension, proxy hiccup) takes up to ~45s before the client
+  // notices and reconnects — long enough that recipients miss the
+  // realtime order push entirely. Tightening to 10s/5s puts the
+  // detection cycle at ~15s, well inside the "is this app even
+  // working?" window users complain about.
+  pingInterval: 10000,
+  pingTimeout: 5000,
   cors: {
     origin: process.env.ALLOWED_ORIGINS
       ? process.env.ALLOWED_ORIGINS.split(",").map((o) => o.trim())
@@ -54,6 +63,15 @@ io.on("connection", (socket) => {
 
   socket.join(`user:${userId}`);
 
+  // #632: Client foreground-recovery probe. The web client emits this
+  // on `visibilitychange` to verify the WebSocket isn't half-open
+  // (silently dead behind Tailscale/NAT) — we ack immediately so a
+  // genuinely-alive connection completes the round-trip, and a dead
+  // one times out on the client side and force-cycles the transport.
+  socket.on("client_health_probe", (ack?: () => void) => {
+    if (typeof ack === "function") ack();
+  });
+
   socket.on("disconnect", () => {
     logger.info({ socketId: socket.id }, "Socket disconnected");
   });

artifacts/tx-os/src/hooks/use-notifications-socket.ts

@@ -100,10 +100,70 @@ export function useNotificationsSocket() {
 
     const socket = io(window.location.origin, {
       path: `${BASE}/api/socket.io`,
+      // #632: Behind Tailscale / proxies on iPad PWA the WebSocket can
+      // be silently dropped by NAT/idle timeouts. Tighten the
+      // reconnection budget so the client retries quickly instead of
+      // sitting on a dead connection while orders pile up server-side.
+      reconnection: true,
+      reconnectionDelay: 500,
+      reconnectionDelayMax: 3000,
+      reconnectionAttempts: Infinity,
+      timeout: 10000,
     });
 
+    // #632: Track whether the socket has been disconnected at any
+    // point so the next successful `connect` can refetch every
+    // realtime-driven query — events that fire while the socket was
+    // dead are lost forever otherwise, which is exactly the bug
+    // (recipients seeing orders only after force-quitting the PWA).
+    let wasDisconnected = false;
+    // #632: Replay every event-driven query key on reconnect. Mirrors
+    // the keys touched by individual socket.on handlers below — keep
+    // this list in sync when adding new realtime channels. Role and
+    // permission keys are included so a `role_permissions_changed`
+    // event that fired while the socket was dead is recovered on
+    // reconnect (per architect review).
+    const refetchRealtimeState = () => {
+      invalidate(getListNotificationsQueryKey());
+      invalidate(getGetHomeStatsQueryKey());
+      invalidate(getListIncomingServiceOrdersQueryKey());
+      invalidate(getListMyServiceOrdersQueryKey());
+      invalidate(["notes"]);
+      invalidate(["note-folders"]);
+      invalidate(["note-folders", "shared-with-me"]);
+      invalidate(["/api/executive-meetings"]);
+      invalidate(["/api/executive-meetings/alert-state"]);
+      invalidate(["/api/executive-meetings/notifications"]);
+      invalidate(getListAppsQueryKey());
+      invalidate(getGetMeQueryKey());
+      invalidate(getListRolesQueryKey());
+      invalidate(getListPermissionsQueryKey());
+    };
+
     socket.on("connect", () => {
+      // Reset the warmup window so a flush of pending notifications
+      // delivered immediately after reconnect doesn't chime.
       connectedAtRef.current = nowMs();
+      if (wasDisconnected) {
+        wasDisconnected = false;
+        refetchRealtimeState();
+      }
+    });
+
+    socket.on("disconnect", (reason) => {
+      wasDisconnected = true;
+      // Surface unexpected disconnects in the console for field
+      // debugging without breaking anything. `io client disconnect` is
+      // the cleanup path on hook teardown — skip the noise for it.
+      if (reason !== "io client disconnect") {
+        // eslint-disable-next-line no-console
+        console.warn("[socket] disconnected:", reason);
+      }
+    });
+
+    socket.on("connect_error", (err) => {
+      // eslint-disable-next-line no-console
+      console.warn("[socket] connect_error:", err.message);
     });
 
     socket.on("notification_created", (payload?: { type?: string }) => {
@@ -434,7 +494,46 @@ export function useNotificationsSocket() {
       },
     );
 
+    // #632: When the PWA / tab comes back to the foreground, iPad's
+    // background suspension may have silently killed the WebSocket
+    // even though `socket.connected` is still `true` (half-open
+    // socket — the client hasn't sent a ping yet). Probing with a
+    // timed roundtrip is the only reliable way to tell. We use the
+    // socket.io v4 ack-timeout helper to round-trip a cheap event;
+    // if the server doesn't ack within 3s, force-cycle the transport
+    // so missed orders surface within ~3s of returning to the app
+    // instead of waiting on the ping-timeout window (~15s).
+    const onVisibility = () => {
+      if (typeof document === "undefined") return;
+      if (document.visibilityState !== "visible") return;
+      if (!socket.connected) {
+        wasDisconnected = true;
+        socket.connect();
+        return;
+      }
+      // Half-open probe: round-trip a cheap event through socket.io's
+      // built-in ack channel. The server doesn't need to register a
+      // handler — when no handler exists, the ack still fires with
+      // no args, so a missing ack within 3s means the transport is
+      // dead and we should force-cycle it.
+      socket.timeout(3000).emit(
+        "client_health_probe",
+        (err: Error | null) => {
+          if (err && socket.connected) {
+            wasDisconnected = true;
+            socket.disconnect().connect();
+          }
+        },
+      );
+    };
+    if (typeof document !== "undefined") {
+      document.addEventListener("visibilitychange", onVisibility);
+    }
+
     return () => {
+      if (typeof document !== "undefined") {
+        document.removeEventListener("visibilitychange", onVisibility);
+      }
       if (rafHandle !== null) {
         if (typeof window !== "undefined" && typeof window.cancelAnimationFrame === "function") {
           window.cancelAnimationFrame(rafHandle);