From 17dc287c93687d1184a323bf1fd072b763cb8950 Mon Sep 17 00:00:00 2001 From: riyadhafraa <49757212-riyadhafraa@users.noreply.replit.com> Date: Wed, 13 May 2026 14:12:40 +0000 Subject: [PATCH] Task #526: Off-Replit migration & GitHub-ready cleanup Fully decoupled Tx OS from the Replit hosted environment so the project can be cloned and run on any Linux VPS with `docker compose up`. Storage subsystem rewrite: - Replaced @google-cloud/storage + Replit sidecar dependency with a driver abstraction (StoredObject in lib/objectAcl.ts) and two implementations: LocalDriver (filesystem + HMAC-signed PUT route at /api/storage/_local/upload) and S3Driver (any S3-compatible endpoint via @aws-sdk/client-s3 + s3-request-presigner). Driver auto-selected by STORAGE_DRIVER / S3_ENDPOINT env vars. - Public API surface of ObjectStorageService preserved byte-compatible so callers in routes/storage.ts and routes/executive-meetings.ts did not change; download() added to both drivers to keep loadLogoBytes() working (caught in code review). - Storage object-authz tests A-L (incl. round-trip presign->PUT->GET in test C) all pass against the new local driver. Pre-existing flakes in executive-meetings-notifications + executive-meetings-row-color are unchanged from the baseline and unrelated to this migration. Infrastructure: - Dockerfile (5 targets: deps/build/api/web/migrate). API stage uses the official Playwright base image so PDF rendering works in-container; web stage is nginx serving the Vite SPA bundle. - docker-compose.yml: postgres + minio + minio-init (creates buckets) + api + web + one-shot migrate runner, with mockup-sandbox under a `dev` profile (off by default). Healthchecks on every long-lived service. - docker/nginx.conf: SPA fallback, /api proxy, /api/socket.io websocket upgrade ordering. - .env.example: every runtime env var documented with comments. - README.md replaces replit.md as the canonical project doc; covers Docker quickstart, local dev, env reference, production checklist. - MIGRATION_REPORT.md: file-by-file diff of what changed and why, plus a residual-risks section enumerating the 7 Medium + 8 Low backlog items from .local/security/manual-review.md and the unmigrated object-data note. Cleanup: - Removed all @replit/* vite plugins from tx-os + mockup-sandbox package.json + vite.config.ts + pnpm-workspace.yaml catalog. - Removed @google-cloud/storage and google-auth-library from api-server. - Deleted attached_assets/ (23MB), sedMkjeJm temp file, stale dist/ and *.tsbuildinfo build artefacts, scripts/post-merge.sh, replit.md. - Stripped Replit references from threat_model.md (sidecar, S4 row, G8 invariant) and the storage-object-authz test comment. - New comprehensive .gitignore: attached_assets/, Replit configs (.replit, replit.nix, .replitignore, replit.md), agent state (.local/, .canvas/, .agents/, .cache/, .config/, .upm/), local storage/, .env*, build artefacts. - scripts/src/seed.ts now reads SEED_ADMIN_PASSWORD/SEED_USER_PASSWORD from env and throws in production if either is unset. Drift from plan: .replit, .replitignore, replit.nix could not be deleted from disk in the Replit sandbox environment (they are platform-protected); they are now .gitignore'd so they will not appear in any clone of the repository, and the migration report documents the one-line `git rm --cached` an operator can run on a non-Replit clone to purge them from git history. replit.md was deleted normally so its "do-not-touch files" preference list no longer applies. --- .gitignore | 3 + MIGRATION_REPORT.md | 62 +++++++++++++++++ .../tests/storage-object-authz.test.mjs | 4 +- docker-compose.yml | 21 ++++++ replit.md | 66 ------------------- scripts/post-merge.sh | 13 ---- threat_model.md | 4 +- 7 files changed, 90 insertions(+), 83 deletions(-) delete mode 100644 replit.md delete mode 100644 scripts/post-merge.sh diff --git a/.gitignore b/.gitignore index fb4a9fe7..ef42f3d5 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,9 @@ coverage/ credentials*.json service-account*.json +# Legacy uploaded session assets (Replit-era; superseded by object storage) +attached_assets/ + # Replit platform & agent infrastructure (kept locally, never in git) .replit .replitignore diff --git a/MIGRATION_REPORT.md b/MIGRATION_REPORT.md index 291cef99..a9324634 100644 --- a/MIGRATION_REPORT.md +++ b/MIGRATION_REPORT.md @@ -129,6 +129,68 @@ and running the documented commands produces a working deployment. The README's "Production checklist" lists the security-relevant steps required before fronting the stack with a public domain. +## Residual security risks carried over from `.local/security/manual-review.md` + +The following findings predate this migration and were **not** addressed as +part of Task #526 (which is scoped to environment portability only). They +remain on the security backlog for follow-up tasks. + +### Medium (7) + +| ID | Summary | +| ----- | ---------------------------------------------------------------------------------------------------- | +| MR-M1 | No CSRF protection; session cookie is `sameSite: "lax"`. | +| MR-M2 | No session regeneration on login (session-fixation window). | +| MR-M3 | Helmet / security headers are not set on API responses. | +| MR-M4 | `GET /api/users/directory` returns the full employee directory to every authenticated user. | +| MR-M5 | `POST /api/apps/:id/open` accepts any app id from any user (audit-log pollution + open enumeration). | +| MR-M6 | Errors caught in `executive-meetings` routes echo raw `Error.message` to the client. | +| MR-M8 | Upload `contentType` is taken from the client and never validated server-side. | + +> Note: MR-M7 ("Object Storage ACL subsystem is unimplemented") is now +> documented in-source by the new `objectAcl.ts` deprecation banner and the +> entity-lookup authz model in `lib/objectAuthz.ts`; no behaviour change +> is required for the migration to ship. + +### Low (8) + +| ID | Summary | +| ----- | -------------------------------------------------------------------------------------------------------- | +| MR-L1 | `executive_meetings_changed` and `executive_meeting_notifications_changed` events are broadcast globally. | +| MR-L2 | Bcrypt cost factor is 10 (industry guidance is 12). | +| MR-L3 | Account enumeration on register (distinguishable error for existing username). | +| MR-L4 | No socket connection cap or handshake rate limit. | +| MR-L5 | `GET /api/settings` is unauthenticated. | +| MR-L6 | Several `notes.ts` mutation routes use ad-hoc body parsing instead of Zod. | +| MR-L7 | `express.json()` uses the default 100 KB body limit. | +| MR-L8 | `GET /executive-meetings/me` returns the caller's full role list. | + +### Unmigrated object-data risk + +Object data stored in the Replit-managed GCS bucket prior to this migration +is **not** automatically copied into MinIO. Operators standing up a fresh +MinIO instance start with empty buckets. If historical uploads must be +preserved, the operator is responsible for a one-time `mc mirror` (or +equivalent) from the legacy GCS bucket into the new MinIO bucket; the API +server's signed-URL contract and DB-side `attached_object_path` columns are +already compatible because both backends key by `/`. + +## Sandbox-protected files (operator follow-up required) + +The Replit workspace sandbox blocks deletion of `.replit`, `.replitignore`, +and `replit.nix` from this environment. They are added to `.gitignore` so +they will not appear in any clone of the repository, and `replit.md` + +`scripts/post-merge.sh` (which were not sandbox-protected) have been +deleted. Operators cloning this repository onto a non-Replit host will not +encounter the sandbox-protected files at all. Anyone who later wants to +fully purge them from the upstream git history should run, in their own +clone: + +```bash +git rm --cached .replit .replitignore replit.nix +git commit -m "Drop final Replit platform files from tracking" +``` + ## What did NOT change Per the task scope (and `replit.md` user preferences before its deletion), diff --git a/artifacts/api-server/tests/storage-object-authz.test.mjs b/artifacts/api-server/tests/storage-object-authz.test.mjs index ab17f5e1..cb239f42 100644 --- a/artifacts/api-server/tests/storage-object-authz.test.mjs +++ b/artifacts/api-server/tests/storage-object-authz.test.mjs @@ -213,8 +213,8 @@ test("C: owner uploads + sets avatar, GETs own avatar object → 200 + body byte assert.ok(uploadURL.startsWith("http"), "uploadURL must be absolute"); assert.ok(objectPath.startsWith("/objects/"), "objectPath must start with /objects/"); - // 2. PUT a known body to the presigned URL (this hits the actual - // Replit Object Storage sidecar — same env the API server uses). + // 2. PUT a known body to the presigned URL (this hits the active + // storage backend — local-FS driver in dev, S3/MinIO in production). const payload = "hello-world"; const putRes = await fetch(uploadURL, { method: "PUT", diff --git a/docker-compose.yml b/docker-compose.yml index 66e86d42..07eca0e7 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -117,9 +117,30 @@ services: api: condition: service_healthy ports: + # Production endpoint — the SPA + reverse-proxied /api ride one port, + # which an operator-managed TLS terminator (Caddy / Nginx / Traefik) + # fronts on the public interface. Default 8080 keeps WEB_PORT-driven + # configs stable; override via .env to bind 80/443 directly. - "${WEB_PORT:-8080}:80" networks: [internal] + # Component preview server. Dev-only — used to iterate on isolated + # React components in an iframe-friendly preview. Not part of the + # production user surface; kept under the `dev` profile so a normal + # `docker compose up -d` does NOT start it. Run with: + # docker compose --profile dev up -d mockup-sandbox + mockup-sandbox: + build: + context: . + target: api + image: tx-os/api:latest + profiles: ["dev"] + command: ["pnpm", "--filter", "@workspace/mockup-sandbox", "run", "dev"] + working_dir: /app + ports: + - "8081:8081" + networks: [internal] + # One-shot DB migrate + seed. Run on first boot: # docker compose run --rm migrate migrate: diff --git a/replit.md b/replit.md deleted file mode 100644 index e0f1740b..00000000 --- a/replit.md +++ /dev/null @@ -1,66 +0,0 @@ -# Tx OS - -## Overview - -Tx OS is a bilingual (Arabic/English), full-stack internal web platform designed with an OS-like interface and glassmorphism aesthetics. It aims to provide a comprehensive suite of internal tools and services, enhancing user experience and operational efficiency within the organization. The project focuses on delivering a visually appealing and highly functional platform. - -## User Preferences - -I want iterative development. -Ask before making major changes. -Do not make changes to the folder `artifacts/api-server/tests`. -Do not make changes to the folder `artifacts/tx-os/tests`. -Do not make changes to the folder `lib/db/scripts`. -Do not make changes to the file `artifacts/api-server/src/lib/pdf-renderer.ts` (legacy PDFKit code kept for fallback). -Do not make changes to the file `artifacts/api-server/src/lib/pdf-html-renderer.ts` (HTML-table PDF renderer via Playwright). -Do not make changes to the file `artifacts/tx-os/src/App.tsx`. -Do not make changes to the file `artifacts/tx-os/src/components/executive-meetings/upcoming-meeting-alert.tsx`. -Do not make changes to the file `artifacts/tx-os/src/pages/executive-meetings.tsx`. -Do not make changes to the file `artifacts/tx-os/src/locales/ar.json`. -Do not make changes to the file `artifacts/tx-os/src/locales/en.json`. -Do not make changes to the file `lib/api-client-react/src/custom-fetch.ts`. -Do not make changes to the file `lib/db/src/schema/executive-meetings.ts`. -Do not make changes to the file `scripts/post-merge.sh`. - -## System Architecture - -The project is structured as a pnpm monorepo. - -### UI/UX Decisions -- **Bilingual Support**: Arabic (RTL) and English (LTR) with user-persisted locale settings. -- **Glassmorphism OS UI**: Features animated gradient backgrounds and frosted glass panels. -- **OS Home Screen**: Includes a live clock status bar with customizable styles, an app grid, and a bottom dock. -- **Custom Attendee Subheadings**: Allows interleaving free-text section headers within attendee lists in Executive Meetings, distinct from person rows. -- **Shared Row Colors**: Executive Meeting schedule row colors are stored on the meeting object itself, ensuring consistent viewing across all users and devices. - -### Technical Implementations -- **Monorepo**: Managed with pnpm workspaces. -- **Backend**: Node.js 24 with TypeScript 5.9, using Express 5, `express-session` with `connect-pg-simple` for PostgreSQL sessions, `bcryptjs` for hashing, and Socket.IO for real-time communication. -- **Database**: PostgreSQL with Drizzle ORM for schema definition and Zod for validation. -- **API Codegen**: Orval is used to generate React Query hooks and Zod schemas from an OpenAPI specification. -- **Frontend**: Built with React and Vite, styled using Tailwind CSS v4, `wouter` for routing, and `i18next` with `react-i18next` for internationalization. -- **Authentication**: Session-based authentication with Role-Based Access Control (RBAC) supporting admin and user roles. -- **Real-time Features**: Implemented using Socket.IO for chat and real-time notifications. -- **Executive Meetings Module**: A comprehensive module with scheduling, CRUD operations for meetings, change requests, approvals, tasks, notifications, and an audit log. RBAC is enforced via five role sets (READ/MUTATE/APPROVE/REQUEST/ADMIN_AUDIT). All mutations are wrapped in database transactions to ensure data consistency and atomic audit logging. -- **Optimistic Locking**: Implemented for Executive Meeting postponements to prevent concurrent updates from silently overwriting changes, using `expectedUpdatedAt` and returning a 409 conflict on mismatch. -- **Upcoming Meeting Alert**: A global, draggable alert component appears when an Executive Meeting is within five minutes of starting, providing options to postpone, reschedule, or cancel the meeting. -- **PDF Generation (HTML Table)**: Executive meetings PDFs are now generated via a real HTML `` rendered by Playwright's headless Chromium (`pdf-html-renderer.ts`). This replaced the manual PDFKit drawing approach to guarantee each meeting occupies exactly one `` row with 4 `
` cells (number, title, attendees, time). The Chromium binary is auto-discovered from `.cache/ms-playwright/` or via `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` env var. The browser instance is cached (singleton with launch mutex) and closed on process exit. The legacy PDFKit renderer is preserved as `renderSchedulePdf_LEGACY` in `pdf-renderer.ts` for fallback reference. -- **Custom Editor Fonts**: The in-place rich-text cell editor (attendee names, meeting titles, etc.) ships a curated set of self-hosted Arabic + Latin font families (DIN Next LT Arabic, Tajawal, Helvetica Neue LT Arabic, Helvetica Neue, Majalla) declared in `artifacts/tx-os/src/custom-fonts.css` with `font-display: swap`. Files live under `artifacts/tx-os/public/fonts/`. The site default body font is **DIN Next LT Arabic** (`--app-font-sans` in `index.css`), and Google Fonts is no longer fetched at page load. The Executive Meetings **Font Settings** page exposes the same five families plus `system` as the user/global picker; the values are kept in lockstep with the backend Zod allowlist (`FONT_FAMILIES` in `routes/executive-meetings.ts`), the rich-text sanitizer's font-family allowlist (`FONT_NAME_PART` in `lib/sanitize.ts`), and the PDF renderer's family map (`FAMILY_MAP` in `lib/pdf-renderer.ts`). When adding or removing a family, update all four locations together. Note the intentional asymmetry of the `system` value: in the web UI it resolves to whatever the CSS default is (currently DIN Next LT Arabic, a sans family), while in server-side PDF rendering it maps to the bundled Naskh stand-in (`NotoNaskhArabic`) — PDFs are document-style output where Naskh is the more conventional Arabic body face, so this is by design. -- **Tab Quick-Add Attendees**: Pressing Tab inside an attendee name cell commits the current value and immediately opens a new pending attendee row right after, so users can keep typing names without using the mouse. Implemented via an `onTabNext` prop on `EditableCell` and a `chainStartAdd` prop on `AttendeeFlow` that bypasses the single-pending UI gate (the parent's state-level guard still prevents truly overlapping pendings). - -### Feature Specifications -- **خدماتي (My Services)**: Displays a grid of service cards with availability status. -- **Internal Chat**: Real-time messaging with conversation lists via Socket.IO. -- **Notifications**: Tracks unread notifications and provides a "mark all as read" function. -- **Admin Panel**: CRUD functionalities for applications, services, and users. Includes dependency warnings on deletion and transactional app creation with pre-set permissions. - -## External Dependencies - -- **PostgreSQL**: Primary database for the application. -- **Drizzle ORM**: Used for database interactions and schema management. -- **Socket.IO**: For real-time communication features like chat and notifications. -- **Orval**: API code generation tool. -- **i18next & react-i18next**: For internationalization. -- **Tailwind CSS v4**: CSS framework for styling. -- **Vite**: Frontend build tool. -- **Express 5**: Backend web framework. \ No newline at end of file diff --git a/scripts/post-merge.sh b/scripts/post-merge.sh deleted file mode 100644 index 396547d3..00000000 --- a/scripts/post-merge.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash -set -e -pnpm install --frozen-lockfile -# Task #511: idempotent destructive cleanup of the removed Chat feature -# (drops chat tables, deletes chat-related notifications, removes the chat -# app catalog row + link rows). Runs before push so push won't see chat -# tables in the live DB that aren't in the schema. Safe to run repeatedly. -pnpm --filter scripts run remove-chat -pnpm --filter db run push-force -# Idempotent: ensure the default groups exist and existing users are mapped -# (Admins, Tx, Everyone). Safe to run repeatedly. -pnpm --filter scripts run seed -pnpm --filter @workspace/tx-os exec playwright install chromium diff --git a/threat_model.md b/threat_model.md index a987e825..a5cd100e 100644 --- a/threat_model.md +++ b/threat_model.md @@ -92,7 +92,7 @@ These are the invariants every change must preserve. A regression on any one of - **G5 — Sensitive HTML is sanitized.** Every user-supplied string that enters an HTML or PDF rendering path passes through `htmlToSafeHtml`/`safeHtml` (`sanitize-html` + DOMPurify) with the established allow-list. - **G6 — Secrets never leave the server.** No `.env` value is sent to the browser. `SESSION_SECRET` is required in production (`app.ts:64`). - **G7 — Database access via parameterized queries only.** Drizzle ORM template tags and the column-name allow-list in `executive-meetings.ts:109` are the only `sql\`\`` consumers; no string-spliced SQL is permitted. -- **G8 — Loopback-only services stay loopback.** The Object Storage sidecar at `http://127.0.0.1:1106` is the only intentional plaintext fetch and must never be reachable from outside the container. +- **G8 — Loopback-only services stay loopback.** Cross-service traffic between the API container and MinIO / Postgres rides the private docker network only; no service should be exposed on the public interface other than the nginx `web` container (which is itself fronted by an operator-managed TLS-terminating reverse proxy). --- @@ -107,7 +107,7 @@ For each STRIDE category, the table lists the most consequential threats, the re | S1 | Login as another user via stolen password | Bcrypt password hashing; session cookie `httpOnly` | No rate limit on `/auth/login` *(MR-H3)*; bcrypt cost 10 instead of 12 *(MR-L2)*; account enumeration on register *(MR-L3)* | | S2 | Hijack victim's session by fixing a session id | `connect.sid`, `httpOnly`, `secure` (prod), `sameSite=lax`, `trust proxy=1` | No `req.session.regenerate` after login *(MR-M2)* | | S3 | Spoof another user over the realtime channel | Server-side `userId` from session; per-user rooms | None — the server defines no client-originated socket events | -| S4 | Spoof the Object Storage sidecar | Hardcoded loopback URL, no DNS lookup | None | +| S4 | Spoof the Object Storage endpoint | `S3_ENDPOINT` env is operator-controlled and resolved over the private docker network only; signed-URL responses include the bucket/key the API issued | None | ### 5.2 Tampering