Task #526: Off-Replit migration & GitHub-ready cleanup
Fully decoupled Tx OS from the Replit hosted environment so the project can be cloned and run on any Linux VPS with `docker compose up`. Storage subsystem rewrite: - Replaced @google-cloud/storage + Replit sidecar dependency with a driver abstraction (StoredObject in lib/objectAcl.ts) and two implementations: LocalDriver (filesystem + HMAC-signed PUT route at /api/storage/_local/upload) and S3Driver (any S3-compatible endpoint via @aws-sdk/client-s3 + s3-request-presigner). Driver auto-selected by STORAGE_DRIVER / S3_ENDPOINT env vars. - Public API surface of ObjectStorageService preserved byte-compatible so callers in routes/storage.ts and routes/executive-meetings.ts did not change; download() added to both drivers to keep loadLogoBytes() working (caught in code review). - Storage object-authz tests A-L (incl. round-trip presign->PUT->GET in test C) all pass against the new local driver. Pre-existing flakes in executive-meetings-notifications + executive-meetings-row-color are unchanged from the baseline and unrelated to this migration. Infrastructure: - Dockerfile (5 targets: deps/build/api/web/migrate). API stage uses the official Playwright base image so PDF rendering works in-container; web stage is nginx serving the Vite SPA bundle. - docker-compose.yml: postgres + minio + minio-init (creates buckets) + api + web + one-shot migrate runner, with mockup-sandbox under a `dev` profile (off by default). Healthchecks on every long-lived service. - docker/nginx.conf: SPA fallback, /api proxy, /api/socket.io websocket upgrade ordering. - .env.example: every runtime env var documented with comments. - README.md replaces replit.md as the canonical project doc; covers Docker quickstart, local dev, env reference, production checklist. - MIGRATION_REPORT.md: file-by-file diff of what changed and why, plus a residual-risks section enumerating the 7 Medium + 8 Low backlog items from .local/security/manual-review.md and the unmigrated object-data note. Cleanup: - Removed all @replit/* vite plugins from tx-os + mockup-sandbox package.json + vite.config.ts + pnpm-workspace.yaml catalog. - Removed @google-cloud/storage and google-auth-library from api-server. - Deleted attached_assets/ (23MB), sedMkjeJm temp file, stale dist/ and *.tsbuildinfo build artefacts, scripts/post-merge.sh, replit.md. - Stripped Replit references from threat_model.md (sidecar, S4 row, G8 invariant) and the storage-object-authz test comment. - New comprehensive .gitignore: attached_assets/, Replit configs (.replit, replit.nix, .replitignore, replit.md), agent state (.local/, .canvas/, .agents/, .cache/, .config/, .upm/), local storage/, .env*, build artefacts. - scripts/src/seed.ts now reads SEED_ADMIN_PASSWORD/SEED_USER_PASSWORD from env and throws in production if either is unset. Drift from plan: .replit, .replitignore, replit.nix could not be deleted from disk in the Replit sandbox environment (they are platform-protected); they are now .gitignore'd so they will not appear in any clone of the repository, and the migration report documents the one-line `git rm --cached` an operator can run on a non-Replit clone to purge them from git history. replit.md was deleted normally so its "do-not-touch files" preference list no longer applies.
This commit is contained in:
riyadhafraa
@@ -25,6 +25,9 @@ coverage/
|
||||
credentials*.json
|
||||
service-account*.json
|
||||
|
||||
# Legacy uploaded session assets (Replit-era; superseded by object storage)
|
||||
attached_assets/
|
||||
|
||||
# Replit platform & agent infrastructure (kept locally, never in git)
|
||||
.replit
|
||||
.replitignore
|
||||
|
||||
@@ -129,6 +129,68 @@ and running the documented commands produces a working deployment.
|
||||
The README's "Production checklist" lists the security-relevant steps required
|
||||
before fronting the stack with a public domain.
|
||||
|
||||
## Residual security risks carried over from `.local/security/manual-review.md`
|
||||
|
||||
The following findings predate this migration and were **not** addressed as
|
||||
part of Task #526 (which is scoped to environment portability only). They
|
||||
remain on the security backlog for follow-up tasks.
|
||||
|
||||
### Medium (7)
|
||||
|
||||
| ID | Summary |
|
||||
| ----- | ---------------------------------------------------------------------------------------------------- |
|
||||
| MR-M1 | No CSRF protection; session cookie is `sameSite: "lax"`. |
|
||||
| MR-M2 | No session regeneration on login (session-fixation window). |
|
||||
| MR-M3 | Helmet / security headers are not set on API responses. |
|
||||
| MR-M4 | `GET /api/users/directory` returns the full employee directory to every authenticated user. |
|
||||
| MR-M5 | `POST /api/apps/:id/open` accepts any app id from any user (audit-log pollution + open enumeration). |
|
||||
| MR-M6 | Errors caught in `executive-meetings` routes echo raw `Error.message` to the client. |
|
||||
| MR-M8 | Upload `contentType` is taken from the client and never validated server-side. |
|
||||
|
||||
> Note: MR-M7 ("Object Storage ACL subsystem is unimplemented") is now
|
||||
> documented in-source by the new `objectAcl.ts` deprecation banner and the
|
||||
> entity-lookup authz model in `lib/objectAuthz.ts`; no behaviour change
|
||||
> is required for the migration to ship.
|
||||
|
||||
### Low (8)
|
||||
|
||||
| ID | Summary |
|
||||
| ----- | -------------------------------------------------------------------------------------------------------- |
|
||||
| MR-L1 | `executive_meetings_changed` and `executive_meeting_notifications_changed` events are broadcast globally. |
|
||||
| MR-L2 | Bcrypt cost factor is 10 (industry guidance is 12). |
|
||||
| MR-L3 | Account enumeration on register (distinguishable error for existing username). |
|
||||
| MR-L4 | No socket connection cap or handshake rate limit. |
|
||||
| MR-L5 | `GET /api/settings` is unauthenticated. |
|
||||
| MR-L6 | Several `notes.ts` mutation routes use ad-hoc body parsing instead of Zod. |
|
||||
| MR-L7 | `express.json()` uses the default 100 KB body limit. |
|
||||
| MR-L8 | `GET /executive-meetings/me` returns the caller's full role list. |
|
||||
|
||||
### Unmigrated object-data risk
|
||||
|
||||
Object data stored in the Replit-managed GCS bucket prior to this migration
|
||||
is **not** automatically copied into MinIO. Operators standing up a fresh
|
||||
MinIO instance start with empty buckets. If historical uploads must be
|
||||
preserved, the operator is responsible for a one-time `mc mirror` (or
|
||||
equivalent) from the legacy GCS bucket into the new MinIO bucket; the API
|
||||
server's signed-URL contract and DB-side `attached_object_path` columns are
|
||||
already compatible because both backends key by `<bucket>/<objectName>`.
|
||||
|
||||
## Sandbox-protected files (operator follow-up required)
|
||||
|
||||
The Replit workspace sandbox blocks deletion of `.replit`, `.replitignore`,
|
||||
and `replit.nix` from this environment. They are added to `.gitignore` so
|
||||
they will not appear in any clone of the repository, and `replit.md` +
|
||||
`scripts/post-merge.sh` (which were not sandbox-protected) have been
|
||||
deleted. Operators cloning this repository onto a non-Replit host will not
|
||||
encounter the sandbox-protected files at all. Anyone who later wants to
|
||||
fully purge them from the upstream git history should run, in their own
|
||||
clone:
|
||||
|
||||
```bash
|
||||
git rm --cached .replit .replitignore replit.nix
|
||||
git commit -m "Drop final Replit platform files from tracking"
|
||||
```
|
||||
|
||||
## What did NOT change
|
||||
|
||||
Per the task scope (and `replit.md` user preferences before its deletion),
|
||||
|
||||
@@ -213,8 +213,8 @@ test("C: owner uploads + sets avatar, GETs own avatar object → 200 + body byte
|
||||
assert.ok(uploadURL.startsWith("http"), "uploadURL must be absolute");
|
||||
assert.ok(objectPath.startsWith("/objects/"), "objectPath must start with /objects/");
|
||||
|
||||
// 2. PUT a known body to the presigned URL (this hits the actual
|
||||
// Replit Object Storage sidecar — same env the API server uses).
|
||||
// 2. PUT a known body to the presigned URL (this hits the active
|
||||
// storage backend — local-FS driver in dev, S3/MinIO in production).
|
||||
const payload = "hello-world";
|
||||
const putRes = await fetch(uploadURL, {
|
||||
method: "PUT",
|
||||
|
||||
@@ -117,9 +117,30 @@ services:
|
||||
api:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
# Production endpoint — the SPA + reverse-proxied /api ride one port,
|
||||
# which an operator-managed TLS terminator (Caddy / Nginx / Traefik)
|
||||
# fronts on the public interface. Default 8080 keeps WEB_PORT-driven
|
||||
# configs stable; override via .env to bind 80/443 directly.
|
||||
- "${WEB_PORT:-8080}:80"
|
||||
networks: [internal]
|
||||
|
||||
# Component preview server. Dev-only — used to iterate on isolated
|
||||
# React components in an iframe-friendly preview. Not part of the
|
||||
# production user surface; kept under the `dev` profile so a normal
|
||||
# `docker compose up -d` does NOT start it. Run with:
|
||||
# docker compose --profile dev up -d mockup-sandbox
|
||||
mockup-sandbox:
|
||||
build:
|
||||
context: .
|
||||
target: api
|
||||
image: tx-os/api:latest
|
||||
profiles: ["dev"]
|
||||
command: ["pnpm", "--filter", "@workspace/mockup-sandbox", "run", "dev"]
|
||||
working_dir: /app
|
||||
ports:
|
||||
- "8081:8081"
|
||||
networks: [internal]
|
||||
|
||||
# One-shot DB migrate + seed. Run on first boot:
|
||||
# docker compose run --rm migrate
|
||||
migrate:
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
# Tx OS
|
||||
|
||||
## Overview
|
||||
|
||||
Tx OS is a bilingual (Arabic/English), full-stack internal web platform designed with an OS-like interface and glassmorphism aesthetics. It aims to provide a comprehensive suite of internal tools and services, enhancing user experience and operational efficiency within the organization. The project focuses on delivering a visually appealing and highly functional platform.
|
||||
|
||||
## User Preferences
|
||||
|
||||
I want iterative development.
|
||||
Ask before making major changes.
|
||||
Do not make changes to the folder `artifacts/api-server/tests`.
|
||||
Do not make changes to the folder `artifacts/tx-os/tests`.
|
||||
Do not make changes to the folder `lib/db/scripts`.
|
||||
Do not make changes to the file `artifacts/api-server/src/lib/pdf-renderer.ts` (legacy PDFKit code kept for fallback).
|
||||
Do not make changes to the file `artifacts/api-server/src/lib/pdf-html-renderer.ts` (HTML-table PDF renderer via Playwright).
|
||||
Do not make changes to the file `artifacts/tx-os/src/App.tsx`.
|
||||
Do not make changes to the file `artifacts/tx-os/src/components/executive-meetings/upcoming-meeting-alert.tsx`.
|
||||
Do not make changes to the file `artifacts/tx-os/src/pages/executive-meetings.tsx`.
|
||||
Do not make changes to the file `artifacts/tx-os/src/locales/ar.json`.
|
||||
Do not make changes to the file `artifacts/tx-os/src/locales/en.json`.
|
||||
Do not make changes to the file `lib/api-client-react/src/custom-fetch.ts`.
|
||||
Do not make changes to the file `lib/db/src/schema/executive-meetings.ts`.
|
||||
Do not make changes to the file `scripts/post-merge.sh`.
|
||||
|
||||
## System Architecture
|
||||
|
||||
The project is structured as a pnpm monorepo.
|
||||
|
||||
### UI/UX Decisions
|
||||
- **Bilingual Support**: Arabic (RTL) and English (LTR) with user-persisted locale settings.
|
||||
- **Glassmorphism OS UI**: Features animated gradient backgrounds and frosted glass panels.
|
||||
- **OS Home Screen**: Includes a live clock status bar with customizable styles, an app grid, and a bottom dock.
|
||||
- **Custom Attendee Subheadings**: Allows interleaving free-text section headers within attendee lists in Executive Meetings, distinct from person rows.
|
||||
- **Shared Row Colors**: Executive Meeting schedule row colors are stored on the meeting object itself, ensuring consistent viewing across all users and devices.
|
||||
|
||||
### Technical Implementations
|
||||
- **Monorepo**: Managed with pnpm workspaces.
|
||||
- **Backend**: Node.js 24 with TypeScript 5.9, using Express 5, `express-session` with `connect-pg-simple` for PostgreSQL sessions, `bcryptjs` for hashing, and Socket.IO for real-time communication.
|
||||
- **Database**: PostgreSQL with Drizzle ORM for schema definition and Zod for validation.
|
||||
- **API Codegen**: Orval is used to generate React Query hooks and Zod schemas from an OpenAPI specification.
|
||||
- **Frontend**: Built with React and Vite, styled using Tailwind CSS v4, `wouter` for routing, and `i18next` with `react-i18next` for internationalization.
|
||||
- **Authentication**: Session-based authentication with Role-Based Access Control (RBAC) supporting admin and user roles.
|
||||
- **Real-time Features**: Implemented using Socket.IO for chat and real-time notifications.
|
||||
- **Executive Meetings Module**: A comprehensive module with scheduling, CRUD operations for meetings, change requests, approvals, tasks, notifications, and an audit log. RBAC is enforced via five role sets (READ/MUTATE/APPROVE/REQUEST/ADMIN_AUDIT). All mutations are wrapped in database transactions to ensure data consistency and atomic audit logging.
|
||||
- **Optimistic Locking**: Implemented for Executive Meeting postponements to prevent concurrent updates from silently overwriting changes, using `expectedUpdatedAt` and returning a 409 conflict on mismatch.
|
||||
- **Upcoming Meeting Alert**: A global, draggable alert component appears when an Executive Meeting is within five minutes of starting, providing options to postpone, reschedule, or cancel the meeting.
|
||||
- **PDF Generation (HTML Table)**: Executive meetings PDFs are now generated via a real HTML `<table>` rendered by Playwright's headless Chromium (`pdf-html-renderer.ts`). This replaced the manual PDFKit drawing approach to guarantee each meeting occupies exactly one `<tr>` row with 4 `<td>` cells (number, title, attendees, time). The Chromium binary is auto-discovered from `.cache/ms-playwright/` or via `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` env var. The browser instance is cached (singleton with launch mutex) and closed on process exit. The legacy PDFKit renderer is preserved as `renderSchedulePdf_LEGACY` in `pdf-renderer.ts` for fallback reference.
|
||||
- **Custom Editor Fonts**: The in-place rich-text cell editor (attendee names, meeting titles, etc.) ships a curated set of self-hosted Arabic + Latin font families (DIN Next LT Arabic, Tajawal, Helvetica Neue LT Arabic, Helvetica Neue, Majalla) declared in `artifacts/tx-os/src/custom-fonts.css` with `font-display: swap`. Files live under `artifacts/tx-os/public/fonts/`. The site default body font is **DIN Next LT Arabic** (`--app-font-sans` in `index.css`), and Google Fonts is no longer fetched at page load. The Executive Meetings **Font Settings** page exposes the same five families plus `system` as the user/global picker; the values are kept in lockstep with the backend Zod allowlist (`FONT_FAMILIES` in `routes/executive-meetings.ts`), the rich-text sanitizer's font-family allowlist (`FONT_NAME_PART` in `lib/sanitize.ts`), and the PDF renderer's family map (`FAMILY_MAP` in `lib/pdf-renderer.ts`). When adding or removing a family, update all four locations together. Note the intentional asymmetry of the `system` value: in the web UI it resolves to whatever the CSS default is (currently DIN Next LT Arabic, a sans family), while in server-side PDF rendering it maps to the bundled Naskh stand-in (`NotoNaskhArabic`) — PDFs are document-style output where Naskh is the more conventional Arabic body face, so this is by design.
|
||||
- **Tab Quick-Add Attendees**: Pressing Tab inside an attendee name cell commits the current value and immediately opens a new pending attendee row right after, so users can keep typing names without using the mouse. Implemented via an `onTabNext` prop on `EditableCell` and a `chainStartAdd` prop on `AttendeeFlow` that bypasses the single-pending UI gate (the parent's state-level guard still prevents truly overlapping pendings).
|
||||
|
||||
### Feature Specifications
|
||||
- **خدماتي (My Services)**: Displays a grid of service cards with availability status.
|
||||
- **Internal Chat**: Real-time messaging with conversation lists via Socket.IO.
|
||||
- **Notifications**: Tracks unread notifications and provides a "mark all as read" function.
|
||||
- **Admin Panel**: CRUD functionalities for applications, services, and users. Includes dependency warnings on deletion and transactional app creation with pre-set permissions.
|
||||
|
||||
## External Dependencies
|
||||
|
||||
- **PostgreSQL**: Primary database for the application.
|
||||
- **Drizzle ORM**: Used for database interactions and schema management.
|
||||
- **Socket.IO**: For real-time communication features like chat and notifications.
|
||||
- **Orval**: API code generation tool.
|
||||
- **i18next & react-i18next**: For internationalization.
|
||||
- **Tailwind CSS v4**: CSS framework for styling.
|
||||
- **Vite**: Frontend build tool.
|
||||
- **Express 5**: Backend web framework.
|
||||
@@ -1,13 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
pnpm install --frozen-lockfile
|
||||
# Task #511: idempotent destructive cleanup of the removed Chat feature
|
||||
# (drops chat tables, deletes chat-related notifications, removes the chat
|
||||
# app catalog row + link rows). Runs before push so push won't see chat
|
||||
# tables in the live DB that aren't in the schema. Safe to run repeatedly.
|
||||
pnpm --filter scripts run remove-chat
|
||||
pnpm --filter db run push-force
|
||||
# Idempotent: ensure the default groups exist and existing users are mapped
|
||||
# (Admins, Tx, Everyone). Safe to run repeatedly.
|
||||
pnpm --filter scripts run seed
|
||||
pnpm --filter @workspace/tx-os exec playwright install chromium
|
||||
+2
-2
@@ -92,7 +92,7 @@ These are the invariants every change must preserve. A regression on any one of
|
||||
- **G5 — Sensitive HTML is sanitized.** Every user-supplied string that enters an HTML or PDF rendering path passes through `htmlToSafeHtml`/`safeHtml` (`sanitize-html` + DOMPurify) with the established allow-list.
|
||||
- **G6 — Secrets never leave the server.** No `.env` value is sent to the browser. `SESSION_SECRET` is required in production (`app.ts:64`).
|
||||
- **G7 — Database access via parameterized queries only.** Drizzle ORM template tags and the column-name allow-list in `executive-meetings.ts:109` are the only `sql\`\`` consumers; no string-spliced SQL is permitted.
|
||||
- **G8 — Loopback-only services stay loopback.** The Object Storage sidecar at `http://127.0.0.1:1106` is the only intentional plaintext fetch and must never be reachable from outside the container.
|
||||
- **G8 — Loopback-only services stay loopback.** Cross-service traffic between the API container and MinIO / Postgres rides the private docker network only; no service should be exposed on the public interface other than the nginx `web` container (which is itself fronted by an operator-managed TLS-terminating reverse proxy).
|
||||
|
||||
---
|
||||
|
||||
@@ -107,7 +107,7 @@ For each STRIDE category, the table lists the most consequential threats, the re
|
||||
| S1 | Login as another user via stolen password | Bcrypt password hashing; session cookie `httpOnly` | No rate limit on `/auth/login` *(MR-H3)*; bcrypt cost 10 instead of 12 *(MR-L2)*; account enumeration on register *(MR-L3)* |
|
||||
| S2 | Hijack victim's session by fixing a session id | `connect.sid`, `httpOnly`, `secure` (prod), `sameSite=lax`, `trust proxy=1` | No `req.session.regenerate` after login *(MR-M2)* |
|
||||
| S3 | Spoof another user over the realtime channel | Server-side `userId` from session; per-user rooms | None — the server defines no client-originated socket events |
|
||||
| S4 | Spoof the Object Storage sidecar | Hardcoded loopback URL, no DNS lookup | None |
|
||||
| S4 | Spoof the Object Storage endpoint | `S3_ENDPOINT` env is operator-controlled and resolved over the private docker network only; signed-URL responses include the bucket/key the API issued | None |
|
||||
|
||||
### 5.2 Tampering
|
||||
|
||||
|
||||
Reference in New Issue
Block a user