Skip to content
MarginaliaDocsGitHub
Contents
product/history/webapp.md

Rendered from the repository — the file stays the source of truth.

Webapp — project history

Status: Snapshot as of 2026-08-18, written by session C4. Covers apps/webapp through session A3 (PR #15). Later sessions append below rather than rewriting. Sources: PR descriptions, handovers/, product/feasibility.md.

What was built

  • 2026-08-17 — Scaffold (PR #1). create-next-app (App Router, TypeScript, Tailwind v4, Turbopack, src dir), shadcn/ui initialized, Drizzle ORM + postgres-js, and a DDD skeleton under src/server/ with a placeholder notebooks table and repository.
  • 2026-08-17 — Domain schema and repositories, session A1 (PR #6). The full Phase 1 domain model: 7 tables (notebooks, sources, chunks, conversations, messages, citations, notes), generated SQL migrations, and owner-scoped repositories per aggregate. Chunks carry citation location metadata — char start/end offsets, nullable page number, a vector(2000) embedding, and a generated tsvector column for the full-text half of future hybrid search. 17 tests against PGlite (in-process Postgres + pgvector) migrated with the actual generated SQL.
  • 2026-08-17 — Auth, notebook library, workspace shell, session A2 (PR #10). Email+password auth via @supabase/ssr, /login + /signup, the notebook library as the authenticated home (instant create, inline rename, delete with confirmation), and the three-column workspace shell (Sources | Chat | Studio) at /notebooks/[id] for later sessions to fill. 24 tests total.
  • 2026-08-17 — Production Dockerfile, session B1 (PR #11). The SSE spike left two keepers in this workspace: the production Dockerfile (Bun installs dependencies, next build and runtime on node:24-slim, standalone output) and output: "standalone" + outputFileTracingRoot in next.config.ts (required in a monorepo). The throwaway /api/spike-stream route also lives here until A4 deletes it (SEC-4 in product/security.md).
  • 2026-08-18 — Source ingestion, session A3 (PR #15). Users add sources (file upload, website URL, pasted text) and they become retrievable, citation-ready chunks: parse → chunk → embed → store, with live status in the Sources panel and a read-only source viewer. 52 tests; verified end-to-end against local Supabase with real Scaleway embeddings (90 chunks across pasted text, a Wikipedia article, and a 15-page arXiv PDF — all embeddings 2000-dim, all offsets slice-exact).

Decisions and why

  • Node, not Bun, runs the production container (feasibility D-1, decided with the owner 2026-08-17). Bun 1.3 has acknowledged bugs on exactly our path: next build segfaults (oven-sh/bun#36866), ~670 MB idle RSS for the standalone server vs ~80 MB on Node (oven-sh/bun#34389), and open streaming issues — and streaming chat is the core feature. Bun stays as package manager, script runner, and test runner.
  • Embedding dimension vector(2000) (PR #6). The chosen embedding model qwen3-embedding-8b (D-4) natively outputs 4096 dimensions — above pgvector’s 2000-dim HNSW ceiling for vector columns and even above halfvec’s 4000. The model is Matryoshka-trained (32–4096 configurable via the API’s dimensions parameter), and Scaleway’s FAQ explicitly recommends 2000 dimensions with pgvector indexes. So the column is vector(2000) and every embedding call passes dimensions: 2000; the constant is exported as EMBEDDING_DIMENSIONS and the returned vector length is asserted before insert (A3).
  • One migration timeline, applied by the Supabase CLI (PR #6). Drizzle generates SQL (never push, D-3 — drizzle-kit push regenerates HNSW indexes without the operator class) into supabase/migrations/ with Supabase-style timestamps, forming one ordered timeline with the hand-written extension/RLS/storage migrations. Details in product/history/supabase.md.
  • Auth: the proxy is convenience, the server-side check is authoritative (PR #10). Token refresh and optimistic redirects live in src/proxy.ts — Next 16 renamed middleware.ts to proxy.ts, verified against the bundled Next docs. But every authenticated page and server action calls requireUser(), which validates the JWT via auth.getClaims() (never getSession()), because the Next docs warn that a matcher change can silently drop proxy coverage. The verified sub claim is the ownerId handed to every repository call; the client never supplies an owner id.
  • Authorization is app-layer first; RLS is defense-in-depth (PRs #6, #10; SEC-5). The app connects via the pooler as postgres, which RLS does not bind, so every repository method takes the owner id and scopes its queries by it. Missing and foreign rows both surface as NotFoundError — no existence leaks (a second user gets a 404 on the first user’s notebook URL, verified in A2’s click-through).
  • Ingestion runs in-process for now (D-2 stage 1, PR #15). Parse → chunk → embed runs via Next’s after(), with job state in the sources.status column from day one, so promoting the same worker code to Scaleway Serverless Jobs (stage 2) later changes neither schema nor UI.
  • Parser choices (PR #15). PDF via unpdf — extractText(pdf) without mergePages gives per-page text, so pages are chunked independently and every chunk carries an unambiguous pageNumber. URL via @mozilla/readability on a linkedom DOM. TXT/Markdown stored as-is with only line endings normalized, so offsets are OS-stable. Parsers are pure (bytes/string in), so tests use committed fixtures and never touch the network. DOCX was consciously left out: mammoth was not a trivial drop-in (new dependency, binary fixtures, its own failure modes); it slots in later as one more parser branch.
  • The offset invariant (PR #15). Every chunk satisfies content.slice(charStart, charEnd) === chunk.text — tested (including overlap and repeated-text cases) and SQL-verified (90/90 chunks in the e2e run). This is the raw material A5’s citation-to-passage navigation stands on.
  • Polling, not Realtime, for ingestion status (PR #15). A 2.5 s poll that only runs while a source is pending/processing. In-process ingestion finishes in seconds, so the window is short; Realtime postgres_changes would need the supabase_realtime publication plus RLS evaluation of the join-based sources policy inside walrus — more moving parts with a silent-failure mode, for no UX gain at these durations. Revisit at D-2 stage 2 when jobs get long.
  • Uploads go browser → Supabase Storage, never through the container (D-5, PR #15): the server receives only the storage path and validates the owner prefix. Kept even after spike S-1 disproved the rumored ~1 MB body limit — the path stays for resumability and RLS, not because of a limit.

Problems and how they were dealt with

  • Bun 1.3 isolated installs break Next standalone tracing. The node_modules/.bun symlink store loses @swc/helpers in the traced output. Found in B1 when the Docker image was verified locally before deploy; resolved by bun install --linker=hoisted in the Dockerfile (PR #11).
  • Next 16 renamed middleware.ts to proxy.ts. Found in A2 by verifying against the Next docs bundled in node_modules rather than memory; the auth refresh went into src/proxy.ts with the exported proxy function (PR #10).
  • The shadcn CLI had changed its flags out from under memory. During the scaffold (session 01, PR #1), bunx shadcn@latest init --yes --base-color neutral failed with unknown option '--base-color' — the current CLI had moved from per-option flags to a preset system. Found by the immediate CLI error on first invocation; resolved by inspecting init --help and switching to init --yes --defaults (the base-nova preset, which carries baseColor: neutral in the generated components.json). This is the CLI-flag flavor of the repo’s “never pin from memory” rule: generator flags are verified against --help at run time, not recalled from training data. (Source: foreman session record, recorded via PR #22 — see “Correcting the record” in product/history/process.md.)
  • The scaffold’s shadcn/ui is the Base UI flavor, not Radix. Noted in A2: composition uses render={...} props, not radix-style asChild — components added later must follow that idiom (handovers/2026-08-17-session-a2-auth-library.md).
  • SSRF via URL sources is only partially guarded (SEC-1). The fetcher blocks loopback/private/link-local hostnames but doesn’t resolve DNS and checks only the original hostname while redirects are followed. Found and documented in the A3 session itself; consciously accepted for the prototype (authenticated users only, no privileged network neighbors) with the hardening trigger recorded in product/security.md.
  • Oversized files are rejected only after parsing (SEC-2). The 20 MB cap is enforced at upload, but word-count limits run post-parse — a crafted PDF can spike CPU in-process. Accepted; the structural fix is D-2 stage 2 (parsing in a disposable job container).
  • Test-runner quirks under Bun + PGlite surfaced when CI first ran the suite with an exit-code check — exit code 99 despite 0 failures, and cold WASM init blowing the 5 s hook timeout on CI runners. The story lives in product/history/infrastructure.md (CI section); A-lane sessions should know both exist (handovers/2026-08-18-session-b2-ci-deploy.md).
  • Browser-automation friction during verification, not app bugs: take_screenshot times out on this Wayland setup (A2, A3 — click-throughs documented in prose instead), and chrome-devtools fill doesn’t fire React’s onChange (A3 dispatched manual input events).

Where the webapp stands

Sources can be added and become embedded, citation-ready chunks; auth and ownership are enforced end-to-end. Next per the roadmap: A4 grounded chat with retrieval and streamed citations (must honor the SEC-3 prompt-injection contract and delete the spike route, closing SEC-4), then A5 citations-to-passage navigation and notes.