diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 86b8366..a8b014e 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,324 +1,21 @@ -# Costco Grocery List - AI Agent Instructions +# Copilot Compatibility Instructions -## Architecture Overview +## Precedence +- Source of truth: `PROJECT_INSTRUCTIONS.md` (repo root). +- Agent workflow constraints: `AGENTS.md` (repo root). +- Bugfix protocol: `DEBUGGING_INSTRUCTIONS.md` (repo root). -This is a full-stack grocery list management app with **role-based access control (RBAC)**: -- **Backend**: Node.js + Express + PostgreSQL (port 5000) -- **Frontend**: React 19 + TypeScript + Vite (port 3000/5173) -- **Deployment**: Docker Compose with separate dev/prod configurations +If any guidance in this file conflicts with the root instruction files, follow the root instruction files. -## Mobile-First Design Principles +## Current stack note +This repository is currently: +- Backend: Express (`backend/`) +- Frontend: React + Vite (`frontend/`) -**CRITICAL**: All UI components MUST be designed for both mobile and desktop from the start. +Apply architecture intent from `PROJECT_INSTRUCTIONS.md` using the current stack mapping in: +- `docs/AGENTIC_CONTRACT_MAP.md` -**Responsive Design Requirements**: -- Use relative units (`rem`, `em`, `%`, `vh/vw`) over fixed pixels where possible -- Implement mobile breakpoints: `480px`, `768px`, `1024px` -- Test layouts at: 320px (small phone), 375px (phone), 768px (tablet), 1024px+ (desktop) -- Avoid horizontal scrolling on mobile devices -- Touch targets minimum 44x44px for mobile usability -- Use `max-width` with `margin: 0 auto` for content containers -- Stack elements vertically on mobile, use flexbox/grid for larger screens -- Hide/collapse navigation into hamburger menus on mobile -- Ensure modals/dropdowns work well on small screens - -**Common Patterns**: -```css -/* Mobile-first approach */ -.container { - padding: 1rem; - max-width: 100%; -} - -@media (min-width: 768px) { - .container { - padding: 2rem; - max-width: 800px; - margin: 0 auto; - } -} -``` - -### Key Design Patterns - -**Dual RBAC System** - Two separate role hierarchies: - -**1. System Roles** (users.role column): -- `system_admin`: Access to Admin Panel for system-wide management (stores, users) -- `user`: Regular system user (default for new registrations) -- Defined in [backend/models/user.model.js](backend/models/user.model.js) -- Used for Admin Panel access control - -**2. Household Roles** (household_members.role column): -- `admin`: Can manage household members, change roles, delete household -- `user`: Can add/edit items, mark as bought (standard member permissions) -- Defined per household membership -- Used for household-level permissions (item management, member management) - -**Important**: Always distinguish between system role and household role: -- **System role**: From `AuthContext` or `req.user.role` - controls Admin Panel access -- **Household role**: From `activeHousehold.role` or `household_members.role` - controls household operations - -**Middleware chain pattern** for protected routes: -```javascript -// System-level protection -router.get("/stores", auth, requireRole("system_admin"), controller.getAllStores); - -// Household-level checks done in controller -router.post("/lists/:householdId/items", auth, controller.addItem); -``` -- `auth` middleware extracts JWT from `Authorization: Bearer ` header -- `requireRole` checks system role only -- Household role checks happen in controllers using `household.model.js` methods - -**Frontend route protection**: -- ``: Requires authentication, redirects to `/login` if no token -- ``: Requires system_admin role for Admin Panel -- Household permissions: Check `activeHousehold.role` in components (not route-level) -- Example in [frontend/src/App.jsx](frontend/src/App.jsx) - -**Multi-Household Architecture**: -- Users can belong to multiple households -- Each household has its own grocery lists, stores, and item classifications -- `HouseholdContext` manages active household selection -- All list operations are scoped to the active household - -## Database Schema - -**PostgreSQL server runs externally** - not in Docker Compose. Connection configured in [backend/.env](backend/.env) via standard environment variables. - -**Core Tables**: - -**users** - System users -- `id` (PK), `username`, `password` (bcrypt), `name`, `display_name` -- `role`: `system_admin` | `user` (default: `viewer` - legacy) -- System-level authentication and authorization - -**households** - Household entities -- `id` (PK), `name`, `invite_code`, `created_by`, `created_at` -- Each household is independent with own lists and members - -**household_members** - Junction table (users โ†” households) -- `id` (PK), `household_id` (FK), `user_id` (FK), `role`, `joined_at` -- `role`: `admin` | `user` (household-level permissions) -- One user can belong to multiple households with different roles - -**items** - Master item catalog -- `id` (PK), `name`, `default_image`, `default_image_mime_type`, `usage_count` -- Shared across all households, case-insensitive unique names - -**stores** - Store definitions (system-wide) -- `id` (PK), `name`, `default_zones` (JSONB array) -- Managed by system_admin in Admin Panel - -**household_stores** - Stores available to each household -- `id` (PK), `household_id` (FK), `store_id` (FK), `is_default` -- Links households to stores they use - -**household_lists** - Grocery list items per household -- `id` (PK), `household_id` (FK), `store_id` (FK), `item_id` (FK) -- `quantity`, `bought`, `custom_image`, `custom_image_mime_type` -- `added_by`, `modified_on` -- Scoped to household + store combination - -**household_list_history** - Tracks quantity contributions -- `id` (PK), `household_list_id` (FK), `quantity`, `added_by`, `added_on` -- Multi-contributor tracking (who added how much) - -**household_item_classifications** - Item classifications per household/store -- `id` (PK), `household_id`, `store_id`, `item_id` -- `item_type`, `item_group`, `zone`, `confidence`, `source` -- Household-specific overrides of global classifications - -**item_classification** - Global item classifications -- `id` (PK), `item_type`, `item_group`, `zone`, `confidence`, `source` -- System-wide defaults for item categorization - -**Legacy Tables** (deprecated, may still exist): -- `grocery_list`, `grocery_history` - Old single-household implementation - -**Important patterns**: -- No formal migration system - schema changes are manual SQL -- Items use case-insensitive matching (`ILIKE`) to prevent duplicates -- JOINs with `ARRAY_AGG` for multi-contributor queries (see [backend/models/list.model.v2.js](backend/models/list.model.v2.js)) -- All list operations require `household_id` parameter for scoping -- Image storage: `bytea` columns for images with separate MIME type columns - -## Development Workflow - -### Local Development -```bash -# Start all services with hot-reload against LOCAL database -docker-compose -f docker-compose.dev.yml up - -# Backend runs nodemon (watches backend/*.js) -# Frontend runs Vite dev server with HMR on port 3000 -``` - -**Key dev setup details**: -- Volume mounts preserve `node_modules` in containers while syncing source code -- Backend uses `Dockerfile` (standard) with `npm run dev` override -- Frontend uses `Dockerfile.dev` with `CHOKIDAR_USEPOLLING=true` for file watching -- Both connect to **external PostgreSQL server** (configured in `backend/.env`) -- No database container in compose - DB is managed separately - -### Production Build -```bash -# Local production build (for testing) -docker-compose -f docker-compose.prod.yml up --build - -# Actual production uses pre-built images -docker-compose up # Pulls from private registry -``` - -### CI/CD Pipeline (Gitea Actions) - -See [.gitea/workflows/deploy.yml](.gitea/workflows/deploy.yml) for full workflow: - -**Build stage** (on push to `main`): -1. Run backend tests (`npm test --if-present`) -2. Build backend image with tags: `:latest` and `:` -3. Build frontend image with tags: `:latest` and `:` -4. Push both images to private registry - -**Deploy stage**: -1. SSH to production server -2. Upload `docker-compose.yml` to deployment directory -3. Pull latest images and restart containers with `docker compose up -d` -4. Prune old images - -**Notify stage**: -- Sends deployment status via webhook - -**Required secrets**: -- `REGISTRY_USER`, `REGISTRY_PASS`: Docker registry credentials -- `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_KEY`: SSH deployment credentials - -### Backend Scripts -- `npm run dev`: Start with nodemon -- `npm run build`: esbuild compilation + copy public assets to `dist/` -- `npm test`: Run Jest tests (currently no tests exist) - -### Frontend Scripts -- `npm run dev`: Vite dev server (port 5173) -- `npm run build`: TypeScript compilation + Vite production build - -### Docker Configurations - -**docker-compose.yml** (production): -- Pulls pre-built images from private registry -- Backend on port 5000, frontend on port 3000 (nginx serves on port 80) -- Requires `backend.env` and `frontend.env` files - -**docker-compose.dev.yml** (local development): -- Builds images locally from Dockerfile/Dockerfile.dev -- Volume mounts for hot-reload: `./backend:/app` and `./frontend:/app` -- Named volumes preserve `node_modules` between rebuilds -- Backend uses `backend/.env` directly -- Frontend uses `Dockerfile.dev` with polling enabled for cross-platform compatibility - -**docker-compose.prod.yml** (local production testing): -- Builds images locally using production Dockerfiles -- Backend: Standard Node.js server -- Frontend: Multi-stage build with nginx serving static files - -## Configuration & Environment - -**Backend** ([backend/.env](backend/.env)): -- Database connection variables (host, user, password, database name) -- `JWT_SECRET`: Token signing key -- `ALLOWED_ORIGINS`: Comma-separated CORS whitelist (supports static origins + `192.168.*.*` IP ranges) -- `PORT`: Server port (default 5000) - -**Frontend** (environment variables): -- `VITE_API_URL`: Backend base URL - -**Config accessed via**: -- Backend: `process.env.VAR_NAME` -- Frontend: `import.meta.env.VITE_VAR_NAME` (see [frontend/src/config.ts](frontend/src/config.ts)) - -## Authentication Flow - -1. User logs in โ†’ backend returns `{token, userId, role, username}` ([backend/controllers/auth.controller.js](backend/controllers/auth.controller.js)) - - `role` is the **system role** (`system_admin` or `user`) -2. Frontend stores in `localStorage` and `AuthContext` ([frontend/src/context/AuthContext.jsx](frontend/src/context/AuthContext.jsx)) -3. `HouseholdContext` loads user's households and sets active household - - Active household includes `household.role` (the **household role**) -4. Axios interceptor auto-attaches `Authorization: Bearer ` header ([frontend/src/api/axios.js](frontend/src/api/axios.js)) -5. Backend validates JWT on protected routes ([backend/middleware/auth.js](backend/middleware/auth.js)) - - Sets `req.user = { id, role, username }` with **system role** -6. Controllers check household membership/role using [backend/models/household.model.js](backend/models/household.model.js) -7. On 401 "Invalid or expired token" response, frontend clears storage and redirects to login - -## Critical Conventions - -### Security Practices -- **Never expose credentials**: Do not hardcode or document actual values for `JWT_SECRET`, database passwords, API keys, or any sensitive configuration -- **No infrastructure details**: Avoid documenting specific IP addresses, domain names, deployment paths, or server locations in code or documentation -- **Environment variables**: Reference `.env` files conceptually - never include actual contents -- **Secrets in CI/CD**: Document that secrets are required, not their values -- **Code review**: Scan all changes for accidentally committed credentials before pushing - -### Backend -- **No SQL injection**: Always use parameterized queries (`$1`, `$2`, etc.) with [backend/db/pool.js](backend/db/pool.js) -- **Password hashing**: Use `bcryptjs` for hashing (see [backend/controllers/auth.controller.js](backend/controllers/auth.controller.js)) -- **CORS**: Dynamic origin validation in [backend/app.js](backend/app.js) allows configured origins + local IPs -- **Error responses**: Return JSON with `{message: "..."}` structure - -### Frontend -- **Mixed JSX/TSX**: Some components are `.jsx` (JavaScript), others `.tsx` (TypeScript) - maintain existing file extensions -- **API calls**: Use centralized `api` instance from [frontend/src/api/axios.js](frontend/src/api/axios.js), not raw axios -- **Role checks**: Access role from `AuthContext`, compare with constants from [frontend/src/constants/roles.js](frontend/src/constants/roles.js) -- **Navigation**: Use React Router's `` for redirects, not `window.location` (except in interceptor) - -## Common Tasks - -**Add a new protected route**: -1. Backend: Add route with `auth` middleware (+ `requireRole(...)` if system role check needed) -2. Frontend: Add route in [frontend/src/App.jsx](frontend/src/App.jsx) wrapped in `` (and `` for Admin Panel) - -**Access user info in backend controller**: -```javascript -const { id, role } = req.user; // Set by auth middleware (system role) -const userId = req.user.id; -``` - -**Check household permissions in backend controller**: -```javascript -const householdRole = await household.getUserRole(householdId, userId); -if (!householdRole) return res.status(403).json({ message: "Not a member of this household" }); -if (householdRole !== 'admin') return res.status(403).json({ message: "Household admin required" }); -``` - -**Check household permissions in frontend**: -```javascript -const { activeHousehold } = useContext(HouseholdContext); -const householdRole = activeHousehold?.role; // 'admin' or 'user' - -// Allow all members except viewers (no viewer role in households) -const canManageItems = householdRole && householdRole !== 'viewer'; // Usually just check if role exists - -// Admin-only actions -const canManageMembers = householdRole === 'admin'; -``` - -**Query grocery items with contributors**: -Use the JOIN pattern in [backend/models/list.model.v2.js](backend/models/list.model.v2.js) - aggregates user names via `household_list_history` table. - -## Testing - -**Backend**: -- Jest configured at root level ([package.json](package.json)) -- Currently **no test files exist** - testing infrastructure needs development -- CI/CD runs `npm test --if-present` but will pass if no tests found -- Focus area: API endpoint testing (use `supertest` with Express) - -**Frontend**: -- ESLint only (see [frontend/eslint.config.js](frontend/eslint.config.js)) -- No test runner configured -- Manual testing workflow in use - -**To add backend tests**: -1. Create `backend/__tests__/` directory -2. Use Jest + Supertest pattern for API tests -3. Mock database calls or use test database +## Safety reminders +- External DB only (`DATABASE_URL`), no DB container assumptions. +- No cron/worker additions unless explicitly approved. +- Never log secrets, receipt bytes, or full invite codes. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..e1b9e9f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,53 @@ +# AGENTS.md - Fiddy (External DB) + +## Authority +- Source of truth: `PROJECT_INSTRUCTIONS.md` (repo root). If conflict, follow it. +- Bugfix protocol: `DEBUGGING_INSTRUCTIONS.md` (repo root). +- Do not implement features unless required to fix the bug. + +## Non-negotiables +- External DB: `DATABASE_URL` points to on-prem Postgres (NOT a container). +- Dev/Prod share schema via migrations in `packages/db/migrations`. +- No cron/worker jobs. Fixes must work without background tasks. +- Server-side RBAC only. Client checks are UX only. + +## Security / logging (hard rules) +- Never log secrets (passwords/tokens/cookies). +- Never log receipt bytes. +- Never log full invite codes; logs/audit store last4 only. + +## Non-regression contracts +- Sessions are DB-backed (`sessions` table) and cookies are HttpOnly. +- Receipt images stored in `receipts` (`bytea`). +- Entries list endpoints must NEVER return receipt bytes. +- API responses must include `request_id`; audit logs must include `request_id`. + +## Architecture boundaries (follow existing patterns; do not invent) +1) API routes: `app/api/**/route.ts` + - Thin: parse/validate + call service, return JSON. +2) Server services: `lib/server/*` + - Own DB + authz. Must include `import "server-only";`. +3) Client wrappers: `lib/client/*` + - Typed fetch + error normalization; always send credentials. +4) Hooks: `hooks/use-*.ts` + - Primary UI-facing API layer; components avoid raw `fetch()`. + +## Next.js dynamic route params (required) +- In `app/api/**/[param]/route.ts`, treat `context.params` as async: + - `const { id } = await context.params;` + +## Working style +- Scan repo first; do not guess file names or patterns. +- Make the smallest change that resolves the issue. +- Keep touched files free of TS warnings and lint errors. +- Add/update tests when API behavior changes (include negative cases). +- Keep text encoding clean (no mojibake). + +## Response icon legend +Use the same status icons defined in `PROJECT_INSTRUCTIONS.md` section "Agent Response Legend (required)": +- `๐Ÿ”„` in progress +- `โœ…` completed +- `๐Ÿงช` verification/test result +- `โš ๏ธ` risk/blocker/manual action +- `โŒ` failure +- `๐Ÿงญ` recommendation/next step diff --git a/DEBUGGING_INSTRUCTIONS.md b/DEBUGGING_INSTRUCTIONS.md new file mode 100644 index 0000000..27b0db1 --- /dev/null +++ b/DEBUGGING_INSTRUCTIONS.md @@ -0,0 +1,48 @@ +# Debugging Instructions - Fiddy + +## Scope and authority +- This file is required for bugfix work. +- `PROJECT_INSTRUCTIONS.md` remains the source of truth for global project rules. +- For debugging tasks, ship the smallest safe fix that resolves the verified issue. + +## Required bugfix workflow +1. Reproduce: + - Capture exact route/page, inputs, actor role, and expected vs actual behavior. + - Record a concrete repro sequence before changing code. +2. Localize: + - Identify the failing boundary (route/controller/model/service/client wrapper/hook/ui). + - Confirm whether failure is validation, authorization, data, or rendering. +3. Fix minimally: + - Modify only the layers needed to resolve the bug. + - Do not introduce parallel mechanisms for the same state flow. +4. Verify: + - Re-run repro. + - Run lint/tests for touched areas. + - Confirm no regression against contracts in `PROJECT_INSTRUCTIONS.md`. + +## Guardrails while debugging +- External DB only: + - Use `DATABASE_URL`. + - Never add a DB container for a fix. +- No background jobs: + - Do not add cron, workers, or polling daemons. +- Security: + - Never log secrets, receipt bytes, or full invite codes. + - Invite logs/audit may include only last4. +- Authorization: + - Enforce RBAC server-side; client checks are UX only. + +## Contract-specific debug checks +- Auth: + - Sessions must remain DB-backed and cookie-based (HttpOnly). +- Receipts: + - List endpoints must never include receipt bytes. + - Byte retrieval must be through dedicated endpoint only. +- Request IDs/audit: + - Ensure `request_id` appears in responses and audit trail for affected paths. + +## Evidence to include with every bugfix +- Root cause summary (one short paragraph). +- Changed files list with rationale. +- Verification steps performed and outcome. +- Any residual risk, fallback, or operator action. diff --git a/PROJECT_INSTRUCTIONS.md b/PROJECT_INSTRUCTIONS.md new file mode 100644 index 0000000..ca0bfbd --- /dev/null +++ b/PROJECT_INSTRUCTIONS.md @@ -0,0 +1,201 @@ +# Project Instructions - Fiddy (External DB) + +## 1) Core expectation +This project connects to an **external Postgres instance (on-prem server)**. Dev and Prod must share the **same schema** through **migrations**. + +## 2) Authority & doc order +1) **PROJECT_INSTRUCTIONS.md** (this file) is the source of truth. +2) **DEBUGGING_INSTRUCTIONS.md** (repo root) is required for bugfix work. +3) Other instruction files (e.g. `.github/copilot-instructions.md`) must not conflict with this doc. + +If anything conflicts, follow **this** doc. + +--- + +## 3) Non-negotiables (hard rules) + +### External DB + migrations +- `DATABASE_URL` points to **on-prem Postgres** (**NOT** a container). +- Dev/Prod share schema via migrations in: `packages/db/migrations`. +- Active migration runbook: `docs/DB_MIGRATION_WORKFLOW.md` (active set + status commands). + +### No background jobs +- **No cron/worker jobs**. Any fix must work without background tasks. + +### Security / logging +- **Never log secrets** (passwords, tokens, session cookies). +- **Never log receipt bytes**. +- **Never log full invite codes** - logs/audit store **last4 only**. + +### Server-side authorization only +- **Server-side RBAC only.** Client checks are UX only and must not be trusted. + +--- + +## 4) Non-regression contracts (do not break) + +### Auth +- Custom email/password auth. +- Sessions are **DB-backed** and stored in table `sessions`. +- Session cookies are **HttpOnly**. + +### Receipts +- Receipt images are stored in Postgres `bytea` table `receipts`. +- **Entries list endpoints must never return receipt image bytes.** +- Receipt bytes are fetched only via a **separate endpoint** when inspecting a single item. + +### Request IDs + audit +- API must generate a **`request_id`** and return it in responses. +- Audit logs must include `request_id`. +- Audit logs must never store full invite codes (store **last4 only**). + +--- + +## 5) Architecture contract (Backend <-> Client <-> Hooks <-> UI) + +### No-assumptions rule (required) +Before making structural changes, first scan the repo and identify: +- where `app/`, `components/`, `features/`, `hooks/`, `lib/` live +- existing API routes and helpers +- patterns already in use +Do not invent files/endpoints/conventions. If something is missing, add it **minimally** and **consistently**. + +### Single mechanism rule (required) +For any cross-component state propagation concern, keep **one** canonical mechanism only: +- Context **OR** custom events **OR** cache invalidation +Do not keep old and new mechanisms in parallel. Remove superseded utilities/imports/files in the same PR. + +### Layering (hard boundaries) +For every domain (auth, groups, entries, receipts, etc.) follow this flow: + +1) **API Route Handlers** - `app/api/.../route.ts` +- Thin: parse/validate input, call a server service, return JSON. +- No direct DB queries in route files unless there is no existing server service. + +2) **Server Services (DB + authorization)** - `lib/server/*` +- Own all DB access and authorization helpers. +- Server-only modules must include: `import "server-only";` +- Prefer small domain modules: `lib/server/auth.ts`, `lib/server/groups.ts`, `lib/server/entries.ts`, `lib/server/receipts.ts`, `lib/server/session.ts`. + +3) **Client API Wrappers** - `lib/client/*` +- Typed fetch helpers only (no React state). +- Centralize fetch + error normalization. +- Always send credentials (cookies) and never trust client-side RBAC. + +4) **Hooks (UI-facing API layer)** - `hooks/use-*.ts` +- Hooks are the primary interface for components/pages to call APIs. +- Components should not call `fetch()` directly unless there is a strong reason. + +### API conventions +- Prefer consistent JSON error shape: + - `{ error: { code: string, message: string }, request_id?: string }` +- Validate inputs at the route boundary (shape/type), authorize in server services. +- Mirror existing REST style used in the project. + +### Next.js route params checklist (required) +For `app/api/**/[param]/route.ts`: +- Treat `context.params` as **async** and `await` it before reading properties. + - Example: `const { id } = await context.params;` + +### Frontend structure preference +- Prefer domain-first structure: `features//...` + `shared/...`. +- Use `components/*` only for compatibility shims during migrations (remove them after imports are migrated). + +### Maintainability thresholds (refactor triggers) +- Component files > **400 lines** should be split into container/presentational parts. +- Hook files > **150 lines** should extract helper functions/services. +- Functions with more than **3 nested branches** should be extracted. + +--- + +## 6) Decisions / constraints (Group Settings) +- Add `GROUP_OWNER` role to group roles; migrate existing groups so the first admin becomes owner. +- Join policy default is `NOT_ACCEPTING`. Policies: `NOT_ACCEPTING`, `AUTO_ACCEPT`, `APPROVAL_REQUIRED`. +- Both owner and admins can approve join requests and manage invite links. +- Invite links: + - TTL limited to 1-7 days. + - Settings are immutable after creation (policy, single-use, etc.). + - Single-use does not override approval-required. + - Expired links are retained and can be revived. + - Single-use links are deleted after successful use. + - Revive resets `used_at` and `revoked_at`, refreshes `expires_at`, and creates a new audit event. +- No cron/worker jobs for now (auto ownership transfer and invite rotation are paused). +- Group role icons must be consistent: owner, admin, member. + +--- + +## 7) Do first (vertical slice) +1) DB migrate command + schema +2) Register/Login/Logout (custom sessions) +3) Protected dashboard page +4) Group create/join + group switcher (approval-based joins + optional join disable) +5) Entries CRUD (no receipt bytes in list) +6) Receipt upload/download endpoints +7) Settings + Reports + +--- + +## 8) Definition of done +- Works via `docker-compose.dev.yml` with external DB +- Migrations applied via `npm run db:migrate` +- Tests + lint pass +- RBAC enforced server-side +- No large files +- No TypeScript warnings or lint errors in touched files +- No new cron/worker dependencies unless explicitly approved +- No orphaned utilities/hooks/contexts after refactors +- No duplicate mechanisms for the same state flow +- Text encoding remains clean in user-facing strings/docs + +--- + +## 9) Desktop + mobile UX checklist (required) +- Touch: long-press affordance for item-level actions when no visible button. +- Mouse: hover affordance on interactive rows/cards. +- Tap targets remain >= 40px on mobile. +- Modal overlays must close on outside click/tap. +- Use bubble notifications for main actions (create/update/delete/join). +- Add Playwright UI tests for new UI features and critical flows. + +--- + +## 10) Tests (required) +- Add/update tests for API behavior changes (auth, groups, entries, receipts). +- Include negative cases where applicable: + - unauthorized + - not-a-member + - invalid input + +--- + +## 11) Agent Response Legend (required) +Use emoji/icons in agent progress and final responses so status is obvious at a glance. + +Legend: +- `๐Ÿ”„` in progress +- `โœ…` completed +- `๐Ÿงช` test/lint/verification result +- `๐Ÿ“„` documentation update +- `๐Ÿ—„๏ธ` database or migration change +- `๐Ÿš€` deploy/release step +- `โš ๏ธ` risk, blocker, or manual operator action needed +- `โŒ` failed command or unsuccessful attempt +- `โ„น๏ธ` informational context +- `๐Ÿงญ` recommendation or next-step option + +Usage rules: +- Include at least one status icon in each substantive agent response. +- Use one icon per bullet/line; avoid icon spam. +- Keep icon meaning consistent with this legend. + +--- + +## 12) Commit Discipline (required) +- Commit in small, logical slices (no broad mixed-purpose commits). +- Each commit must: + - follow Conventional Commits style (`feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`) + - include only related files for that slice + - exclude secrets, credentials, and generated noise +- Run verification before commit when applicable (lint/tests/build or targeted checks for touched areas). +- Prefer frequent checkpoint commits during agentic work rather than one large end-state commit. +- If a rule or contract changes, commit docs first (or in the same atomic slice as enforcing code). diff --git a/backend/controllers/auth.controller.js b/backend/controllers/auth.controller.js index c1866c6..7574efe 100644 --- a/backend/controllers/auth.controller.js +++ b/backend/controllers/auth.controller.js @@ -2,10 +2,10 @@ const bcrypt = require("bcryptjs"); const jwt = require("jsonwebtoken"); const User = require("../models/user.model"); -exports.register = async (req, res) => { - let { username, password, name } = req.body; - username = username.toLowerCase(); - console.log(`๐Ÿ†• Registration attempt for ${name} => username:${username}, password:${password}`); +exports.register = async (req, res) => { + let { username, password, name } = req.body; + username = username.toLowerCase(); + console.log(`Registration attempt for ${name} => username:${username}`); try { const hash = await bcrypt.hash(password, 10); @@ -28,11 +28,11 @@ exports.login = async (req, res) => { return res.status(401).json({ message: "User not found" }); } - const valid = await bcrypt.compare(password, user.password); - if (!valid) { - console.log(`โ›” Login attempt for user ${username} with password ${password}`); - return res.status(401).json({ message: "Invalid credentials" }); - } + const valid = await bcrypt.compare(password, user.password); + if (!valid) { + console.log(`Invalid login attempt for user ${username}`); + return res.status(401).json({ message: "Invalid credentials" }); + } const token = jwt.sign( { id: user.id, role: user.role }, diff --git a/docs/AGENTIC_CONTRACT_MAP.md b/docs/AGENTIC_CONTRACT_MAP.md new file mode 100644 index 0000000..c27da56 --- /dev/null +++ b/docs/AGENTIC_CONTRACT_MAP.md @@ -0,0 +1,49 @@ +# Agentic Contract Map (Current Stack) + +This file maps `PROJECT_INSTRUCTIONS.md` architecture intent to the current repository stack. + +## Current stack +- Backend: Express (`backend/`) +- Frontend: React + Vite (`frontend/`) + +## Contract mapping + +### API Route Handlers (`app/api/**/route.ts` intent) +Current equivalent: +- `backend/routes/*.js` +- `backend/controllers/*.js` + +Expectation: +- Keep these thin for parsing/validation and response shape. +- Delegate DB and authorization-heavy logic to model/service layers. + +### Server Services (`lib/server/*` intent) +Current equivalent: +- `backend/models/*.js` +- `backend/middleware/*.js` +- `backend/db/*` + +Expectation: +- Concentrate DB access and authorization logic in these backend layers. +- Avoid raw DB usage directly in route files unless no service/model exists. + +### Client Wrappers (`lib/client/*` intent) +Current equivalent: +- `frontend/src/api/*.js` + +Expectation: +- Centralize fetch/axios calls and error normalization here. +- Always send credentials/authorization headers as required. + +### Hooks (`hooks/use-*.ts` intent) +Current equivalent: +- `frontend/src/context/*` +- `frontend/src/utils/*` for route guards + +Expectation: +- Keep components free of direct raw network calls where possible. +- Favor one canonical state propagation mechanism per concern. + +## Notes +- This map does not force a framework migration. +- It defines how to apply the contract consistently in the existing codebase. diff --git a/docs/DB_MIGRATION_WORKFLOW.md b/docs/DB_MIGRATION_WORKFLOW.md new file mode 100644 index 0000000..3eae37a --- /dev/null +++ b/docs/DB_MIGRATION_WORKFLOW.md @@ -0,0 +1,50 @@ +# DB Migration Workflow (External Postgres) + +This project uses an external on-prem Postgres database. Migration files are canonical in: + +- `packages/db/migrations` + +## Preconditions +- `DATABASE_URL` is set and points to the on-prem Postgres instance. +- `psql` is installed and available in PATH. +- You are in repo root. + +## Commands +- Apply pending migrations: + - `npm run db:migrate` +- Show migration status: + - `npm run db:migrate:status` +- Fail if pending migrations exist: + - `npm run db:migrate:verify` + +## Active migration set +Migration files are applied in lexicographic filename order from `packages/db/migrations`. + +Current baseline files: +- `add_display_name_column.sql` +- `add_image_columns.sql` +- `add_modified_on_column.sql` +- `add_notes_column.sql` +- `create_item_classification_table.sql` +- `multi_household_architecture.sql` + +## Tracking table +Applied migrations are recorded in: + +- `schema_migrations(filename text unique, applied_at timestamptz)` + +## Expected operator flow +1. Check status: + - `npm run db:migrate:status` +2. Apply pending: + - `npm run db:migrate` +3. Verify clean state: + - `npm run db:migrate:verify` + +## Troubleshooting +- `DATABASE_URL is required`: + - Export/set `DATABASE_URL` in your environment. +- `psql executable was not found in PATH`: + - Install PostgreSQL client tools and retry. +- SQL failure: + - Fix migration SQL and rerun; only successful files are recorded in `schema_migrations`. diff --git a/docs/README.md b/docs/README.md index 0d9285e..b2487a5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -39,7 +39,10 @@ Historical documentation of completed features. Useful for reference but not act These files remain at the project root for easy access: - **[../README.md](../README.md)** - Project overview and quick start -- **[../.github/copilot-instructions.md](../.github/copilot-instructions.md)** - AI assistant instructions (architecture, RBAC, conventions) +- **[../PROJECT_INSTRUCTIONS.md](../PROJECT_INSTRUCTIONS.md)** - Canonical project constraints and delivery contract +- **[../AGENTS.md](../AGENTS.md)** - Agent behavior and guardrails +- **[../DEBUGGING_INSTRUCTIONS.md](../DEBUGGING_INSTRUCTIONS.md)** - Required bugfix workflow +- **[../.github/copilot-instructions.md](../.github/copilot-instructions.md)** - Copilot compatibility shim to root instructions --- @@ -51,9 +54,9 @@ These files remain at the project root for easy access: **Working on mobile UI?** โ†’ Check [MOBILE_RESPONSIVE_AUDIT.md](guides/MOBILE_RESPONSIVE_AUDIT.md) -**Need architecture context?** โ†’ Read [../.github/copilot-instructions.md](../.github/copilot-instructions.md) +**Need architecture context?** โ†’ Read [AGENTIC_CONTRACT_MAP.md](AGENTIC_CONTRACT_MAP.md) and [../PROJECT_INSTRUCTIONS.md](../PROJECT_INSTRUCTIONS.md) -**Running migrations?** โ†’ Follow [MIGRATION_GUIDE.md](migration/MIGRATION_GUIDE.md) +**Running migrations?** โ†’ Follow [DB_MIGRATION_WORKFLOW.md](DB_MIGRATION_WORKFLOW.md) --- diff --git a/package.json b/package.json index 96a5802..1914b2c 100644 --- a/package.json +++ b/package.json @@ -1,4 +1,9 @@ { + "scripts": { + "db:migrate": "node scripts/db-migrate.js", + "db:migrate:status": "node scripts/db-migrate-status.js", + "db:migrate:verify": "node scripts/db-migrate-verify.js" + }, "devDependencies": { "cross-env": "^10.1.0", "jest": "^30.2.0", diff --git a/packages/db/migrations/README.md b/packages/db/migrations/README.md new file mode 100644 index 0000000..60cb0f3 --- /dev/null +++ b/packages/db/migrations/README.md @@ -0,0 +1,9 @@ +# Migration Directory + +This directory is the canonical location for SQL migrations. + +- Use `npm run db:migrate` to apply pending migrations. +- Use `npm run db:migrate:status` to view applied/pending migrations. +- Use `npm run db:migrate:verify` to fail when pending migrations exist. + +Do not place new canonical migrations under `backend/migrations`. diff --git a/packages/db/migrations/add_display_name_column.sql b/packages/db/migrations/add_display_name_column.sql new file mode 100644 index 0000000..37e559e --- /dev/null +++ b/packages/db/migrations/add_display_name_column.sql @@ -0,0 +1,10 @@ +-- Add display_name column to users table +-- This allows users to have a friendly name separate from their username + +ALTER TABLE users +ADD COLUMN IF NOT EXISTS display_name VARCHAR(100); + +-- Set display_name to name for existing users (as default) +UPDATE users +SET display_name = name +WHERE display_name IS NULL; diff --git a/packages/db/migrations/add_image_columns.sql b/packages/db/migrations/add_image_columns.sql new file mode 100644 index 0000000..6f5a061 --- /dev/null +++ b/packages/db/migrations/add_image_columns.sql @@ -0,0 +1,20 @@ +# Database Migration: Add Image Support + +Run these SQL commands on your PostgreSQL database: + +```sql +-- Add image columns to grocery_list table +ALTER TABLE grocery_list +ADD COLUMN item_image BYTEA, +ADD COLUMN image_mime_type VARCHAR(50); + +-- Optional: Add index for faster queries when filtering by items with images +CREATE INDEX idx_grocery_list_has_image ON grocery_list ((item_image IS NOT NULL)); +``` + +## To Verify: +```sql +\d grocery_list +``` + +You should see the new columns `item_image` and `image_mime_type`. diff --git a/packages/db/migrations/add_modified_on_column.sql b/packages/db/migrations/add_modified_on_column.sql new file mode 100644 index 0000000..2034edc --- /dev/null +++ b/packages/db/migrations/add_modified_on_column.sql @@ -0,0 +1,8 @@ +-- Add modified_on column to grocery_list table +ALTER TABLE grocery_list +ADD COLUMN modified_on TIMESTAMP DEFAULT NOW(); + +-- Set modified_on to NOW() for existing records +UPDATE grocery_list +SET modified_on = NOW() +WHERE modified_on IS NULL; diff --git a/packages/db/migrations/add_notes_column.sql b/packages/db/migrations/add_notes_column.sql new file mode 100644 index 0000000..977dd22 --- /dev/null +++ b/packages/db/migrations/add_notes_column.sql @@ -0,0 +1,7 @@ +-- Add notes column to household_lists table +-- This allows users to add custom notes/descriptions to list items + +ALTER TABLE household_lists +ADD COLUMN IF NOT EXISTS notes TEXT; + +COMMENT ON COLUMN household_lists.notes IS 'Optional user notes/description for the item'; diff --git a/packages/db/migrations/create_item_classification_table.sql b/packages/db/migrations/create_item_classification_table.sql new file mode 100644 index 0000000..effc63a --- /dev/null +++ b/packages/db/migrations/create_item_classification_table.sql @@ -0,0 +1,29 @@ +-- Migration: Create item_classification table +-- This table stores classification data for items in the grocery_list table +-- Each row in grocery_list can have ONE corresponding classification row + +CREATE TABLE IF NOT EXISTS item_classification ( + id INTEGER PRIMARY KEY REFERENCES grocery_list(id) ON DELETE CASCADE, + item_type VARCHAR(50) NOT NULL, + item_group VARCHAR(100) NOT NULL, + zone VARCHAR(100), + confidence DECIMAL(3,2) DEFAULT 1.0 CHECK (confidence >= 0 AND confidence <= 1), + source VARCHAR(20) DEFAULT 'user' CHECK (source IN ('user', 'ml', 'default')), + created_at TIMESTAMP DEFAULT NOW(), + updated_at TIMESTAMP DEFAULT NOW() +); + +-- Index for faster lookups by type +CREATE INDEX IF NOT EXISTS idx_item_classification_type ON item_classification(item_type); + +-- Index for zone-based queries +CREATE INDEX IF NOT EXISTS idx_item_classification_zone ON item_classification(zone); + +-- Comments +COMMENT ON TABLE item_classification IS 'Stores classification metadata for grocery list items'; +COMMENT ON COLUMN item_classification.id IS 'Foreign key to grocery_list.id (one-to-one relationship)'; +COMMENT ON COLUMN item_classification.item_type IS 'High-level category (produce, meat, dairy, etc.)'; +COMMENT ON COLUMN item_classification.item_group IS 'Subcategory within item_type (filtered by type)'; +COMMENT ON COLUMN item_classification.zone IS 'Store zone/location (optional)'; +COMMENT ON COLUMN item_classification.confidence IS 'Confidence score 0-1 (1.0 for user-provided, lower for ML-predicted)'; +COMMENT ON COLUMN item_classification.source IS 'Source of classification: user, ml, or default'; diff --git a/packages/db/migrations/multi_household_architecture.sql b/packages/db/migrations/multi_household_architecture.sql new file mode 100644 index 0000000..b64a567 --- /dev/null +++ b/packages/db/migrations/multi_household_architecture.sql @@ -0,0 +1,397 @@ +-- ============================================================================ +-- Multi-Household & Multi-Store Architecture Migration +-- ============================================================================ +-- This migration transforms the single-list app into a multi-tenant system +-- supporting multiple households, each with multiple stores. +-- +-- IMPORTANT: Backup your database before running this migration! +-- pg_dump grocery_list > backup_$(date +%Y%m%d).sql +-- +-- Migration Strategy: +-- 1. Create new tables +-- 2. Create "Main Household" for existing users +-- 3. Migrate existing data to new structure +-- 4. Update roles (keep users.role for system admin) +-- 5. Verify data integrity +-- 6. (Manual step) Drop old tables after verification +-- ============================================================================ + +BEGIN; + +-- ============================================================================ +-- STEP 1: CREATE NEW TABLES +-- ============================================================================ + +-- Households table +CREATE TABLE IF NOT EXISTS households ( + id SERIAL PRIMARY KEY, + name VARCHAR(100) NOT NULL, + created_at TIMESTAMP DEFAULT NOW(), + created_by INTEGER REFERENCES users(id) ON DELETE SET NULL, + invite_code VARCHAR(20) UNIQUE NOT NULL, + code_expires_at TIMESTAMP +); + +CREATE INDEX idx_households_invite_code ON households(invite_code); +COMMENT ON TABLE households IS 'Household groups (families, roommates, etc.)'; +COMMENT ON COLUMN households.invite_code IS 'Unique code for inviting users to join household'; + +-- Store types table +CREATE TABLE IF NOT EXISTS stores ( + id SERIAL PRIMARY KEY, + name VARCHAR(50) NOT NULL UNIQUE, + default_zones JSONB, + created_at TIMESTAMP DEFAULT NOW() +); + +COMMENT ON TABLE stores IS 'Store types/chains (Costco, Target, Walmart, etc.)'; +COMMENT ON COLUMN stores.default_zones IS 'JSON array of default zone names for this store type'; + +-- User-Household membership with per-household roles +CREATE TABLE IF NOT EXISTS household_members ( + id SERIAL PRIMARY KEY, + household_id INTEGER REFERENCES households(id) ON DELETE CASCADE, + user_id INTEGER REFERENCES users(id) ON DELETE CASCADE, + role VARCHAR(20) NOT NULL CHECK (role IN ('admin', 'user')), + joined_at TIMESTAMP DEFAULT NOW(), + UNIQUE(household_id, user_id) +); + +CREATE INDEX idx_household_members_user ON household_members(user_id); +CREATE INDEX idx_household_members_household ON household_members(household_id); +COMMENT ON TABLE household_members IS 'User membership in households with per-household roles'; +COMMENT ON COLUMN household_members.role IS 'admin: full control, user: standard member'; + +-- Household-Store relationship +CREATE TABLE IF NOT EXISTS household_stores ( + id SERIAL PRIMARY KEY, + household_id INTEGER REFERENCES households(id) ON DELETE CASCADE, + store_id INTEGER REFERENCES stores(id) ON DELETE CASCADE, + is_default BOOLEAN DEFAULT FALSE, + added_at TIMESTAMP DEFAULT NOW(), + UNIQUE(household_id, store_id) +); + +CREATE INDEX idx_household_stores_household ON household_stores(household_id); +COMMENT ON TABLE household_stores IS 'Which stores each household shops at'; + +-- Master item catalog (shared across all households) +CREATE TABLE IF NOT EXISTS items ( + id SERIAL PRIMARY KEY, + name VARCHAR(255) NOT NULL UNIQUE, + default_image BYTEA, + default_image_mime_type VARCHAR(50), + created_at TIMESTAMP DEFAULT NOW(), + usage_count INTEGER DEFAULT 0 +); + +CREATE INDEX idx_items_name ON items(name); +CREATE INDEX idx_items_usage_count ON items(usage_count DESC); +COMMENT ON TABLE items IS 'Master item catalog shared across all households'; +COMMENT ON COLUMN items.usage_count IS 'Popularity metric for suggestions'; + +-- Household-specific grocery lists (per store) +CREATE TABLE IF NOT EXISTS household_lists ( + id SERIAL PRIMARY KEY, + household_id INTEGER REFERENCES households(id) ON DELETE CASCADE, + store_id INTEGER REFERENCES stores(id) ON DELETE CASCADE, + item_id INTEGER REFERENCES items(id) ON DELETE CASCADE, + quantity INTEGER NOT NULL DEFAULT 1, + bought BOOLEAN DEFAULT FALSE, + custom_image BYTEA, + custom_image_mime_type VARCHAR(50), + added_by INTEGER REFERENCES users(id) ON DELETE SET NULL, + modified_on TIMESTAMP DEFAULT NOW(), + UNIQUE(household_id, store_id, item_id) +); + +CREATE INDEX idx_household_lists_household_store ON household_lists(household_id, store_id); +CREATE INDEX idx_household_lists_bought ON household_lists(household_id, store_id, bought); +CREATE INDEX idx_household_lists_modified ON household_lists(modified_on DESC); +COMMENT ON TABLE household_lists IS 'Grocery lists scoped to household + store combination'; + +-- Household-specific item classifications (per store) +CREATE TABLE IF NOT EXISTS household_item_classifications ( + id SERIAL PRIMARY KEY, + household_id INTEGER REFERENCES households(id) ON DELETE CASCADE, + store_id INTEGER REFERENCES stores(id) ON DELETE CASCADE, + item_id INTEGER REFERENCES items(id) ON DELETE CASCADE, + item_type VARCHAR(50), + item_group VARCHAR(100), + zone VARCHAR(100), + confidence DECIMAL(3,2) DEFAULT 1.0 CHECK (confidence >= 0 AND confidence <= 1), + source VARCHAR(20) DEFAULT 'user' CHECK (source IN ('user', 'ml', 'default')), + created_at TIMESTAMP DEFAULT NOW(), + updated_at TIMESTAMP DEFAULT NOW(), + UNIQUE(household_id, store_id, item_id) +); + +CREATE INDEX idx_household_classifications ON household_item_classifications(household_id, store_id); +CREATE INDEX idx_household_classifications_type ON household_item_classifications(item_type); +CREATE INDEX idx_household_classifications_zone ON household_item_classifications(zone); +COMMENT ON TABLE household_item_classifications IS 'Item classifications scoped to household + store'; + +-- History tracking +CREATE TABLE IF NOT EXISTS household_list_history ( + id SERIAL PRIMARY KEY, + household_list_id INTEGER REFERENCES household_lists(id) ON DELETE CASCADE, + quantity INTEGER NOT NULL, + added_by INTEGER REFERENCES users(id) ON DELETE SET NULL, + added_on TIMESTAMP DEFAULT NOW() +); + +CREATE INDEX idx_household_history_list ON household_list_history(household_list_id); +CREATE INDEX idx_household_history_user ON household_list_history(added_by); +CREATE INDEX idx_household_history_date ON household_list_history(added_on DESC); +COMMENT ON TABLE household_list_history IS 'Tracks who added items and when'; + +-- ============================================================================ +-- STEP 2: CREATE DEFAULT HOUSEHOLD AND STORE +-- ============================================================================ + +-- Create default household for existing users +INSERT INTO households (name, created_by, invite_code) +SELECT + 'Main Household', + (SELECT id FROM users WHERE role = 'admin' LIMIT 1), -- First admin as creator + 'MAIN' || LPAD(FLOOR(RANDOM() * 1000000)::TEXT, 6, '0') -- Random 6-digit code +WHERE NOT EXISTS (SELECT 1 FROM households WHERE name = 'Main Household'); + +-- Create default Costco store +INSERT INTO stores (name, default_zones) +VALUES ( + 'Costco', + '{ + "zones": [ + "Entrance & Seasonal", + "Fresh Produce", + "Meat & Seafood", + "Dairy & Refrigerated", + "Deli & Prepared Foods", + "Bakery & Bread", + "Frozen Foods", + "Beverages", + "Snacks & Candy", + "Pantry & Dry Goods", + "Health & Beauty", + "Household & Cleaning", + "Other" + ] + }'::jsonb +) +ON CONFLICT (name) DO NOTHING; + +-- Link default household to default store +INSERT INTO household_stores (household_id, store_id, is_default) +SELECT + (SELECT id FROM households WHERE name = 'Main Household'), + (SELECT id FROM stores WHERE name = 'Costco'), + TRUE +WHERE NOT EXISTS ( + SELECT 1 FROM household_stores + WHERE household_id = (SELECT id FROM households WHERE name = 'Main Household') +); + +-- ============================================================================ +-- STEP 3: MIGRATE USERS TO HOUSEHOLD MEMBERS +-- ============================================================================ + +-- Add all existing users to Main Household +-- Old admins become household admins, others become standard users +INSERT INTO household_members (household_id, user_id, role) +SELECT + (SELECT id FROM households WHERE name = 'Main Household'), + id, + CASE + WHEN role = 'admin' THEN 'admin' + ELSE 'user' + END +FROM users +WHERE NOT EXISTS ( + SELECT 1 FROM household_members hm + WHERE hm.user_id = users.id + AND hm.household_id = (SELECT id FROM households WHERE name = 'Main Household') +); + +-- ============================================================================ +-- STEP 4: MIGRATE ITEMS TO MASTER CATALOG +-- ============================================================================ + +-- Extract unique items from grocery_list into master items table +INSERT INTO items (name, default_image, default_image_mime_type, created_at, usage_count) +SELECT + LOWER(TRIM(item_name)) as name, + item_image, + image_mime_type, + MIN(modified_on) as created_at, + COUNT(*) as usage_count +FROM grocery_list +WHERE NOT EXISTS ( + SELECT 1 FROM items WHERE LOWER(items.name) = LOWER(TRIM(grocery_list.item_name)) +) +GROUP BY LOWER(TRIM(item_name)), item_image, image_mime_type +ON CONFLICT (name) DO NOTHING; + +-- ============================================================================ +-- STEP 5: MIGRATE GROCERY_LIST TO HOUSEHOLD_LISTS +-- ============================================================================ + +-- Migrate current list to household_lists +INSERT INTO household_lists ( + household_id, + store_id, + item_id, + quantity, + bought, + custom_image, + custom_image_mime_type, + added_by, + modified_on +) +SELECT + (SELECT id FROM households WHERE name = 'Main Household'), + (SELECT id FROM stores WHERE name = 'Costco'), + i.id, + gl.quantity, + gl.bought, + CASE WHEN gl.item_image != i.default_image THEN gl.item_image ELSE NULL END, -- Only store if different + CASE WHEN gl.item_image != i.default_image THEN gl.image_mime_type ELSE NULL END, + gl.added_by, + gl.modified_on +FROM grocery_list gl +JOIN items i ON LOWER(i.name) = LOWER(TRIM(gl.item_name)) +WHERE NOT EXISTS ( + SELECT 1 FROM household_lists hl + WHERE hl.household_id = (SELECT id FROM households WHERE name = 'Main Household') + AND hl.store_id = (SELECT id FROM stores WHERE name = 'Costco') + AND hl.item_id = i.id +) +ON CONFLICT (household_id, store_id, item_id) DO NOTHING; + +-- ============================================================================ +-- STEP 6: MIGRATE ITEM_CLASSIFICATION TO HOUSEHOLD_ITEM_CLASSIFICATIONS +-- ============================================================================ + +-- Migrate classifications +INSERT INTO household_item_classifications ( + household_id, + store_id, + item_id, + item_type, + item_group, + zone, + confidence, + source, + created_at, + updated_at +) +SELECT + (SELECT id FROM households WHERE name = 'Main Household'), + (SELECT id FROM stores WHERE name = 'Costco'), + i.id, + ic.item_type, + ic.item_group, + ic.zone, + ic.confidence, + ic.source, + ic.created_at, + ic.updated_at +FROM item_classification ic +JOIN grocery_list gl ON ic.id = gl.id +JOIN items i ON LOWER(i.name) = LOWER(TRIM(gl.item_name)) +WHERE NOT EXISTS ( + SELECT 1 FROM household_item_classifications hic + WHERE hic.household_id = (SELECT id FROM households WHERE name = 'Main Household') + AND hic.store_id = (SELECT id FROM stores WHERE name = 'Costco') + AND hic.item_id = i.id +) +ON CONFLICT (household_id, store_id, item_id) DO NOTHING; + +-- ============================================================================ +-- STEP 7: MIGRATE GROCERY_HISTORY TO HOUSEHOLD_LIST_HISTORY +-- ============================================================================ + +-- Migrate history records +INSERT INTO household_list_history (household_list_id, quantity, added_by, added_on) +SELECT + hl.id, + gh.quantity, + gh.added_by, + gh.added_on +FROM grocery_history gh +JOIN grocery_list gl ON gh.list_item_id = gl.id +JOIN items i ON LOWER(i.name) = LOWER(TRIM(gl.item_name)) +JOIN household_lists hl ON hl.item_id = i.id + AND hl.household_id = (SELECT id FROM households WHERE name = 'Main Household') + AND hl.store_id = (SELECT id FROM stores WHERE name = 'Costco') +WHERE NOT EXISTS ( + SELECT 1 FROM household_list_history hlh + WHERE hlh.household_list_id = hl.id + AND hlh.added_by = gh.added_by + AND hlh.added_on = gh.added_on +); + +-- ============================================================================ +-- STEP 8: UPDATE USER ROLES (SYSTEM-WIDE) +-- ============================================================================ + +-- Update system roles: admin โ†’ system_admin, others โ†’ user +UPDATE users +SET role = 'system_admin' +WHERE role = 'admin'; + +UPDATE users +SET role = 'user' +WHERE role IN ('editor', 'viewer'); + +-- ============================================================================ +-- VERIFICATION QUERIES +-- ============================================================================ + +-- Run these to verify migration success: + +-- Check household created +-- SELECT * FROM households; + +-- Check all users added to household +-- SELECT u.username, u.role as system_role, hm.role as household_role +-- FROM users u +-- JOIN household_members hm ON u.id = hm.user_id +-- ORDER BY u.id; + +-- Check items migrated +-- SELECT COUNT(*) as total_items FROM items; +-- SELECT COUNT(*) as original_items FROM (SELECT DISTINCT item_name FROM grocery_list) sub; + +-- Check lists migrated +-- SELECT COUNT(*) as new_lists FROM household_lists; +-- SELECT COUNT(*) as old_lists FROM grocery_list; + +-- Check classifications migrated +-- SELECT COUNT(*) as new_classifications FROM household_item_classifications; +-- SELECT COUNT(*) as old_classifications FROM item_classification; + +-- Check history migrated +-- SELECT COUNT(*) as new_history FROM household_list_history; +-- SELECT COUNT(*) as old_history FROM grocery_history; + +-- ============================================================================ +-- MANUAL STEPS AFTER VERIFICATION +-- ============================================================================ + +-- After verifying data integrity, uncomment and run these to clean up: + +-- DROP TABLE IF EXISTS grocery_history CASCADE; +-- DROP TABLE IF EXISTS item_classification CASCADE; +-- DROP TABLE IF EXISTS grocery_list CASCADE; + +COMMIT; + +-- ============================================================================ +-- ROLLBACK (if something goes wrong) +-- ============================================================================ + +-- ROLLBACK; + +-- Then restore from backup: +-- psql -U your_user -d grocery_list < backup_YYYYMMDD.sql diff --git a/run-migration.bat b/run-migration.bat index 3b405e3..993cab1 100644 --- a/run-migration.bat +++ b/run-migration.bat @@ -1,80 +1,21 @@ @echo off -REM Multi-Household Migration Runner (Windows) -REM This script handles the complete migration process with safety checks +setlocal -setlocal enabledelayedexpansion - -REM Database configuration -set DB_USER=postgres -set DB_HOST=192.168.7.112 -set DB_NAME=grocery -set PGPASSWORD=Asdwed123A. - -set BACKUP_DIR=backend\migrations\backups -set TIMESTAMP=%date:~-4%%date:~-10,2%%date:~-7,2%_%time:~0,2%%time:~3,2%%time:~6,2% -set TIMESTAMP=%TIMESTAMP: =0% -set BACKUP_FILE=%BACKUP_DIR%\backup_%TIMESTAMP%.sql - -echo ================================================ -echo Multi-Household Architecture Migration -echo ================================================ -echo. - -REM Create backup directory -if not exist "%BACKUP_DIR%" mkdir "%BACKUP_DIR%" - -REM Step 1: Backup (SKIPPED - using database template copy) -echo [1/5] Backup: SKIPPED (using 'grocery' database copy) -echo. - -REM Step 2: Show current stats -echo [2/5] Current database statistics: -psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% -c "SELECT 'Users' as table_name, COUNT(*) as count FROM users UNION ALL SELECT 'Grocery Items', COUNT(*) FROM grocery_list UNION ALL SELECT 'Classifications', COUNT(*) FROM item_classification UNION ALL SELECT 'History Records', COUNT(*) FROM grocery_history;" -echo. - -REM Step 3: Confirm -echo [3/5] Ready to run migration -echo Database: %DB_NAME% on %DB_HOST% -echo Backup: %BACKUP_FILE% -echo. -set /p CONFIRM="Continue with migration? (yes/no): " -if /i not "%CONFIRM%"=="yes" ( - echo Migration cancelled. - exit /b 0 +if "%DATABASE_URL%"=="" ( + echo DATABASE_URL is required. Aborting. + exit /b 1 ) -echo. -REM Step 4: Run migration -echo [4/5] Running migration script... -psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% -f backend\migrations\multi_household_architecture.sql -if %errorlevel% neq 0 ( - echo [ERROR] Migration failed! Rolling back... - echo Restoring from backup: %BACKUP_FILE% - psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% < "%BACKUP_FILE%" - exit /b 1 -) -echo [OK] Migration completed successfully -echo. +echo Checking migration status... +call npm run db:migrate:status +if errorlevel 1 exit /b 1 -REM Step 5: Verification -echo [5/5] Verifying migration... -psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% -c "SELECT id, name, invite_code FROM households;" -psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% -c "SELECT u.id, u.username, u.role as system_role, hm.role as household_role FROM users u LEFT JOIN household_members hm ON u.id = hm.user_id ORDER BY u.id LIMIT 10;" -psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% -c "SELECT 'Items' as metric, COUNT(*)::text as count FROM items UNION ALL SELECT 'Household Lists', COUNT(*)::text FROM household_lists UNION ALL SELECT 'Classifications', COUNT(*)::text FROM household_item_classifications UNION ALL SELECT 'History Records', COUNT(*)::text FROM household_list_history;" -echo. +echo Applying pending migrations... +call npm run db:migrate +if errorlevel 1 exit /b 1 -echo ================================================ -echo Migration Complete! -echo ================================================ -echo. -echo Next Steps: -echo 1. Review verification results above -echo 2. Test the application -echo 3. If issues found, rollback with: -echo psql -h %DB_HOST% -U %DB_USER% -d %DB_NAME% ^< %BACKUP_FILE% -echo 4. If successful, proceed to Sprint 2 (Backend API) -echo. -echo Backup location: %BACKUP_FILE% -echo. +echo Final migration status... +call npm run db:migrate:status +if errorlevel 1 exit /b 1 -pause +echo Done. diff --git a/run-migration.sh b/run-migration.sh index 8089521..48f0d90 100644 --- a/run-migration.sh +++ b/run-migration.sh @@ -1,146 +1,24 @@ #!/bin/bash -# Multi-Household Migration Runner -# This script handles the complete migration process with safety checks +set -euo pipefail -set -e # Exit on error - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -# Database configuration (from .env) -DB_USER="postgres" -DB_HOST="192.168.7.112" -DB_NAME="grocery" -export PGPASSWORD="Asdwed123A." - -BACKUP_DIR="./backend/migrations/backups" -TIMESTAMP=$(date +%Y%m%d_%H%M%S) -BACKUP_FILE="${BACKUP_DIR}/backup_${TIMESTAMP}.sql" - -echo -e "${BLUE}โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—${NC}" -echo -e "${BLUE}โ•‘ Multi-Household Architecture Migration โ•‘${NC}" -echo -e "${BLUE}โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" -echo "" - -# Create backup directory if it doesn't exist -mkdir -p "$BACKUP_DIR" - -# Step 1: Backup -echo -e "${YELLOW}[1/5] Creating database backup...${NC}" -pg_dump -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" > "$BACKUP_FILE" -if [ $? -eq 0 ]; then - echo -e "${GREEN}โœ“ Backup created: $BACKUP_FILE${NC}" -else - echo -e "${RED}โœ— Backup failed!${NC}" - exit 1 -fi -echo "" - -# Step 2: Show current stats -echo -e "${YELLOW}[2/5] Current database statistics:${NC}" -psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -c " -SELECT - 'Users' as table_name, COUNT(*) as count FROM users -UNION ALL -SELECT 'Grocery Items', COUNT(*) FROM grocery_list -UNION ALL -SELECT 'Classifications', COUNT(*) FROM item_classification -UNION ALL -SELECT 'History Records', COUNT(*) FROM grocery_history; -" -echo "" - -# Step 3: Confirm -echo -e "${YELLOW}[3/5] Ready to run migration${NC}" -echo -e "Database: ${BLUE}$DB_NAME${NC} on ${BLUE}$DB_HOST${NC}" -echo -e "Backup: ${GREEN}$BACKUP_FILE${NC}" -echo "" -read -p "Continue with migration? (yes/no): " -r -echo "" -if [[ ! $REPLY =~ ^[Yy]es$ ]]; then - echo -e "${RED}Migration cancelled.${NC}" - exit 1 +if ! command -v node >/dev/null 2>&1; then + echo "node is required." + exit 1 fi -# Step 4: Run migration -echo -e "${YELLOW}[4/5] Running migration script...${NC}" -psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -f backend/migrations/multi_household_architecture.sql -if [ $? -eq 0 ]; then - echo -e "${GREEN}โœ“ Migration completed successfully${NC}" -else - echo -e "${RED}โœ— Migration failed! Rolling back...${NC}" - echo -e "${YELLOW}Restoring from backup: $BACKUP_FILE${NC}" - psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" < "$BACKUP_FILE" - exit 1 +if [ -z "${DATABASE_URL:-}" ]; then + echo "DATABASE_URL is required. Aborting." + exit 1 fi -echo "" -# Step 5: Verification -echo -e "${YELLOW}[5/5] Verifying migration...${NC}" -psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" << 'EOF' -\echo '' -\echo '=== Household Created ===' -SELECT id, name, invite_code FROM households; +echo "Checking migration status..." +npm run db:migrate:status -\echo '' -\echo '=== User Roles ===' -SELECT u.id, u.username, u.role as system_role, hm.role as household_role -FROM users u -LEFT JOIN household_members hm ON u.id = hm.user_id -ORDER BY u.id -LIMIT 10; +echo "Applying pending migrations..." +npm run db:migrate -\echo '' -\echo '=== Migration Counts ===' -SELECT - 'Items (Master Catalog)' as metric, COUNT(*)::text as count FROM items -UNION ALL -SELECT 'Household Lists', COUNT(*)::text FROM household_lists -UNION ALL -SELECT 'Classifications', COUNT(*)::text FROM household_item_classifications -UNION ALL -SELECT 'History Records', COUNT(*)::text FROM household_list_history -UNION ALL -SELECT 'Household Members', COUNT(*)::text FROM household_members -UNION ALL -SELECT 'Stores', COUNT(*)::text FROM stores; +echo "Final migration status..." +npm run db:migrate:status -\echo '' -\echo '=== Data Integrity Checks ===' -\echo 'Users without household membership (should be 0):' -SELECT COUNT(*) FROM users u -LEFT JOIN household_members hm ON u.id = hm.user_id -WHERE hm.id IS NULL; - -\echo '' -\echo 'Lists without valid items (should be 0):' -SELECT COUNT(*) FROM household_lists hl -LEFT JOIN items i ON hl.item_id = i.id -WHERE i.id IS NULL; - -\echo '' -\echo 'History without valid lists (should be 0):' -SELECT COUNT(*) FROM household_list_history hlh -LEFT JOIN household_lists hl ON hlh.household_list_id = hl.id -WHERE hl.id IS NULL; -EOF - -echo "" -echo -e "${GREEN}โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—${NC}" -echo -e "${GREEN}โ•‘ Migration Complete! โ•‘${NC}" -echo -e "${GREEN}โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" -echo "" -echo -e "${BLUE}Next Steps:${NC}" -echo -e "1. Review verification results above" -echo -e "2. Test the application" -echo -e "3. If issues found, rollback with:" -echo -e " ${YELLOW}psql -h $DB_HOST -U $DB_USER -d $DB_NAME < $BACKUP_FILE${NC}" -echo -e "4. If successful, proceed to Sprint 2 (Backend API)" -echo "" -echo -e "${YELLOW}Backup location: $BACKUP_FILE${NC}" -echo "" +echo "Done." diff --git a/scripts/db-migrate-common.js b/scripts/db-migrate-common.js new file mode 100644 index 0000000..efe26c1 --- /dev/null +++ b/scripts/db-migrate-common.js @@ -0,0 +1,108 @@ +"use strict"; + +const fs = require("fs"); +const path = require("path"); +const { spawnSync } = require("child_process"); + +const migrationsDir = path.resolve( + __dirname, + "..", + "packages", + "db", + "migrations" +); + +function ensureDatabaseUrl() { + const databaseUrl = process.env.DATABASE_URL; + if (!databaseUrl) { + throw new Error("DATABASE_URL is required."); + } + return databaseUrl; +} + +function ensurePsql() { + const result = spawnSync("psql", ["--version"], { stdio: "pipe" }); + if (result.error || result.status !== 0) { + throw new Error("psql executable was not found in PATH."); + } +} + +function ensureMigrationsDir() { + if (!fs.existsSync(migrationsDir)) { + throw new Error(`Migrations directory not found: ${migrationsDir}`); + } +} + +function getMigrationFiles() { + ensureMigrationsDir(); + return fs + .readdirSync(migrationsDir) + .filter((file) => file.endsWith(".sql")) + .sort((a, b) => a.localeCompare(b)); +} + +function runPsql(databaseUrl, args) { + const result = spawnSync("psql", [databaseUrl, ...args], { + stdio: "pipe", + encoding: "utf8", + }); + if (result.status !== 0) { + const stderr = (result.stderr || "").trim(); + const stdout = (result.stdout || "").trim(); + const details = [stderr, stdout].filter(Boolean).join("\n"); + throw new Error(details || "psql command failed"); + } + return result.stdout || ""; +} + +function escapeSqlLiteral(value) { + return value.replace(/'/g, "''"); +} + +function ensureSchemaMigrationsTable(databaseUrl) { + runPsql(databaseUrl, [ + "-v", + "ON_ERROR_STOP=1", + "-c", + "CREATE TABLE IF NOT EXISTS schema_migrations (filename TEXT PRIMARY KEY, applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW());", + ]); +} + +function getAppliedMigrations(databaseUrl) { + const output = runPsql(databaseUrl, [ + "-At", + "-v", + "ON_ERROR_STOP=1", + "-c", + "SELECT filename FROM schema_migrations ORDER BY filename ASC;", + ]); + return new Set( + output + .split(/\r?\n/) + .map((line) => line.trim()) + .filter(Boolean) + ); +} + +function applyMigration(databaseUrl, filename) { + const fullPath = path.join(migrationsDir, filename); + runPsql(databaseUrl, ["-v", "ON_ERROR_STOP=1", "-f", fullPath]); + runPsql(databaseUrl, [ + "-v", + "ON_ERROR_STOP=1", + "-c", + `INSERT INTO schema_migrations (filename) VALUES ('${escapeSqlLiteral( + filename + )}') ON CONFLICT DO NOTHING;`, + ]); +} + +module.exports = { + applyMigration, + ensureDatabaseUrl, + ensurePsql, + ensureSchemaMigrationsTable, + getAppliedMigrations, + getMigrationFiles, + migrationsDir, +}; diff --git a/scripts/db-migrate-status.js b/scripts/db-migrate-status.js new file mode 100644 index 0000000..135e0ab --- /dev/null +++ b/scripts/db-migrate-status.js @@ -0,0 +1,42 @@ +"use strict"; + +const { + ensureDatabaseUrl, + ensurePsql, + ensureSchemaMigrationsTable, + getAppliedMigrations, + getMigrationFiles, +} = require("./db-migrate-common"); + +function main() { + if (process.argv.includes("--help")) { + console.log("Usage: npm run db:migrate:status"); + process.exit(0); + } + + const databaseUrl = ensureDatabaseUrl(); + ensurePsql(); + ensureSchemaMigrationsTable(databaseUrl); + + const files = getMigrationFiles(); + const applied = getAppliedMigrations(databaseUrl); + + let pendingCount = 0; + for (const file of files) { + const status = applied.has(file) ? "APPLIED" : "PENDING"; + if (status === "PENDING") pendingCount += 1; + console.log(`${status} ${file}`); + } + + console.log(""); + console.log(`Total: ${files.length}`); + console.log(`Applied: ${files.length - pendingCount}`); + console.log(`Pending: ${pendingCount}`); +} + +try { + main(); +} catch (error) { + console.error(error.message); + process.exit(1); +} diff --git a/scripts/db-migrate-verify.js b/scripts/db-migrate-verify.js new file mode 100644 index 0000000..06f636d --- /dev/null +++ b/scripts/db-migrate-verify.js @@ -0,0 +1,41 @@ +"use strict"; + +const { + ensureDatabaseUrl, + ensurePsql, + ensureSchemaMigrationsTable, + getAppliedMigrations, + getMigrationFiles, +} = require("./db-migrate-common"); + +function main() { + if (process.argv.includes("--help")) { + console.log("Usage: npm run db:migrate:verify"); + process.exit(0); + } + + const databaseUrl = ensureDatabaseUrl(); + ensurePsql(); + ensureSchemaMigrationsTable(databaseUrl); + + const files = getMigrationFiles(); + const applied = getAppliedMigrations(databaseUrl); + const pending = files.filter((file) => !applied.has(file)); + + if (pending.length > 0) { + console.error("Pending migrations detected:"); + for (const file of pending) { + console.error(`- ${file}`); + } + process.exit(1); + } + + console.log("Migration verification passed. No pending migrations."); +} + +try { + main(); +} catch (error) { + console.error(error.message); + process.exit(1); +} diff --git a/scripts/db-migrate.js b/scripts/db-migrate.js new file mode 100644 index 0000000..d9e2b9a --- /dev/null +++ b/scripts/db-migrate.js @@ -0,0 +1,44 @@ +"use strict"; + +const { + applyMigration, + ensureDatabaseUrl, + ensurePsql, + ensureSchemaMigrationsTable, + getAppliedMigrations, + getMigrationFiles, +} = require("./db-migrate-common"); + +function main() { + if (process.argv.includes("--help")) { + console.log("Usage: npm run db:migrate"); + process.exit(0); + } + + const databaseUrl = ensureDatabaseUrl(); + ensurePsql(); + ensureSchemaMigrationsTable(databaseUrl); + + const files = getMigrationFiles(); + const applied = getAppliedMigrations(databaseUrl); + const pending = files.filter((file) => !applied.has(file)); + + if (pending.length === 0) { + console.log("No pending migrations."); + return; + } + + for (const file of pending) { + console.log(`Applying: ${file}`); + applyMigration(databaseUrl, file); + } + + console.log(`Applied ${pending.length} migration(s).`); +} + +try { + main(); +} catch (error) { + console.error(error.message); + process.exit(1); +}