create AI_template

2025-11-18 03:15:04 +02:00
parent f8c854e3e1
commit 3f4a98d42d
17 changed files with 932 additions and 0 deletions
--- a/docs/backend/api-design.md
+++ b/docs/backend/api-design.md
@@ -0,0 +1,57 @@
+# Backend: API Design
+
+---
+**Last Updated:** 2025-01-17  
+**Phase:** Phase 0 (Planning)  
+**Status:** Draft — finalize in Phase 1  
+**Owner:** Backend Architect  
+**References:**
+- `/docs/backend/architecture.md`
+- `/docs/backend/payment-flow.md`
+- `/docs/backend/security.md`
+---
+
+## 1. Principles
+- REST with versioning (`/api/v1`), consistent envelope and errors.
+- Tenant-scoped resources; role-based authorization.
+- Idempotent ingestion/webhook endpoints; trace IDs for debugging.
+
+## 2. Core Resources (high-level)
+- `/auth` — login, tenant context, token refresh.
+- `/tenants` — tenant profile, roles, invites.
+- `/integrations` — connect sources (OAuth2/webhooks), view health, rotate secrets, webhook endpoints.
+- `/records` — list/filter, detail (raw + normalized + reasoning), bulk actions, reprocess.
+- `/rules` — CRUD, enable/disable, test/preview, stats.
+- `/processing` — trigger re-evaluation (rules → embeddings → LLM), inspect outcomes.
+- `/approvals` — approval queue, approve/override, optional rule creation.
+- `/reports` — dashboards/summary generation, export status/download.
+- `/billing` — session/portal links, subscription status (webhook-driven).
+- `/events` — read-only event log feed for downstream agents.
+- `/files/attachments` — uploads/metadata (if exposed).
+
+## 3. Payments/Billing (Provider-agnostic)
+- `POST /billing/session` — create checkout session for subscription/plan change.
+- `GET /billing/status` — current subscription status (webhook-driven source of truth).
+- `GET /billing/portal` — customer portal link for payment method/plan management.
+- `POST /billing/webhooks/provider` — verify signature; idempotent subscription updates; log billing events.
+
+## 4. Ingestion/Webhooks
+- Receivers for external providers (e.g., `/integrations/webhooks/{provider}`): verify signature, dedupe, enqueue to BullMQ; emit `INGESTED` with `source_agent`.
+
+## 5. Processing & Approvals
+- `POST /processing/run` — trigger re-evaluation for a scoped set (source/date/status).
+- `POST /approvals/{id}/approve` — accept decision; log `APPROVED`.
+- `POST /approvals/{id}/override` — override outcome; optional rule payload; log `APPROVED` and `RULE_CREATED` when applicable.
+
+## 6. Events
+- `GET /events` — read-only feed (paginated) for downstream agents/clients; filters: tenant, type, time range, source_agent.
+
+## 7. Response Envelope (example)
+```json
+{
+  "data": { ... },
+  "error": null,
+  "meta": { "traceId": "..." }
+}
+```
+Errors: `data = null`, `error` with code/message/context.
--- a/docs/backend/architecture.md
+++ b/docs/backend/architecture.md
@@ -0,0 +1,63 @@
+# Backend: Architecture (Recommendations)
+
+---
+**Last Updated:** 2025-01-17  
+**Phase:** Phase 0 (Planning)  
+**Status:** Draft — finalize in Phase 1  
+**Owner:** Backend Architect  
+**References:**
+- `/docs/project-overview.md`
+- `/docs/backend/api-design.md`
+- `/docs/backend/security.md`
+- `/docs/backend/payment-flow.md`
+---
+
+> Recommendations for Phase 0. Lock decisions in Phase 1.
+
+## 1. Approach & Stack
+- Style: modular monolith with clear modules; containerized.
+- Language/Runtime: Node.js (LTS) + TypeScript.
+- Framework: Express or Fastify with modular structure and DI where helpful.
+- DB: Postgres (managed: Supabase/RDS). Vector: `pgvector` for embeddings.
+- Queue: BullMQ (Redis) for ingestion, categorization, notifications.
+- Auth: Clerk/Auth.js (or equivalent) with tenant-aware RBAC.
+- Payments: provider-hosted subscriptions; no raw card data stored.
+- LLM: OpenAI API (or equivalent) via single helper (e.g., `callLLM()`), configurable model/params.
+
+## 2. Modules (logical)
+- `auth` — tenants, users, roles, sessions.
+- `integrations` — external connectors, OAuth2, webhooks, connection health.
+- `records` — normalized record store, statuses, `reasoning_trace` JSONB.
+- `rules` — rule definitions, evaluation order, testing, hit stats.
+- `processing` — pipeline: rule engine → embeddings similarity → LLM fallback; writes `PROCESSED`, updates records.
+- `approvals` — queues for human review, overrides, optional rule creation; logs `TX_APPROVED`/`RULE_CREATED` with `source_agent`.
+- `reports` — dashboards/exports, history.
+- `billing` — provider checkout/portal, webhooks, plan enforcement per tenant.
+- `events` — audit/event log (`EventLog`), read-only `/api/events` for downstream agents.
+- `files/receipts` — attachment storage metadata (`Receipt` with file URL/mime).
+
+## 3. API Layers
+- HTTP API (REST) with versioning (`/api/v1`).
+- Service layer for business logic and transactions.
+- Repositories for data access; use migrations for schema evolution.
+
+## 4. Infrastructure & Ops
+- Environments: dev/stage/prod; Docker images; CI/CD.
+- Observability: structured logging, metrics, tracing; dead-letter queues for failed jobs.
+- Secrets management per environment; rotate webhook/LLM/payment provider secrets.
+
+## 5. Data & Schema Notes
+- Records: store raw payload + normalized fields + `reasoning_trace` JSONB (model, rationale, confidence, source).
+- EventLog: include `source_agent` (default `balance`) and payload for auditability; ensure filters by tenant/time/type.
+- Embeddings: table keyed by transaction/merchant strings to support similarity search; index with `pgvector`.
+- Multi-tenant: all core tables carry `tenantId` and enforce scoped queries; `User` role per tenant.
+
+## 6. Payment & Messaging (High-Level)
+- Payment provider: initiate sessions via backend; handle webhooks idempotently; map provider status to internal billing/subscription states; update tenant access.
+- Notifications: optional email/webhook callbacks to surface ingestion/categorization failures; keep out of PII exposure.
+
+## 7. Queues (BullMQ)
+- `records:ingest` — normalize webhook payloads, write `Record`, emit `INGESTED`.
+- `records:process` — rule engine → embeddings similarity → LLM fallback; emit `PROCESSED` with `reasoning_trace`.
+- `reports:generate` — build P&L/exports, emit `REPORT_GENERATED`.
+- Dead-letter queues per stream; retries with backoff; idempotent handlers keyed by external event IDs.
--- a/docs/backend/overview.md
+++ b/docs/backend/overview.md
@@ -0,0 +1,33 @@
+# Backend: Overview
+
+---
+**Last Updated:** 2025-01-17  
+**Phase:** Phase 0 (Planning)  
+**Status:** Approved  
+**Owner:** Backend Architect  
+**References:**
+- `/docs/project-overview.md`
+- `/docs/backend/architecture.md`
+- `/docs/backend/payment-flow.md`
+---
+
+## 1. Role of Backend
+- Own business logic for ingestion, processing/classification (rules + embeddings + LLM fallback), approvals, reporting, billing, and audit.
+- Integrate safely with external providers (OAuth2/webhooks, payment provider, LLM provider) and expose consistent APIs + events.
+- Enforce security: tenant isolation, RBAC, webhook verification, event/audit logging.
+
+## 2. Main Domain Areas
+- **Auth & Tenants:** authentication/authorization, roles, tenant-scoped access.
+- **Integrations:** external providers via OAuth2/webhooks; connection health.
+- **Records:** normalized feeds, statuses (ingested, processed, needs_approval, approved, failed), `reasoning_trace` JSONB.
+- **Rules & Processing:** rules engine, embeddings similarity, LLM fallback; logging with `source_agent`.
+- **Approvals:** human-in-the-loop decisions, overrides, optional rule creation; audit trail.
+- **Reports & Exports:** dashboards/summaries with export history.
+- **Billing:** provider-hosted subscriptions, tenant-scoped access control, webhooks.
+- **Events:** `/api/events` feed for downstream agents and internal observability.
+
+## 3. Integrations
+- **External data providers:** OAuth2 + webhooks; signatures/verification; idempotent writes via workers.
+- **Payment provider:** subscriptions, checkout/portal; webhooks for lifecycle events.
+- **LLM provider:** OpenAI API via single helper; configurable model.
+- **Queues:** BullMQ (Redis) for ingestion/categorization/notifications.
--- a/docs/backend/payment-flow.md
+++ b/docs/backend/payment-flow.md
@@ -0,0 +1,45 @@
+# Payment & Billing Flow (Provider‑agnostic)
+
+---
+**Last Updated:** 2025-01-17  
+**Phase:** Phase 0 (Planning)  
+**Status:** Draft — finalize in Phase 1  
+**Owner:** Backend Architect  
+**References:**
+- `/docs/project-overview.md`
+- `/docs/backend/architecture.md`
+- `/docs/backend/api-design.md`
+- `/docs/backend/security.md`
+---
+
+Single source of truth for subscription billing with an external payment provider. No raw card data is handled by our app.
+
+## 1) High-Level Flow
+1. Frontend calls backend to create a Checkout/Portal Session (`POST /billing/session`) with tenant/context.
+2. Backend creates/updates the Customer (if needed) and Session at the payment provider; returns `sessionUrl`.
+3. User completes checkout on provider‑hosted page.
+4. Payment provider sends webhooks (e.g., subscription created/updated/canceled).
+5. Backend verifies signatures, updates subscription state in DB, and logs events (`BILLING_UPDATED`).
+6. Frontend reads subscription status from backend (`/billing/status`) — webhook‑driven source of truth.
+
+## 2) Required Endpoints
+- `POST /billing/session` — init checkout/plan change. Request: tenant ID inferred from auth; plan/tier; success/cancel URLs. Response: `sessionUrl`.
+- `GET /billing/status` — return subscription status, current plan, renewal/cancel info (cached from webhook‑driven DB state).
+- `POST /billing/webhooks/provider` — verify signature; idempotent upsert of subscription state; update tenant access.
+
+## 3) Data Model Notes
+- Store provider customer ID, subscription ID, price/plan, status, current period dates, cancel_at/canceled_at, seats (if applicable).
+- All writes must be idempotent (webhook retries). Use provider event IDs for dedupe.
+- Permissions are enforced per tenant based on subscription status.
+
+## 4) States & Mapping (common)
+- `trialing`, `active`, `past_due`, `canceled`, `incomplete/incomplete_expired`, `unpaid` → map to internal states that drive access. Deny critical actions when subscription is inactive.
+
+## 5) Security & Compliance
+- Verify webhook signatures with provider secret; reject if invalid; log failures with trace IDs.
+- Do not log sensitive payloads; store only necessary billing identifiers.
+- Expose no card data; rely on provider‑hosted UIs (Checkout/Portal) for payment method management.
+
+## 6) Frontend UX Notes
+- Use provider trust cues and clear statuses after redirect: success/pending/fail with recovery (retry checkout, contact support).
+- All status displays must read from backend (webhook‑driven), not query params.
--- a/docs/backend/security.md
+++ b/docs/backend/security.md
@@ -0,0 +1,51 @@
+# Backend: Security (Template)
+
+---
+**Last Updated:** 2025-01-17  
+**Phase:** Phase 0 (Planning)  
+**Status:** Draft — finalize in Phase 1  
+**Owner:** Backend Architect, Security  
+**References:**
+- `/docs/backend/architecture.md`
+- `/docs/backend/payment-flow.md`
+- `/docs/backend/api-design.md`
+---
+
+## 1. Goals
+- Protect sensitive data and ensure tenant isolation.
+- Secure integrations (webhooks/OAuth) and billing.
+- Maintain auditability for all automated and manual actions.
+
+## 2. Authentication & Authorization
+- Use managed auth (Clerk/Auth.js or equivalent) with secure sessions/tokens.
+- Enforce tenant-scoped RBAC; validate tenant context on every request.
+- Rate limit auth endpoints; protect refresh flows.
+
+## 3. Data Protection
+- TLS everywhere; secure headers.
+- Least-privilege DB roles; encrypt secrets at rest; rotate keys.
+- Avoid logging sensitive payloads (bank data, webhook bodies, LLM prompts/responses beyond trace references).
+- Store `reasoning_trace` JSONB with care; avoid PII in traces.
+
+## 4. Integrations & Webhooks
+- Verify signatures for all inbound webhooks (external providers, payment provider); reject invalid; log with trace IDs.
+- Idempotent processing with event ID dedupe; dead-letter queue for failures.
+- Expose webhook URLs only over HTTPS; regenerate secrets on rotation.
+
+## 5. Payments (provider‑agnostic)
+- Use provider‑hosted Checkout/Portal; never handle raw card data.
+- Store minimal billing identifiers (customer/subscription IDs, status, timestamps, plan/price).
+- Enforce access based on subscription state; deny actions when inactive.
+
+## 6. LLM Safety
+- All LLM calls go through a single helper; centralize redaction, logging, and parameter control.
+- Strip/obfuscate sensitive fields before sending to LLM; log only references in traces.
+
+## 7. Audit & Events
+- Log domain events to `EventLog` with `source_agent`; include user ID, tenant, timestamps, and relevant context.
+- Provide read-only `/api/events` with pagination and filtering; protect by tenant/role.
+
+## 8. Common Controls
+- OWASP Top 10 coverage (SQLi, XSS, CSRF, IDOR, SSRF, etc.).
+- Input validation on all external inputs; use allowlists for redirects/URLs.
+- Backup/restore plan and monitoring/alerting for infra and queues.