# Epic TW-98 — Adpilot API Gateway (`api.lucos.com`)

## Context

Adpilot is a repo-understanding system. Users connect GitHub or Bitbucket repositories; the system indexes all commit history and code into a vector DB (Qdrant). Users can then ask natural-language questions and get RAG-powered answers with code-writing assistance.

`api.lucos.com` is the **API Gateway** that sits in front of two upstream services:

| Upstream | Port | Purpose |
|---|---|---|
| `rag-query-service` | 6005 | Vector search, RAG queries |
| `repo-sync` | 6001 | Repo indexing and sync jobs |

The gateway handles auth, org/team context, plan entitlements, rate limiting, provider OAuth flows, and trusted-header injection before proxying to upstreams.

**Stack:** Node.js, Express 5, MongoDB (Mongoose), Redis, JWT, AES-256-GCM

---

## Ticket Status Overview

| Ticket | Title | Status |
|---|---|---|
| TW-116 | Scaffold Express gateway | ✅ Done |
| TW-115 | JWT auth + org/team context | ✅ Done |
| TW-114 | Proxy routes to RAG + repo-sync | ✅ Done |
| TW-117 | SSE streaming proxy | ✅ Done |
| TW-118 | Entitlement middleware | ✅ Done |
| TW-119 | Rate limiting + CORS | ✅ Done |
| TW-132 | Data models (orgs, teams, plans, subscriptions) | ✅ Done |
| TW-133 | Repo limit gate | ✅ Done |
| TW-134 | Model-family gate | ✅ Done |
| TW-136 | GitHub App installation flow | ✅ Done |
| TW-137 | AES-256-GCM token encryption | ✅ Done |
| TW-138 | Bitbucket OAuth flow | ✅ Done |
| TW-139 | Provider repo discovery & cache | ✅ Done |
| TW-140 | Provider disconnect & revoke | ✅ Done |
| TW-141 | Provider connection security tests | ✅ Done |
| TW-142 | Import repos to indexing with plan gate | ✅ Done |
| TW-120 | Gateway security & integration tests | ✅ Done |
| TW-135 | Pro → Free downgrade repo policy (product decision) | ⏳ Product decision needed |

---

## Completed Tickets

---

### TW-116 — Scaffold API Gateway

**What it asked for:** Set up the Express app skeleton with health check, Docker support, and environment config.

**What was done:**

- `src/app.js` — Express app with CORS, Helmet, compression, cookie-parser, JSON body parsing, tracing middleware, `/healthz` endpoint
- `src/middlewares/tracing.middleware.js` — generates a `uuid v4` `X-Request-Id` per request (or passes through if already present), emits structured JSON logs on `res.finish` with `{ requestId, method, path, status, latencyMs, userId, orgId, planCode }`
- `server.js` — startup connects MongoDB + Redis (Redis failure is non-fatal/warn)
- `Dockerfile` — Node 20 Alpine, `PORT=8080`, `NODE_ENV=production`
- `docker-compose.yml` — gateway on 8080, MongoDB, Redis; upstream services commented out pending image availability
- `.env.example` — complete env var reference
- `SETUP.md` — API table, curl examples, env var reference

**Key design:** `X-Request-Id` is forwarded to all upstreams so traces can be correlated across services.

---

### TW-115 — JWT Auth Middleware with Org/Team Context

**What it asked for:** Every authenticated request must carry full org/team/plan context so downstream middlewares and upstreams don't need to re-query.

**What was done:**

- `src/middlewares/auth.middleware.js` — rewritten as async; verifies JWT then calls `resolveUserOrgContext(userId)` to enrich `req.user` with org/team/plan data. Exports `requireOwner` guard (checks `req.user.orgRole === 'owner'`).
- `src/services/org.service.js` — `resolveUserOrgContext(userId)` performs DB lookups on `OrgMembership`, `TeamMembership`, `TeamRepo`, and `OrgSubscription` and returns:

```js
{
  orgId, orgRole,
  isOrgMember,      // true = org-wide repo access; false = team-only
  teamIds,
  planCode,
  subscriptionStatus,
  repoIds           // accessible repo IDs for team-only members
}
```

**Key design decision:** JWT payload is kept minimal (userId, email); org/team/plan context is resolved from DB on every request so membership changes take effect immediately (no stale JWT payloads).

---

### TW-117 — SSE Streaming Proxy

**What it asked for:** Forward RAG `/query` responses as Server-Sent Events with zero buffering.

**What was done:**

- `src/services/proxy.service.js` — two proxy modes:
  - `proxyRequest()` — standard full-response proxy using Node built-in `http`/`https`; 30s timeout; pipes upstream response directly to client
  - `proxySSE()` — SSE mode: sets `Content-Type: text/event-stream`, `X-Accel-Buffering: no`, calls `res.flushHeaders()` immediately, then `proxyRes.pipe(res)` with no intermediate buffering; destroys upstream connection if client disconnects

- `src/routes/proxy.routes.js` — detects `Accept: text/event-stream` on `POST /query` and routes to `proxySSE`; all other routes use `proxyRequest`

**Trusted headers injected on every upstream call:**

| Header | Source |
|---|---|
| `X-Request-Id` | `req.requestId` |
| `X-User-Id` | `req.user.userId` |
| `X-User-Email` | `req.user.email` |
| `X-Org-Id` | `req.user.orgId` |
| `X-Org-Role` | `req.user.orgRole` |
| `X-Team-Ids` | `req.user.teamIds.join(',')` |
| `X-Plan-Code` | `req.user.planCode` |
| `X-Repo-Ids` | team-only members' accessible repos |
| `X-Allowed-Model-Families` | set by `requireModelAccess` |
| `X-Default-Model-Family` | set by `requireModelAccess` |

---

### TW-118 — Entitlement Middleware

**What it asked for:** Block requests from orgs with inactive subscriptions, and enforce repo-level access control.

**What was done:**

- `src/middlewares/entitlement.middleware.js`:
  - `requireActiveSubscription` — blocks `canceled` and `past_due` subscriptions with `{ code: 'SUBSCRIPTION_INACTIVE' }`
  - `requireRepoAccess({ paramName?, bodyField? })` — extracts a `repoId` from route param or body; org members (`isOrgMember=true`) pass automatically; team-only members are checked against their resolved `repoIds` list

**Repo ACL rule:** Org members access all org repos. Team-only members access only repos assigned to their teams (resolved at auth time by `resolveUserOrgContext`).

**Wired on:**
- `GET /repos/:id/status` → `requireRepoAccess({ paramName: 'id' })`
- `POST /search`, `POST /query`, `GET /repos`, `POST /admin/repos/:id/sync` → `requireActiveSubscription`

---

### TW-119 — Rate Limiting + Multi-Origin CORS

**What it asked for:** Per-user rate limits with Redis backing; CORS supporting multiple allowed origins.

**What was done:**

- `src/middlewares/rate-limit.middleware.js`:
  - `generalLimiter` — 100 req/min (configurable via `RATE_LIMIT_GENERAL_MAX`)
  - `queryLimiter` — 20 req/min (configurable via `RATE_LIMIT_QUERY_MAX`)
  - Key is `req.user.userId` when authenticated; falls back to `ipKeyGenerator` (from `express-rate-limit` v7) for unauthenticated requests
- `src/config/redis.js` — lazy `getRedisClient()`, `connectRedis()` with graceful degradation
- `src/app.js` — `CORS_ORIGINS` env var accepts a comma-separated list; falling back to `CORS_ORIGIN` for backwards compatibility

---

### TW-132 — Data Models (Orgs, Teams, Subscriptions, Plans)

**What it asked for:** Complete MongoDB schema set for the org/team/plan domain.

**Models created:**

| File | Purpose |
|---|---|
| `organization.model.js` | type (virtual/real), planCode, subscriptionStatus, createdBy |
| `org-membership.model.js` | userId ↔ orgId ↔ role (owner/developer/viewer); compound unique index |
| `team.model.js` | orgId, name, description |
| `team-membership.model.js` | userId ↔ teamId; compound unique index |
| `team-repo.model.js` | teamId ↔ repoId; compound unique index |
| `org-subscription.model.js` | orgId (unique), planCode, status (active/trialing/past_due/canceled), startedAt, renewedAt, canceledAt |
| `org-usage.model.js` | orgId (unique), indexedRepoCount |
| `plan.model.js` | planCode, maxIndexedRepos, allowedModelFamilies[], defaultModelFamily, features[], monthlyPrice, yearlyPrice |
| `provider-connection.model.js` | orgId, provider (github/bitbucket), connectionType, encryptedCredentials, status, scopes, expiresAt |
| `provider-repo.model.js` | orgId, connectionId, providerRepoId, fullName, visibility, isImportable, lastSeenAt |
| `provider-audit-log.model.js` | orgId, connectionId, actorUserId, action (connect/disconnect/refresh/revoke/import/sync/error), metadata |

**Virtual org auto-creation:** `auth.service.js` (email/password signup) and `auth-google.service.js` (Google OAuth) both call `createVirtualOrg(userId, email)` on new user creation — every user gets a personal workspace automatically on sign-up.

**Plan seed (`scripts/seed.js`):**

| Plan | maxIndexedRepos | allowedModelFamilies | defaultModelFamily |
|---|---|---|---|
| free | 2 | openai-4-series | openai-4-series |
| pro | 20 | openai-4-series, openai-5-series | openai-5-series |

---

### TW-133 — Indexed Repo Limit Gate

**What it asked for:** Block sync/import requests when an org has hit the plan's maximum indexed repository count.

**What was done:**

- `src/middlewares/plan-gate.middleware.js` → `requireRepoLimit`:
  1. Reads `req.user.orgId`
  2. Calls `getOrgPlan(orgId)` → gets plan + subscription
  3. Calls `getIndexedRepoCount(orgId)` → reads `OrgUsage.indexedRepoCount`
  4. If `count >= plan.maxIndexedRepos`, returns `403 { code: 'REPO_LIMIT_EXCEEDED', planCode, maxIndexedRepos, currentCount }`

**Wired on:** `POST /api/v1/admin/repos/:id/sync`
**Also needed on (TW-142):** `POST /api/v1/repos/import`

---

### TW-134 — Model-Family Gate

**What it asked for:** Validate the requested AI model is allowed under the org's plan; inject plan default if no model is specified.

**What was done:**

- `src/middlewares/plan-gate.middleware.js` → `requireModelAccess`:
  1. Reads `req.body.model`
  2. Maps it to a family via `MODEL_FAMILY_MAP` (gpt-4, gpt-4o, gpt-4o-mini, gpt-4-turbo → `openai-4-series`; o1, gpt-5 → `openai-5-series`)
  3. If no model provided — injects `plan.defaultModelFamily` into `req.body.model`
  4. If model family not in `plan.allowedModelFamilies` — returns `403 { code: 'MODEL_NOT_ALLOWED', modelFamily, allowedModelFamilies }`
  5. Sets `req.planModelContext` which proxy service forwards as `X-Allowed-Model-Families` and `X-Default-Model-Family` headers

**Wired on:** `POST /api/v1/search`, `POST /api/v1/query`

---

### TW-136 — GitHub App Installation Flow

**What it asked for:** Owners can connect their org to GitHub via a GitHub App installation. The flow must be CSRF-safe.

**What was done:**

- `src/services/github.service.js`:
  - `generateOAuthState(orgId, userId, provider)` — signs a 10-minute JWT containing `{ orgId, userId, provider, type: 'oauth_state' }` using `JWT_SECRET` — this is the CSRF-safe state token
  - `verifyOAuthState(state, expectedProvider)` — verifies the JWT and checks provider matches
  - `buildGitHubConnectUrl(orgId, userId)` — generates the `https://github.com/apps/<slug>/installations/new?state=<jwt>` redirect URL
  - `handleGitHubCallback({ installationId, setupAction, state })` — verifies state, upserts `ProviderConnection` with AES-encrypted `installationId`, writes to `ProviderAuditLog`

- `src/routes/integrations.routes.js`:
  - `POST /api/v1/integrations/github/connect` (requireAuth + requireOwner) — returns `{ redirectUrl }`
  - `GET /api/v1/integrations/github/callback` — GitHub redirects here after install; on success, redirects to `FRONTEND_URL/integrations/github?connected=true`

---

### TW-137 — AES-256-GCM Token Encryption

**What it asked for:** Provider OAuth tokens and installation IDs stored in MongoDB must be encrypted at rest.

**What was done:**

- `src/utils/encrypt.js`:
  - `encrypt(plaintext)` — generates random 16-byte IV, encrypts with AES-256-GCM using `PROVIDER_TOKEN_ENCRYPTION_KEY`, returns `iv:authTag:ciphertext` as a single hex string
  - `decrypt(ciphertext)` — splits on `:`, reconstructs cipher, verifies auth tag, returns plaintext

```
Format: <16-byte-iv-hex>:<16-byte-auth-tag-hex>:<ciphertext-hex>
Key:    64 hex chars = 32 bytes (AES-256)
```

Used by both `github.service.js` and `bitbucket.service.js` to encrypt credentials before storing in `ProviderConnection.encryptedCredentials`.

---

### TW-138 — Bitbucket OAuth Flow

**What it asked for:** Owners can connect their org to Bitbucket via OAuth 2.0 (Authorization Code grant).

**What was done:**

- `src/services/bitbucket.service.js`:
  - Reuses `generateOAuthState`/`verifyOAuthState` from `github.service.js`
  - `buildBitbucketConnectUrl(orgId, userId)` — builds Bitbucket OAuth authorization URL with signed state
  - `exchangeCodeForToken(code)` — performs Basic-auth POST to `https://bitbucket.org/site/oauth2/access_token` using Node built-in `https`
  - `fetchBitbucketUser(accessToken)` — fetches `https://api.bitbucket.org/2.0/user` to get account name/ID
  - `handleBitbucketCallback({ code, state })` — verifies state, exchanges code for tokens, fetches user info, stores AES-encrypted `{ accessToken, refreshToken, tokenExpiresAt }` in `ProviderConnection`, writes audit log

- `src/routes/integrations.routes.js`:
  - `POST /api/v1/integrations/bitbucket/connect` (requireAuth + requireOwner) — returns `{ redirectUrl }`
  - `GET /api/v1/integrations/bitbucket/callback` — Bitbucket redirects here; handles `?error=` query param; on success redirects to `FRONTEND_URL/integrations/bitbucket?connected=true`

---

## Pending Tickets

---

### TW-139 — Provider Repo Discovery & Cache

**What it asks for:** After connecting GitHub/Bitbucket, list all repos visible to the installation/OAuth token. Cache results in `ProviderRepo` collection with `lastSeenAt` so re-discovery doesn't hammer provider APIs.

**Key tasks:**
- GitHub: use GitHub App API with installation token (generate via App private key + `jsonwebtoken`)
- Bitbucket: use stored access token; handle token refresh if expired
- Upsert results into `ProviderRepo` collection
- `GET /api/v1/integrations/:provider/repos` endpoint

---

### TW-140 — Provider Disconnect & Revoked-Token Handling

**What it asks for:** Owners can disconnect a provider. System must also handle revoked tokens gracefully (GitHub App uninstall webhook, Bitbucket token revocation).

**Key tasks:**
- `DELETE /api/v1/integrations/:provider/connect` — sets `ProviderConnection.status = 'revoked'`, writes audit log
- GitHub webhook handler for `installation.deleted` event (verify signature with `GITHUB_WEBHOOK_SECRET`)
- Propagate disconnect to `repo-sync` service so indexed repos are marked stale

---

### TW-142 — Import Provider Repos into Indexing

**What it asks for:** Owner selects repos from `ProviderRepo` to import for indexing. Must enforce plan's `maxIndexedRepos` limit.

**Key tasks:**
- `POST /api/v1/repos/import` — body: `{ providerRepoIds: [] }`
- Apply `requireRepoLimit` middleware (already built in TW-133)
- Call `repo-sync` upstream to start indexing job
- Increment `OrgUsage.indexedRepoCount` via `incrementIndexedRepoCount(orgId, delta)`

---

### TW-120 — Gateway Security & Integration Tests

**What it asks for:** Security hardening and end-to-end integration tests for the full gateway flow.

**Key tasks:**
- Test auth middleware: expired JWT, invalid JWT, missing token
- Test entitlement middleware: canceled subscription, repo access denied
- Test plan-gate: repo limit exceeded, model not allowed
- Test proxy: upstream 502, 504 timeout handling
- Test rate limiter: per-user key isolation
- Security review: no secrets in logs, no raw token in headers forwarded downstream

---

### TW-141 — Source Provider Connection Security Tests

**Key tasks:**
- Test state JWT expiry (10-min window)
- Test provider mismatch in state token
- Test GitHub callback with `setup_action=delete`
- Test Bitbucket callback with `?error=access_denied`
- Test encrypted credentials round-trip (encrypt → store → decrypt)

---

### TW-135 — Pro → Free Downgrade Repo Policy (Product Decision)

**What it asks for:** Define what happens to indexed repos when an org downgrades from Pro (max 20 repos) to Free (max 2 repos).

**Options to decide:**
1. Hard block — keep existing 2, immediately unindex the rest
2. Soft grace period — allow existing indexed repos to remain until next billing cycle, then enforce
3. User-controlled — owner chooses which 2 to keep

**This is a product/design decision, not a coding task.** Assigned to product owner for resolution before TW-142 can be fully implemented.

---

## Architecture Summary

```
Client
  │
  ▼
api.lucos.com (Express Gateway :8080)
  │
  ├─── tracing middleware       (X-Request-Id)
  ├─── CORS + Helmet + rate-limit
  ├─── requireAuth              (JWT verify + resolveUserOrgContext DB lookup)
  ├─── requireActiveSubscription (subscription status check)
  ├─── requireRepoAccess        (org member bypass / team repo list check)
  ├─── requireModelAccess       (plan model family validation + default injection)
  ├─── requireRepoLimit         (indexed repo count vs plan cap)
  │
  ├─── /api/v1/integrations/github/*    → github.service.js
  ├─── /api/v1/integrations/bitbucket/* → bitbucket.service.js
  │
  ├─── /api/v1/search     ─────────────────────────► rag-query-service :6005
  ├─── /api/v1/query      ─── SSE streaming ────────► rag-query-service :6005
  ├─── /api/v1/repos      ─────────────────────────► rag-query-service :6005
  ├─── /api/v1/repos/:id/status ───────────────────► repo-sync :6001
  └─── /api/v1/admin/repos/:id/sync ───────────────► repo-sync :6001
```

**Trusted headers forwarded to every upstream:**
`X-Request-Id`, `X-User-Id`, `X-User-Email`, `X-Org-Id`, `X-Org-Role`, `X-Team-Ids`, `X-Plan-Code`, `X-Repo-Ids`, `X-Allowed-Model-Families`, `X-Default-Model-Family`