# Catalog index (TW-299)

Embeds **reviewed** schema catalogs into Qdrant collection **`lucos_business_catalog`**.

Never writes to `adpilot_embeddings` (code RAG). Never embeds row data or blocked columns (`pii` / `secret` / `deny_index` / `unknown`).

```
TW-297 harvest → TW-298 sensitivity:apply/gate → TW-299 catalog:index → TW-300 catalog_search
```

## Prerequisites

1. Catalog must pass `npm run sensitivity:gate`
2. Live mode needs `OPENAI_API_KEY` (Qdrant URL defaults to local)

## Env exports (copy/paste for local)

```bash
# Defaults — only OPENAI_API_KEY is required for live index
export QDRANT_URL=http://127.0.0.1:6333
export QDRANT_BUSINESS_COLLECTION=lucos_business_catalog
export EMBEDDING_MODEL=text-embedding-3-small
export EMBEDDING_DIMENSION=1536
# export QDRANT_API_KEY=          # only if your Qdrant requires auth
export OPENAI_API_KEY=sk-...      # required for live (not for --dry-run / --fixture)
# Low OpenAI tier (~10 RPM): keep delay; higher tiers can set EMBEDDING_BATCH_DELAY_MS=0
# export EMBEDDING_BATCH_DELAY_MS=7000
# export EMBEDDING_MAX_RETRIES=8
```

Or copy from `.env.example` into a local `.env` (never commit secrets).

## Commands

```bash
# Gate + chunk counts only (no network)
npm run catalog:index -- --catalog catalogs/schemas/staging/admin.json --dry-run

# CI / local without secrets (mock vectors + in-memory store)
npm run catalog:index -- --catalog catalogs/schemas/staging/admin.json --fixture

# Live index (uses defaults above if unset; needs OPENAI_API_KEY)
npm run catalog:index -- --catalog catalogs/schemas/staging/admin.json
```

## Chunking

- One Qdrant point **per table**
- Embed text lists **indexable columns only** (name + type)
- Re-index deletes prior points for the same `environment` + `targetId` first
- Creates collection `lucos_business_catalog` if missing (1536-dim, Cosine)

## Env

| Var | Default |
| --- | --- |
| `QDRANT_URL` | `http://127.0.0.1:6333` |
| `QDRANT_API_KEY` | optional (empty) |
| `QDRANT_BUSINESS_COLLECTION` | `lucos_business_catalog` |
| `OPENAI_API_KEY` | **required** for live |
| `EMBEDDING_MODEL` | `text-embedding-3-small` |
| `EMBEDDING_DIMENSION` | `1536` |
| `EMBEDDING_BATCH_SIZE` | `64` |
| `EMBEDDING_BATCH_DELAY_MS` | `7000` (0 = no pacing) |
| `EMBEDDING_MAX_RETRIES` | `8` |

## Hard rules

- Separate collection from code embeddings
- TW-298 gate must pass
- No PII/secret/deny_index column names in vectors or payloads
- Never commit API keys

## Search (TW-300)

MCP tool `catalog_search` uses the same collection. See `connectors/catalog/search.ts`.