# Repo Docs / Knowledge Base Ingestion Service

A FastAPI + MongoDB service that ingests human-written repository documentation (README, ADRs, runbooks, etc.), parses it into sections, and chunks it for downstream embedding and retrieval.

## Architecture

```
Redis files.changed (XREADGROUP docs-ingestion-service)
        │  file_kind=docs
        ▼
  DocParseRun (dedupe_key per snapshot)
        │
        ▼
  detect_doc_type  ──▶  repo_documents (upsert)
        │
        ▼
  MarkdownParser   ──▶  repo_document_sections
        │
        ▼
  TextChunker      ──▶  doc_chunks (repo_id + snapshot_id)
        │
        ▼
  indexing_runs.processed_docs_files += 1
        │
        ▼
  XACK (strict — only after durable Mongo write)

POST /api/v1/ingest (HTTP path unchanged — UUID parse runs)
```

## MongoDB Collections

| Collection | Purpose |
|---|---|
| `repo_documents` | Raw document metadata + content |
| `repo_document_sections` | Parsed heading sections |
| `doc_chunks` | Text chunks ready for embedding |
| `doc_parse_runs` | Ingestion run status tracking (stream runs keyed by `dedupe_key`) |

## Supported Doc Types

`readme` · `docs` · `architecture` · `contributing` · `deployment` · `onboarding` · `adr` · `runbook`

## Setup

```bash
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload
```

## Docker

```bash
cd docker
docker compose up --build
```

## API

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/v1/health` | Liveness and dependency status |
| `GET` | `/api/v1/ready` | Readiness probe (Mongo required) |
| `GET` | `/api/v1/version` | Service metadata/version |
| `POST` | `/api/v1/ingest/` | Queue a document for ingestion |
| `GET` | `/api/v1/ingest/status/{run_id}` | Poll ingestion run status |
| `GET` | `/api/v1/chunks/{repo_id}` | Retrieve chunks (filterable by `doc_path`) |

### Ingest a document

```bash
curl -X POST http://localhost:8000/api/v1/ingest/ \
  -H "Content-Type: application/json" \
  -d '{
    "repo_id": "my-org/my-repo",
    "doc_path": "README.md",
    "content": "# Overview\nThis service does X.\n\n## Setup\nRun `make install`.",
    "commit_hash": "abc123"
  }'
```

## Running Tests

```bash
pip install -r tests/requirements-test.txt
pytest tests/unit/
```

## Environment Variables

Required and supported configuration is provided through `.env`:

- `APP_NAME`
- `ENV`
- `ENVIRONMENT`
- `PORT`
- `SERVICE_VERSION`
- `API_PREFIX`
- `MONGO_URI`
- `DATABASE_NAME`
- `LOG_LEVEL`
- `LOG_PATH`
- `LOG_ROTATION`
- `LOG_RETENTION`
- `LOG_SERIALIZE`
- `WORKER_CONCURRENCY`
- `REDIS_URL`
- `REDIS_STREAM_ENABLED`
- `REDIS_STREAM_PREFIX`
- `REDIS_STREAM_NAME`
- `REDIS_STREAM_GROUP` (default `docs-ingestion-service`)
- `REDIS_CONSUMER_NAME` (default hostname)
- `REDIS_STREAM_MAX_RETRIES` (default `5`)
- `REDIS_STREAM_AUTOCLAIM_MS` / `REDIS_STREAM_AUTOCLAIM_COUNT`
- `REDIS_STREAM_DLQ_NAME` (default `files.changed.dlq`)
- `REDIS_STREAM_BLOCK_MS`
- `REDIS_STREAM_COUNT`
- `GIT_WORKSPACE_PATH`
- `INDEXING_RUNS_ENABLED` / `INDEXING_RUNS_DATABASE` (default `adpilot_repo_sync`)
- `QUEUE_ENABLED` (placeholder)
- `QUEUE_BACKEND` (placeholder)
- `QUEUE_NAME` (placeholder)
- `QUEUE_POLL_INTERVAL_SECONDS` (placeholder)

See `.env.example` for defaults.