# WebCrawlers SEO Automation Platform An AI-powered SEO automation platform built on CodeIgniter 4. Logged-in users manage website projects, run technical audits, track rankings, research keywords, generate and review AI content, and measure performance. The dashboard renders server-side and hydrates itself through a versioned REST API that mobile apps and public integrations can reuse. --- ## Table of Contents 1. [How This Repo Works](#how-this-repo-works) 2. [Developer Setup](#developer-setup) 3. [URL Structure](#url-structure) 4. [Environment Detection](#environment-detection) 5. [Configuration & Secrets](#configuration--secrets) 6. [Database](#database) 7. [Background Jobs](#background-jobs) 8. [Authentication & Authorization](#authentication--authorization) 9. [Project Architecture](#project-architecture) 10. [Folder Structure](#folder-structure) 11. [Adding a New Dashboard Page](#adding-a-new-dashboard-page) 12. [Adding a New API Endpoint](#adding-a-new-api-endpoint) 13. [Design Assets](#design-assets) 14. [External Integrations](#external-integrations) 15. [Feature Status](#feature-status) 16. [Known Gaps](#known-gaps) 17. [Related Documents](#related-documents) --- ## How This Repo Works The project runs on a **shared dev server** at `/var/www/php82//webcrawlers_latest.com/`. Multiple developers work on the same box; each clones into their own home directory under `/var/www/php82//`. The server is already configured with PHP 8.2 and MySQL. There is **no Docker and no local install step**. You edit files directly on the server (via SSH, SFTP, or an IDE with remote support) and hit the URL in your browser to test. There is **no `.env` file** — config lives in tracked PHP files so every developer shares the same settings after a `git pull`. See [Configuration & Secrets](#configuration--secrets) for why that needs to change before production. --- ## Developer Setup ### 1. Clone the repo ```bash cd /var/www/php82// git clone webcrawlers_latest.com cd webcrawlers_latest.com ``` ### 2. Install dependencies `vendor/` is **committed to the repo**, so a fresh clone is already runnable and this step is normally unnecessary. Run it only after a `composer.json` change: ```bash composer install ``` ### 3. Set folder permissions ```bash chmod -R 775 writable/ ``` ### 4. Set up the database Run the migrations and seeds — see [Database](#database). On the shared dev server the credentials resolve automatically, so this is usually just: ```bash php8.2 spark migrate php8.2 spark db:seed DatabaseSeeder ``` [schema.md](schema.md) has the full runbook, including the staging/production procedure and the verification queries. ### 5. Access the site | Path | URL | |---|---| | Public website | `http://192.168.30.106/php82//webcrawlers_latest.com/public/` | | Dashboard | `http://192.168.30.106/php82//webcrawlers_latest.com/public/?/dashboard` | | API | `http://192.168.30.106/php82//webcrawlers_latest.com/public/?/api/v1/dashboard` | > The `?/` prefix is the query-string router used on the shared dev server, which has no rewrite rules. `App::__construct()` detects the sandbox path and sets `indexPage = '?'` automatically, so `site_url()` and `url_to()` produce working links without any manual config. ### 6. Fill in API keys Open `app/Config/Constants.php` and set the `WC_*` constants — see [Configuration & Secrets](#configuration--secrets). On the shared dev server these are already populated. --- ## URL Structure ``` webcrawlers.com/ ← public marketing site (Pages, Blog, Auth, Lead) webcrawlers.com/ ← logged-in dashboard UI (one controller per module) webcrawlers.com/api/v1/ ← REST API (Api\V1\* controllers) webcrawlers.com/px/* ← tracking pixel ingest (public, CORS) webcrawlers.com/webhooks/* ← DataForSEO pingbacks (public) ``` **Dashboard routes are flat, not nested under `/dashboard`.** Each module is a top-level path with its own controller and its own `['filter' => 'auth']`: ```php $routes->get('competitors', 'CompetitorsController::index', ['filter' => 'auth']); $routes->get('rank-tracking','RankTrackingController::index', ['filter' => 'auth']); $routes->get('settings/users','UserController::index', ['filter' => 'auth']); ``` Only `/dashboard` itself maps to the `Dashboard` controller (the home page). There is no route group for the UI — the filter is declared per route. All `/api/v1/*` routes **are** in a group, protected by `apiauth`. The full route list is in [app/Config/Routes.php](app/Config/Routes.php). --- ## Environment Detection `CI_ENVIRONMENT` is set in `public/index.php` before CI4 boots, and `App::__construct()` applies matching URL config: | Runtime context | `CI_ENVIRONMENT` | `baseURL` / `indexPage` | |---|---|---| | `/var/www/phpNN/...` (any dev sandbox) | `development` | Host + sandbox path + `/public/`, `indexPage = '?'` | | `staging.webcrawlers.com` | `development` | Follows request scheme | | `dev.webcrawlers.com` | `development` | Follows request scheme | | `192.168.*`, `127.*`, `localhost` | `development` | Follows request scheme | | anything else | `production` | Declared `$baseURL` | Note that staging currently resolves to `development`, not `testing` — errors and the debug toolbar behave as they do in dev there. Only unrecognised hosts fall through to `production`. Setting `CI_ENVIRONMENT` in the server environment or on the CLI overrides all of the above; the detection block only runs when it is unset. That is why the CLI examples in [schema.md](schema.md) prefix commands with `CI_ENVIRONMENT=development`. --- ## Configuration & Secrets **No `.env` file is used.** All values live in `app/Config/Constants.php` so they are shared across the team via Git. > ⚠️ **These are live credentials in a tracked file.** Google OAuth, DataForSEO, OpenAI, Gemini, Sticky.io and ZeptoMail secrets are in Git history and must be treated as compromised. Before production: rotate every key, move the values to the server environment or a `.gitignore`d `.env`, and restore empty defaults here. | Constant | Purpose | |---|---| | `WC_ENCRYPTION_KEY` | Signs API tokens (HMAC-SHA256). Generate with `php -r "echo bin2hex(random_bytes(32));"` | | `WC_OPENAI_API_KEY` / `WC_OPENAI_MODEL` | OpenAI — content generation, analysis | | `WC_GEMINI_API_KEY` / `WC_GEMINI_MODEL` | Gemini — fallback content provider | | `WC_DATAFORSEO_LOGIN` / `_PASSWORD` / `_API_KEY` | DataForSEO account credentials | | `WC_GOOGLE_CLIENT_ID` / `_SECRET` / `_REDIRECT_URI` | Google OAuth2 (GSC / GA4 / GBP) | | `WC_STICKY_API_URL` / `_URL_V2` / `_USERNAME` / `_PASSWORD` | Sticky.io subscription billing gateway | | `ZEPTOMAIL_API_URL` / `ZEPTOMAIL_API_KEY` | Transactional email (OTP, invites, receipts) | `WC_HUBSPOT_API_KEY` and `WC_FRESHDESK_*` are present but commented out — no CRM integration is wired up yet. `Encryption.php` references `WC_ENCRYPTION_KEY` directly so the CI4 encryption service and HMAC token signing always use the same key. Changing it invalidates every issued API token. **Database credentials** come from `MSACOMMON_PATH . 'dbConstants.php'` — a shared library on the dev server. `Database.php` reads `MSACOMMON_DB_HOST_CMS`, `MSACOMMON_DB_USER_CMS` and `MSACOMMON_DB_PASS_CMS` from it automatically. --- ## Database - **Driver:** MySQL 5.7 via MySQLi - **Collation:** `utf8mb4_unicode_ci` on every table and column (8.0-only collations are unavailable) - **Migrations:** `app/Database/Migrations/` — 15 consolidated first-run migrations - **Seeds:** `app/Database/Seeds/` — countries, modules, module permissions, plans, owner accounts - **Dev data dump:** `dev-data/` — see `dev-data/README.md` **[schema.md](schema.md) is the authoritative schema reference.** It documents all 39 tables column by column, with indices, foreign keys, the migration-to-table map, and a full sysadmin runbook for staging and production. Do not use `TASKS.md` for column definitions — it predates the current schema. ```bash php8.2 spark migrate # apply all migrations php8.2 spark migrate:status # 15 rows, all batch 1, on a fresh DB php8.2 spark db:seed DatabaseSeeder ``` Migration order is load-bearing: `wc_users` / `wc_countries` come first because everything keys off them, and `wc_websites` must exist before `wc_automation_queue` (foreign key). Never run them individually or out of order. > Four tables — `wc_evidence`, `wc_evidence_services`, `wc_evidence_reviews`, `wc_change_plan_requests` — are used by shipped code but have **no migration**. A fresh database will not have them and the Evidence pages and change-plan flow will fail. See [Known Gaps](#known-gaps). --- ## Background Jobs **No Redis, no Supervisor** — both need sudo and are unavailable on the shared dev server. Async work uses a **DB-backed queue** in `wc_automation_queue`. ### How it runs Statuses move `pending → processing → completed`, with automatic retry up to 3 times before `failed`. Last run time per site is tracked in `wc_websites.last_automation_run`. Two triggers, both hitting the same processor: **Page-load trigger (Phase 1, current).** Visiting the SEO Automation dashboard calls `POST /api/v1/automation/process` with a website ID. Rate-limited to once per day per website; `localStorage` in `seo-automation.js` avoids redundant calls. **Cron (Phase 2).** `webcrawlers_cron.sh` in the repo root runs the processor and appends to `writable/logs/automation_queue.log`: ```bash */5 * * * * /home//webcrawlers_cron.sh ``` Or call spark directly: ```bash php8.2 spark automation:process --limit=10 ``` See [AUTOMATION_PHASE_2_CRON_SETUP.md](AUTOMATION_PHASE_2_CRON_SETUP.md) for the full setup. ### Inspecting the queue ```sql SELECT * FROM wc_automation_queue WHERE status = 'pending'; ``` ### What the processor does `AutomationProcess` (`app/Commands/AutomationProcess.php`) and `Api\V1\AutomationController` claim pending rows and dispatch by job type — audit crawls via `AuditSyncService`, opportunity generation, rank syncing. There is no separate job-class directory; the work lives in the service classes under `app/Libraries/`. > There is no general-purpose `queue_jobs` table and no `spark queue:work` command. `automation:process` is the only queue worker in the codebase. --- ## Authentication & Authorization ### Dashboard UI (session-based) - Sign in → OTP → `session()->set(['user_id' => ..., 'logged_in' => true])` - `AuthFilter` runs on every route carrying `['filter' => 'auth']`: it checks `session('logged_in')`, reloads the user row, and redirects to `/signin` if either fails. - It then enforces plan state via `PlanAccess::requiresCheckout()` — users owing checkout are redirected to `/payment` unless they are already on a checkout page. Members waiting on an admin get an explanatory flash message. ### API (Bearer token or session) `ApiAuthFilter` accepts either: - **Bearer token** — `base64(JSON{uid, iat}) . "." . HMAC-SHA256(payload, WC_ENCRYPTION_KEY)`, valid for 24 hours, compared with `hash_equals()`. - **Session** — falls back to `session('logged_in')` when no `Authorization` header is present, which is what same-origin dashboard calls use. Either way it reloads the user and applies the same `PlanAccess` checkout gate, returning `401` for auth failures and `403` for plan-denied requests. Tokens are minted per page render by `DashboardModule::generateApiToken()` and published to the browser as ``. `window.wcFetch()` (defined in `layout/bottom.php`) attaches the Bearer header and the CSRF header to every call. This means the same API serves dashboard JS (session or token), future mobile apps (token only), and a future public API. ### CSRF `CsrfFilter` extends the framework filter and, instead of throwing, regenerates the token and redirects back with the input preserved so the user can resubmit immediately. It applies globally to POST/PUT/PATCH/DELETE, with these paths excepted in `Config/Filters.php`: `api/*`, `webhooks/*`, `keyword-research/*`, `px/*`. ### Roles, teams and plans - `TeamAccess` resolves which user IDs a given user may see; models call `visibleUserIds()` so team members share a workspace's data. - `PlanAccess` and `Config/SubscriptionPlans.php` define per-plan limits (domains, pages crawled, AI drafts, locations, team seats) for Beta Partner, Managed Growth and Enterprise. - `wc_modules` / `wc_module_permission_mapping` gate sidebar entries — each nav item in `layout/nav.php` carries a `module` key matching `wc_modules.code`. --- ## Project Architecture ``` Public marketing site │ ▼ Pages / Blog / Auth / Lead controllers │ ▼ Dashboard module controllers ──────► flat / routes (HTML, AuthFilter) extends DashboardModule │ │ ▼ │ layout/top + view + layout/bottom │ JS calls wcFetch() on load ▼ API v1 controllers ────────────────► /api/v1/* (JSON, ApiAuthFilter) extends BaseApiController │ ▼ Service Layer (app/Libraries/) ┌──────────────────────────────────────────────────┐ │ DataForSeo SiteOverviewService │ │ AuditSyncService OwnCrawlAuditService │ │ CrawlEngine CrawlNotifier │ │ RankTrackingService KeywordResearchService │ │ AiContentService AiContentAnalyzerService │ │ ContentScoringService GoogleApiService │ │ EvidenceService EvidenceServicesService │ │ EvidenceReviewsService GoogleBusinessReviews… │ │ StickyLibrary Otp │ │ PlanAccess TeamAccess │ │ *ExportService (CSV / XLSX / PDF) │ └──────────────────────────────────────────────────┘ │ ▼ Models (app/Models/) ──► MySQL (wc_* tables) │ ▼ wc_automation_queue │ ▼ php spark automation:process (cron / API trigger) │ ▼ ┌──────────────────────────────────────────────────┐ │ DataForSEO (OnPage · Keywords · SERP · Labs) │ │ OpenAI + Gemini (content generation, analysis) │ │ Google APIs (GSC · GA4 · GBP, via OAuth2) │ │ Sticky.io (subscription billing) │ │ ZeptoMail (transactional email) │ │ Own CrawlEngine (in-house technical crawl) │ └──────────────────────────────────────────────────┘ ``` **The pattern is Controller → Library → Model → MySQL.** Controllers stay thin: they resolve the user and selected website, call a service, shape the result for the view, and render. Business logic, external API calls and caching belong in `app/Libraries/`. [CompetitorsController.php](app/Controllers/CompetitorsController.php) is a good short example. --- ## Folder Structure ``` app/ Commands/ AutomationProcess.php spark automation:process — the only queue worker Config/ App.php baseURL + indexPage detection per environment Constants.php WC_* API keys and secrets Encryption.php references WC_ENCRYPTION_KEY Filters.php filter aliases + global CSRF with exceptions Routes.php all routes — public, dashboard, api/v1, px, webhooks SubscriptionPlans.php plan codes, prices and per-plan limits OnPageChecks.php audit check definitions Controllers/ BaseController.php framework base DashboardModule.php base for every dashboard page — render(), token minting Auth.php sign-up, sign-in, OTP, password reset Pages.php Blog.php Lead.php public site Dashboard.php dashboard home Controller.php one per dashboard module (53 files at root) Api/ KeywordResearchController.php /api/keyword-research/* KeywordListsController.php /api/keyword-lists/* V1/ BaseApiController.php token decode + ok/created/error/notFound Controller.php 19 REST resources Database/ Migrations/ 15 consolidated first-run migrations Seeds/ DatabaseSeeder + country, module, plan, user seeders Filters/ AuthFilter.php session + plan guard for dashboard routes ApiAuthFilter.php Bearer token or session + plan guard for /api/v1/* CsrfFilter.php CSRF with graceful recovery Libraries/ the service layer — 23 classes, see architecture above Models/ 33 models over the wc_* tables Views/ layouts/ public site layouts (main, auth) includes/ public site header, footer, menus, scripts dashboard/ layout/ top.php HTML shell — sidebar, topbar, meta tags, extra_css bottom.php closing tags, wcFetch() bootstrap, extra_js nav.php sidebar map; `module` keys gate by plan components.php dash_*() render helpers index.php dashboard home .php one view per module (44 files incl. index and stub) stub.php placeholder for pages not yet built .php marketing, auth, pricing, payment views public/ index.php entry point — sets CI_ENVIRONMENT, boots CI4 px.js tracking pixel served to client websites assets/ ← the LIVE assets the app actually loads css/ dashboard.css + one file per page (82 files) js/ dashboard.js + one file per page (64 files) img/ video/ webcrawlers-dashboard-assets/ ← design reference only, not served *.php original design files includes/ original layout includes assets/ stale copies of css/js — do NOT edit these schema.md authoritative database reference + deploy runbook webcrawlers_cron.sh automation queue cron entry point dev-data/ dev database dump + generator ``` --- ## Adding a New Dashboard Page 1. **Add a flat route** in [app/Config/Routes.php](app/Config/Routes.php) with the auth filter — not inside a group: ```php $routes->get('my-page', 'MyPageController::index', ['filter' => 'auth']); ``` 2. **Create the controller** in `app/Controllers/`, extending `DashboardModule`: ```php class MyPageController extends DashboardModule { public function index(): string { $userId = (int) session()->get('user_id'); $websites = (new WebsiteModel())->getForUser($userId); return $this->render('dashboard/my-page', [ 'active' => 'my-page', // matches the key in nav.php 'page_title' => 'My Page', 'page_kicker' => 'Site', // breadcrumb group label 'extra_css' => ['my-page'], // loads assets/css/my-page.css 'extra_js' => ['my-page'], // loads assets/js/my-page.js 'extra_cdn_js' => [], // full URLs, e.g. Chart.js 'websites' => $websites, ]); } } ``` `render()` composes `layout/top` + your view + `layout/bottom` and injects `user`, `api_token`, `csrf_token` and `csrf_hash` automatically. 3. **Add the nav entry** in `app/Views/dashboard/layout/nav.php` if it needs a sidebar link. The `module` key must match a `wc_modules.code` row, or the item will be hidden for every plan. 4. **Create the view** at `app/Views/dashboard/my-page.php`. Use the design reference at `public/webcrawlers-dashboard-assets/.php` as the template: take everything between the `layout-top.php` and `layout-bottom.php` includes, point hrefs at `site_url('...')`, and use the `dash_*()` helpers from `layout/components.php`. Until the view file exists the page renders `stub.php`, so nothing crashes. 5. **Add a service and API endpoint** if the page needs data beyond the initial render — see below. Keep queries and external calls out of the controller. --- ## Adding a New API Endpoint 1. **Create a controller** in `app/Controllers/Api/V1/` extending `BaseApiController`: ```php class MyController extends BaseApiController { public function index(): ResponseInterface { // $this->apiUserId is the authenticated user $data = (new MyService())->list($this->apiUserId); return $this->ok(['items' => $data]); } } ``` `BaseApiController` provides `ok()`, `created()`, `error($message, $status)` and `notFound()`. Use these rather than building responses by hand — `AutomationController` extends `ResourceController` instead and is the inconsistent exception, not the pattern to copy. 2. **Register the route** inside the `api/v1` group in [app/Config/Routes.php](app/Config/Routes.php): ```php $routes->get('my-resource', 'MyController::index'); ``` Order matters: declare specific paths **before** wildcards, or `(:num)` routes will swallow them. The `websites/plan` and `websites/(:num)` pairs in the group show the required ordering. 3. **Call it from the page JS** with `wcFetch()`, which attaches the Bearer token and CSRF header: ```js wcFetch('my-resource') .then(r => r.json()) .then(data => { /* render */ }); ``` Note the API group is exempt from CSRF (`Config/Filters.php`) because token regeneration breaks repeated polling POSTs; authentication is the Bearer token or session instead. --- ## Design Assets The UI design lives in `public/webcrawlers-dashboard-assets/`. The `*.php` files and `includes/` are **reference only** — they are not CI4 views and are not routed. They exist so developers can open the intended design in a browser without logging in. | Asset | Path | Live? | |---|---|---| | Page designs | `public/webcrawlers-dashboard-assets/*.php` | No | | Reference layout shell | `public/webcrawlers-dashboard-assets/includes/` | No | | Reference CSS/JS copies | `public/webcrawlers-dashboard-assets/assets/` | No | | **Dashboard CSS** | `public/assets/css/` | **Yes** | | **Dashboard JS** | `public/assets/js/` | **Yes** | **Edit the files under `public/assets/`, not the copies under `webcrawlers-dashboard-assets/assets/`.** `extra_css => ['competitors']` resolves to `base_url('assets/css/competitors.css')`, i.e. `public/assets/css/competitors.css`. Both trees contain a `competitors.css`; only the first is served. Each is cache-busted with `filemtime()`. When building a view, open the matching design file side by side; `layout/top.php` and `layout/bottom.php` already emit the same HTML shell as the design's `layout-top.php` / `layout-bottom.php`. --- ## External Integrations | Capability | Service | Constants | Implemented by | |---|---|---|---| | Technical audit / crawl | DataForSEO OnPage | `WC_DATAFORSEO_*` | `DataForSeo`, `AuditSyncService` | | In-house crawl | own PHP crawler | — | `CrawlEngine`, `OwnCrawlAuditService` | | Keyword research, gap analysis | DataForSEO Keywords / Labs | `WC_DATAFORSEO_*` | `KeywordResearchService` | | Rank tracking | DataForSEO SERP + pingbacks | `WC_DATAFORSEO_*` | `RankTrackingService` | | Site overview, competitors, backlinks | DataForSEO Labs | `WC_DATAFORSEO_*` | `SiteOverviewService` | | Content briefs and scoring | DataForSEO + LLM | both | `ContentScoringService` | | AI content generation | OpenAI, Gemini fallback | `WC_OPENAI_*`, `WC_GEMINI_*` | `AiContentService` | | AI page analysis and suggestions | OpenAI | `WC_OPENAI_*` | `AiContentAnalyzerService` | | Search Console / GA4 / Business Profile | Google APIs (OAuth2) | `WC_GOOGLE_*` | `GoogleApiService`, `GoogleOAuthController` | | Subscription billing | Sticky.io | `WC_STICKY_*` | `StickyLibrary` | | Transactional email | ZeptoMail | `ZEPTOMAIL_*` | `Otp`, invite and receipt flows | | Conversion / vitals tracking | own pixel | — | `PixelController`, `public/px.js` | | CRM (leads, demos) | not integrated | commented out | leads stored in `wc_leads` only | Every DataForSEO call is logged to `wc_dataforseo_logs` via `DataForSeo::withLogContext()`, which is the first place to look when an audit or ranking refresh misbehaves. Google integrations are **built and routed**, not deferred. [DATAFORSEO_VS_GOOGLE_APIS_ANALYSIS.md](DATAFORSEO_VS_GOOGLE_APIS_ANALYSIS.md) records the original build-vs-defer analysis and is kept for historical rationale only — it does not describe current state. --- ## Feature Status Grouped as the sidebar groups them. "Built" means the page is backed by a service and API endpoints; "UI only" means the controller renders a static view and no API route or service exists behind it yet. | Group | Module | State | |---|---|---| | Home | Dashboard | Built | | Site | Website, Crawl Monitoring, Page Inventory, Technical Issues, Crawl History | Built | | Site | Site Overview | Built (DataForSEO Labs, DB-cached) | | Keywords | Keywords, Rank Tracking, Keyword Research & Gap Analysis | Built | | AI Content | Generation, History, Content Scoring & Briefs, Content Analyzer | Built | | Discover | SEO Automation | Built | | Discover | Competitors | Built | | Discover | Opportunity Queue | **UI only** | | Evidence | Library, Services, Reviews | Built — full API + services, but **tables not migrated** (see [Known Gaps](#known-gaps)) | | Evidence | Brand Rules | **UI only** | | Publish | Review Queue, Scheduler, Deployments | **UI only** | | Measure | Search Performance, Conversions, Attribution, Locations, AI Visibility | **UI only** | | Reports | Audit reports, exports (CSV / XLSX / PDF), sharing | Built | | Operations | Notifications | Built (SUB-25) | | Operations | Audit History | Built (hydrates from `GET /api/v1/reports`) | | Operations | Job Health | **UI only** | | Settings | Users & Roles, Modules, Connections, Billing, Leads | Built | | Settings | Workspace | **UI only** | | Billing | Payment, checkout, subscription, history, cancel, change plan | Built via Sticky.io | A quick way to tell them apart: a UI-only module's page JS under `public/assets/js/` contains no `wcFetch()` call, and the module name appears nowhere in the `api/v1` route group. `ConversionsController` plus `conversions.js` is the canonical example — a 14-line render and a script with no API calls, so the page shows static markup only. Do not read a thin controller as unfinished on its own — `RankTrackingController` is 20 lines because the page hydrates through `Api\V1\RankTrackingController` and `RankTrackingService`. Thin controller **plus** no API route is the signal. `TASKS.md` and `JIRA_TICKETS.md` hold the original per-ticket breakdown. Both predate much of the above — treat the code and this table as authoritative. --- ## Known Gaps Live issues a new developer should know about, rather than rediscover: 1. **Four tables have no migration.** `wc_evidence`, `wc_evidence_services`, `wc_evidence_reviews` and `wc_change_plan_requests` are read and written by shipped code but are not created by any migration, so a fresh `php spark migrate` yields a database where the Evidence pages and the change-plan flow fail. Column lists inferred from the models are documented in [schema.md](schema.md#tables-used-by-code-but-not-created-by-migration); the fix is to write the migrations, not to hand-create tables per environment. 2. **Secrets are committed.** See the warning in [Configuration & Secrets](#configuration--secrets). Rotation is required before any public deployment. 3. **Test coverage is the CI4 skeleton only.** `tests/` contains the framework's example tests and one health check. There are no tests for the service layer, the filters, or the API. `phpstan.neon` and `psalm.xml` are configured and are the current safety net. 4. **`AutomationController` diverges from the API pattern** — it extends `ResourceController` and uses `respond()` while every other v1 controller extends `BaseApiController`. Follow `BaseApiController` for new endpoints. 5. **Staging runs as `development`.** `public/index.php` maps `staging.webcrawlers.com` to the `development` environment, so error display and the toolbar behave as they do in dev. Intentional or not, it is worth knowing before debugging there. 6. **Root-level markdown has accumulated.** Roughly thirty design and status documents sit in the repo root, many superseded. [schema.md](schema.md) and this file are the two kept current; treat the rest as point-in-time notes. 7. **`.gitignore` contains `*.md`.** Already-tracked files like this one and `schema.md` are unaffected, but a **new** markdown file will be silently ignored by `git add` and needs `git add -f`. This is a likely reason documentation has drifted from the code. --- ## Related Documents | Document | What it is | Current? | |---|---|---| | [schema.md](schema.md) | Full database reference + staging/production deploy runbook | Yes — authoritative | | [AUTOMATION_PHASE_2_CRON_SETUP.md](AUTOMATION_PHASE_2_CRON_SETUP.md) | Cron setup for the automation queue | Yes | | `dev-data/README.md` | Dev database dump and how to regenerate it | Yes | | [DATAFORSEO_VS_GOOGLE_APIS_ANALYSIS.md](DATAFORSEO_VS_GOOGLE_APIS_ANALYSIS.md) | Build-vs-defer analysis for Google APIs | Historical rationale | | `TASKS.md`, `JIRA_TICKETS.md` | Original per-ticket task breakdown | Partly superseded | | `CSRF_SECURITY_TEST_REPORT.md`, `PHP82_*.md`, `SUB4_*.md`, `WEBSITE_FLOW_*.md`, `ONBOARDING_*.md`, `REDIRECT_*.md`, etc. | Point-in-time design and review notes | Historical |