# TASKS.md — WebCrawlers SEO Automation Platform

> **Tech Stack:** PHP 8.2 · CodeIgniter 4 · MySQL · DB Queue (no Redis) · OpenAI · DataForSEO · Google APIs · Own Crawler  
> **⚠️ No sudo access** — Redis and Supervisor are not available. All background processing uses a DB-based queue + user-level cron.
> **Legend:** `[x]` = done · `[ ]` = pending

---

## SUB-22 · Contact Us — Form Submission & Confirmation

### Backend
- [ ] Create `wc_leads` table migration: `id, name, email, phone, company, message, source, status, created_at`
- [ ] Create `LeadModel.php` with insert + duplicate-check (by email + source within 24h)
- [ ] Create `Lead` controller: `POST /contact` endpoint with CSRF validation
- [ ] Validate all fields server-side (required fields, email format, max length)
- [ ] Store submission to `wc_leads` table with `source = 'contact'`
- [ ] Add CRM integration stub: `CRMService::push($lead)` (HubSpot / Freshdesk — confirm which)
- [ ] Return JSON: `{ success: true }` or `{ success: false, errors: [...] }`

### Frontend
- [ ] Wire contact form to `POST /contact` via AJAX/fetch
- [ ] Show success pop-up on `{ success: true }` response
- [ ] Show inline error messages on validation failure
- [ ] Show generic error message + retry on server/network failure
- [ ] Disable submit button during request to prevent double-submit

---

## SUB-23 · Book Demo — Form Submission & Confirmation

### Backend
- [ ] Reuse `wc_leads` table; add `demo_date, demo_time, company_size` columns via migration
- [ ] Create `POST /book-demo` endpoint in `Lead` controller
- [ ] Validate fields server-side (required fields, valid date/time, email format)
- [ ] Store submission with `source = 'book_demo'`
- [ ] Duplicate check: same email + `source = 'book_demo'` within 24h
- [ ] Push to CRM via `CRMService::push($lead)`

### Frontend
- [ ] Wire book-demo form to `POST /book-demo` via AJAX/fetch
- [ ] Show success pop-up on successful submission
- [ ] Show inline validation errors
- [ ] Show error + retry on failure
- [ ] Disable submit during request

---

## SUB-4 · Homepage & Project Onboarding (Logged-In)

### Auth & Routing
- [x] Create `AuthFilter` — redirects unauthenticated users to `/signin`
- [x] Create `ApiAuthFilter` — validates Bearer JWT or session for `/api/v1/*`
- [x] Register `auth` + `apiauth` aliases in `Filters.php`
- [x] Register `/dashboard/*` route group with `AuthFilter` in `Routes.php`
- [x] Redirect to `/dashboard` after successful login in `Auth.php`

### Dashboard Shell (Layout)
- [x] Create `app/Views/dashboard/layout/top.php` — full sidebar + topbar HTML shell
- [x] Create `app/Views/dashboard/layout/bottom.php` — shell close + `wcFetch()` JS helper
- [x] Create `app/Views/dashboard/layout/nav.php` — sidebar nav with `site_url()` hrefs
- [x] Create `app/Views/dashboard/layout/components.php` — `dash_*()` render helpers
- [x] Create `app/Views/dashboard/stub.php` — auto-shown for unbuilt pages (no crash)
- [x] Dashboard controller `render()` with stub fallback

### Dashboard Command Centre Page
- [x] Create `app/Views/dashboard/index.php` — KPI tiles, pipeline health, recent activity
- [x] `GET /api/v1/dashboard` — `DashboardController::index()` (stub, returns null KPIs)
- [ ] Replace stub data with real GSC + crawl data once integrations are built

### Homepage Features (pending)
- [ ] Build AI assistant prompt box (input → `POST /api/v1/ai/chat`)
- [ ] Add Recommended Playbooks section with 3–4 action cards
- [ ] Build Active Projects list on homepage with domain, status, score
- [ ] Add search, status filter, sort controls to projects list
- [ ] Build "Create Project" modal (delegates to SUB-5 onboarding wizard)
- [ ] Handle success, validation, error, duplicate project states

---

## SUB-5 · SEO Automation Page — Create and Onboard a Website Project

> **Reuse:** `WebsiteModel`, `AuditSyncService::start()`, `AuditModel`, `DataForSeo` library, `PlanAccess`, DFS webhook pingback — all already built.

### Database
- [ ] Migration: add `pixel_code VARCHAR(64)`, `pixel_verified_at DATETIME NULL`, `automation_enabled TINYINT(1) DEFAULT 0` to `wc_websites`
- [ ] Migration: `wc_website_integrations` — `id, website_id, user_id, type (gsc|ga4|gbp), status (connected|disconnected|pending), property_id, property_name, data(JSON), connected_at, created_at, updated_at`

### API Controller (`app/Controllers/Api/V1/WebsiteController.php`)

**Websites list + summary (powers the table and summary strip)**
- [ ] `GET /api/v1/websites` — list user's websites joined with latest `wc_audits` row; return domain, pixel status, automation_enabled, healthy_pages, onpage_score, subscores (content/authority/technical/ux), last finished_at, audit status; plus summary counts: active, processing, needs_attention, avg_score
- [ ] Support query params: `search` (domain), `status` filter, `sort` (score|date|domain), `page`, `per_page`

**Plan limits (powers the wizard header chip and page cap presets)**
- [ ] `GET /api/v1/websites/plan` — return `{ plan_name, sites_cap, sites_used, sites_left, pages_cap }` using `PlanAccess::sessionPlan()` + count of user's `wc_websites` rows

**Wizard Step 1 — validate domain**
- [ ] `POST /api/v1/websites/validate` — strip protocol/path to get bare domain; HTTP HEAD request (5s timeout, follow redirects); return `{ reachable: bool, domain, title, favicon_url, error? }`; reject duplicate domain for this user

**Wizard completion — create project**
- [ ] `POST /api/v1/websites` — body: `{ url, max_crawl_pages, integrations: [{type,property_id,property_name}], business: {name,location,services,competitors}, pixel_install_method }`:
  1. Call `WebsiteModel::findOrCreateForUser()` with domain + crawl config
  2. Generate `pixel_code = strtolower(bin2hex(random_bytes(8)))` (unique 16-char), store on website row
  3. Save integrations to `wc_website_integrations` (status = connected/pending)
  4. Call `AuditSyncService::start()` with website id → queues DFS OnPage task
  5. Return `{ website_id, audit_id, pixel_code, pixel_snippet: '<script>...' }` — idempotent (second POST same domain updates, does not duplicate)

**Row actions**
- [ ] `POST /api/v1/websites/{id}/scan` — create new audit via `AuditSyncService::start()` (re-crawl)
- [ ] `POST /api/v1/websites/{id}/verify-pixel` — HTTP GET domain HTML, search for pixel_code string; update `pixel_verified_at` if found; return `{ verified: bool }`
- [ ] `PUT /api/v1/websites/{id}` — update `automation_enabled`, `max_crawl_pages`, integrations
- [ ] `DELETE /api/v1/websites/{id}` — soft-delete or hard-delete website + audits

### Routes (`app/Config/Routes.php`)
- [ ] Register all website endpoints inside `api/v1` group (after existing reports routes)
- [ ] `GET api/v1/websites/plan` must be registered **before** `GET api/v1/websites/(:num)` to avoid route collision

### Dashboard Controller (`app/Controllers/Dashboard.php`)
- [ ] Update `seoAutomation()` to pass plan data to view: `$plan = PlanAccess::sessionPlan()`, sites_used count, `SubscriptionPlans::label($plan)`

### CI4 View (`app/Views/dashboard/seo-automation.php`)
- [ ] Create from design `public/webcrawlers-dashboard-assets/seo-automation.php`
- [ ] Replace `$dash_plan`, `$dash_sites_used`, `$dash_sites_cap`, `$dash_sites_left` with controller-passed variables
- [ ] `[data-sa-root]` attributes: `data-empty`, `data-pagecap`, `data-planname`, `data-sitesleft` — populated from controller
- [ ] `$pagecap` → from plan's `pages_cap` (default 10000 if enterprise/null)
- [ ] `$page_actions` chip and "Create project" button — replace `$sites_txt` with controller variable
- [ ] Replace `dash_status(...)` with `<?= dash_status(...) ?>` (available via components.php) 
- [ ] The table body `[data-sa-tbody]`, summary `[data-sum-*]` and pagination are JS-rendered from `wcFetch('websites')` — no server-side data needed in view
- [ ] Wizard HTML: copy as-is — Step 1/2/3/Done panels, all driven by `assets/js/seo-automation.js` which calls `wcFetch('websites/validate')`, `wcFetch('websites', {method:'POST',...})` etc.

### JS Wiring Check (no JS changes needed — verify these `wcFetch` calls match routes)
- [ ] Confirm `seo-automation.js` calls `wcFetch('websites')` for table load ← maps to `GET /api/v1/websites`
- [ ] Confirm `wcFetch('websites/validate', {method:'POST',...})` ← maps to `POST /api/v1/websites/validate`
- [ ] Confirm `wcFetch('websites', {method:'POST',...})` for project creation ← maps to `POST /api/v1/websites`
- [ ] Confirm `wcFetch('websites/plan')` for plan chip data ← maps to `GET /api/v1/websites/plan`
- [ ] Confirm `wcFetch('websites/{id}/scan', {method:'POST'})`, `verify-pixel`, `PUT`, `DELETE`

---

## SUB-9 · Site Audit Reports

### Database
- [ ] Migration: `wc_issues` — `id, crawl_run_id, url, type, severity (critical|high|medium|low), description, recommendation, status (open|ignored|resolved), created_at`
- [ ] Migration: `wc_audit_reports` — `id, project_id, crawl_run_id, health_score, total_pages, total_issues, generated_at`

### Backend — Crawl Engine
- [ ] Implement `CrawlEngine` in `app/Libraries/` — fetch URL, follow links within domain, respect robots.txt, respect crawl limit
- [ ] Extract per-page: status code, title, meta description, H1–H6, canonical, robots meta, schema, internal/external links, image alt texts
- [ ] Store each page in `wc_crawl_pages`

### Backend — Audit Analyzer
- [ ] Create `AuditAnalyzerService.php`
- [ ] Detect issues: missing/duplicate titles & meta, missing/multiple H1, missing alt, 404s, redirect chains, broken links, orphan pages, missing canonical, thin content (<300 words), missing schema
- [ ] Calculate health score
- [ ] `GET /api/v1/audit/report/{project_id}` — latest report + grouped issues
- [ ] `GET /api/v1/audit/history/{project_id}` — list all past runs
- [ ] `POST /api/v1/audit/start` — queue crawl + audit job
- [ ] `PUT /api/v1/audit/issue/{id}` — update issue status

### Frontend — Audit Dashboard View
- [ ] Create `app/Views/dashboard/issues.php` from design `public/webcrawlers-dashboard-assets/issues.php`
- [ ] Health score widget + issue count by severity
- [ ] Issue list: name, severity badge, affected page count, status
- [ ] Filter by severity, issue type, status
- [ ] Expandable rows: description, affected pages, recommended fix
- [ ] Mark resolved / ignore per issue
- [ ] Compare with previous audit view
- [ ] Progress indicator during active crawl
- [ ] Export (CSV)

---

## SUB-6 · Content Scoring & Briefs

### Database
- [ ] Migration: `wc_content_analyses` — `id, user_id, project_id, keyword, content(TEXT), score, brief(JSON), suggestions(JSON), created_at`
- [ ] Migration: `wc_content_briefs` — `id, analysis_id, h1, h2s(JSON), h3s(JSON), word_count, keywords(JSON), topics(JSON), faqs(JSON), guidelines(TEXT), exported_at`

### Backend
- [ ] `POST /api/v1/content/analyze` — accept keyword + content, return score + suggestions
- [ ] `ContentScorerService.php` — score on keyword usage, readability, headings, word count, topic coverage, intent (0–100)
- [ ] `ContentBriefService.php` — generate brief via OpenAI: headings, keywords, topics, FAQs, word count
- [ ] Cache brief by keyword hash in DB
- [ ] `POST /api/v1/content/brief` — generate / retrieve brief
- [ ] `GET /api/v1/content/brief/{id}/export` — return as JSON/HTML

### Frontend
- [ ] Create `app/Views/dashboard/content-scoring.php` from design `public/webcrawlers-dashboard-assets/content-scoring.php`
- [ ] Primary keyword input + content paste/type area
- [ ] Live Content Score gauge (0–100), debounced 800ms
- [ ] Optimization suggestions panel
- [ ] Generated brief panel: keywords, headings, word count, topics, FAQs, guidelines
- [ ] Copy Brief + Export Brief buttons

---

## SUB-21 · AI Content Writer

### Database
- [ ] Migration: `wc_content_drafts` — `id, user_id, project_id, brief_id, title, content(LONGTEXT), keyword, status (draft|approved|published), version, created_at, updated_at`

### Backend
- [ ] `POST /api/v1/ai/write` — generate full article from brief via OpenAI
- [ ] `POST /api/v1/ai/rewrite-section` — rewrite a selection
- [ ] `POST /api/v1/ai/expand` — expand a section
- [ ] `POST /api/v1/ai/shorten` — shorten a section
- [ ] `POST /api/v1/ai/change-tone` — Professional / Friendly / Conversational
- [ ] `POST /api/v1/content/draft/save` — save with version increment
- [ ] `GET /api/v1/content/draft/{id}` — retrieve draft

### Frontend
- [ ] Create `app/Views/dashboard/ai-content.php` from design `public/webcrawlers-dashboard-assets/ai-content-generation.php`
- [ ] Brief panel + content canvas
- [ ] Generate Article, Regenerate Article buttons
- [ ] Inline AI controls per section: Rewrite, Expand, Shorten, Simplify, Change Tone
- [ ] Tone selector, Save Draft button
- [ ] Loading/streaming states

---

## SUB-7 · Keyword Research & Gap Analysis

### Database
- [ ] Migration: `wc_keyword_searches` — `id, user_id, keyword, country, results(JSON), created_at`
- [ ] Migration: `wc_saved_keywords` — `id, user_id, project_id, keyword, volume, difficulty, intent, cpc, created_at`
- [ ] Migration: `wc_keyword_gap_runs` — `id, user_id, our_domain, competitors(JSON), results(JSON), created_at`

### Backend
- [ ] `POST /api/v1/keywords/search` — DataForSEO keyword ideas, cache 24h
- [ ] `POST /api/v1/keywords/save` — save selected keywords
- [ ] `GET /api/v1/keywords/saved` — list saved with filter/sort/pagination
- [ ] `POST /api/v1/keywords/gap` — DataForSEO domain vs domain, classify missing/shared/unique
- [ ] `GET /api/v1/keywords/gap/{id}` — retrieve gap results

### Frontend
- [ ] Create `app/Views/dashboard/keyword-research.php` from design `public/webcrawlers-dashboard-assets/keyword-research.php`
- [ ] Keyword input + country selector, results table, filter/sort
- [ ] Select → Save / Send to Brief / Create Opportunity
- [ ] Gap analysis: domain + up to 5 competitors, Missing/Shared/Unique tabs
- [ ] Export CSV

---

## SUB-8 · Rank Tracking

### Database
- [ ] Migration: `wc_tracked_keywords` — `id, project_id, keyword, landing_page, country, device, search_engine, is_active, created_at`
- [ ] Migration: `wc_keyword_rankings` — `id, tracked_keyword_id, position, previous_position, change, crawled_at`
- [ ] Migration: `wc_ranking_competitors` — `id, project_id, domain`
- [ ] Migration: `wc_competitor_rankings` — `id, tracked_keyword_id, competitor_id, position, crawled_at`
- [ ] Migration: `wc_ranking_alerts` — `id, project_id, tracked_keyword_id, alert_type, triggered_at, sent_at`

### Backend
- [ ] `POST /api/v1/rank-tracking/keywords` — add tracked keywords
- [ ] `DELETE /api/v1/rank-tracking/keywords/{id}` — remove/archive
- [ ] `GET /api/v1/rank-tracking/keywords` — list with position, change, volume, intent
- [ ] `POST /api/v1/rank-tracking/competitors` — add competitor
- [ ] `GET /api/v1/rank-tracking/dashboard` — summary metrics
- [ ] `GET /api/v1/rank-tracking/history/{keyword_id}` — daily/weekly/monthly history
- [ ] `GET /api/v1/rank-tracking/export` — CSV export
- [ ] `RankCheckJob.php` — daily cron, DataForSEO SERP API, store positions
- [ ] Alert engine: Top 10/3 entry, significant drop, competitor overtake, lost ranking

### Frontend
- [ ] Create `app/Views/dashboard/rank-tracking.php` from design `public/webcrawlers-dashboard-assets/rank-tracking.php`
- [ ] Dashboard metrics strip, tracked keywords table, filter panel
- [ ] Ranking history chart per keyword
- [ ] Competitor comparison table
- [ ] Alerts list, Export button

---

## SUB-10 · AI Content Changes

### Database
- [ ] Migration: `wc_content_change_jobs` — `id, project_id, page_url, status, current_content(LONGTEXT), ai_suggestions(JSON), accepted_changes(JSON), draft_content(LONGTEXT), created_at, updated_at`
- [ ] Migration: `wc_content_change_audit` — `id, change_job_id, user_id, action, change_type, before(TEXT), after(TEXT), timestamp`

### Backend
- [ ] `POST /api/v1/content-changes/analyze` — fetch page, call OpenAI for SEO suggestions
- [ ] `AIContentChangeService.php` — structured suggestions per element (title, meta, H1-H6, paragraphs, FAQs, internal links, schema), each with type/reason/impact/confidence
- [ ] `POST /api/v1/content-changes/save-draft` — save accepted/edited changes
- [ ] `POST /api/v1/content-changes/submit` — submit to publishing workflow
- [ ] Log all accept/reject/edit actions to audit table

### Frontend
- [ ] Create `app/Views/dashboard/ai-content-analyze.php` from design `public/webcrawlers-dashboard-assets/ai-content-analyze.php`
- [ ] Page selector from project crawled pages
- [ ] Side-by-side diff view: Current | AI Suggested
- [ ] Per-suggestion: Accept / Reject / Edit, type/reason/impact/confidence
- [ ] Accept All, Save Draft, Submit for Publishing buttons
- [ ] Audit trail view

---

## Infrastructure Tasks

> ⚠️ No sudo access — no apt/brew/npm -g. All background processing uses DB queue + user-level cron.

### Queue (DB-based)
- [ ] Migration: `wc_queue_jobs` — `id, type, payload(JSON), status (pending|running|done|failed), attempts, run_at, created_at, updated_at`
- [ ] Create `QueueService.php`: `push(type, payload)` → insert row; `pop(type)` → `SELECT … FOR UPDATE` claim
- [ ] Create `app/Commands/QueueWork.php` — `spark queue:work` CLI command
- [ ] Create `app/Commands/RankCheck.php` — `spark rank:check` CLI command

### Cron (user-level, no sudo)
- [ ] `crontab -e`: `* * * * * php /var/www/php82/<user>/webcrawlers_latest.com/spark queue:work`
- [ ] `crontab -e`: `0 2 * * * php /var/www/php82/<user>/webcrawlers_latest.com/spark rank:check`

### Auth & API
- [x] `AuthFilter` — session guard for `/dashboard/*`
- [x] `ApiAuthFilter` — JWT Bearer or session for `/api/v1/*`
- [x] `app/Libraries/Jwt.php` — HS256 sign/verify, no external library
- [x] `WC_API_TOKENS` — pre-generated JWTs for external clients in `Constants.php`
- [x] `WC_ENCRYPTION_KEY` — signs JWT tokens, set in `Constants.php`
- [x] Environment auto-detection in `public/index.php` (dev/staging/production)

---

## Shared / Cross-Cutting Tasks

- [x] `WC_*` constants for all API keys in `app/Config/Constants.php`
- [x] `app/Config/Encryption.php` reads `WC_ENCRYPTION_KEY`
- [x] CI_ENVIRONMENT auto-set before CI4 boots (dev sandbox / staging / production)
- [x] Debug toolbar disabled globally
- [x] `GET /api/v1/dashboard` stub endpoint
- [x] `/api/docs` one-page API reference
- [ ] Create `CRMService.php` — HubSpot or Freshdesk (confirm vendor)
- [ ] Create `GoogleService.php` — OAuth2 + GSC / GA4 / GBP API wrappers
- [ ] Create `AIService.php` — OpenAI GPT-4o wrapper with retry
- [ ] Create `DataForSEOService.php` — HTTP client for DataForSEO API
- [ ] Add rate limiting filter for AI and external API endpoints
- [ ] Write all DB migrations listed above


---

## SUB-22 · Contact Us — Form Submission & Confirmation

### Backend
- [ ] Create `leads` table migration: `id, name, email, phone, company, message, source, status, created_at`
- [ ] Create `LeadModel.php` with insert + duplicate-check (by email + source within 24h)
- [ ] Create `Lead` controller: `POST /contact` endpoint with CSRF validation
- [ ] Validate all fields server-side (required fields, email format, max length)
- [ ] Store submission to `leads` table with `source = 'contact'`
- [ ] Add CRM integration stub: `CRMService::push($lead)` (HubSpot / Freshdesk — confirm which)
- [ ] Return JSON: `{ success: true }` or `{ success: false, errors: [...] }`

### Frontend
- [ ] Wire contact form to `POST /contact` via AJAX/fetch
- [ ] Show success pop-up on `{ success: true }` response
- [ ] Show inline error messages on validation failure
- [ ] Show generic error message + retry on server/network failure
- [ ] Disable submit button during request to prevent double-submit

---

## SUB-23 · Book Demo — Form Submission & Confirmation

### Backend
- [ ] Reuse `leads` table; add `demo_date, demo_time, company_size` columns via migration
- [ ] Create `POST /book-demo` endpoint in `Lead` controller
- [ ] Validate fields server-side (required fields, valid date/time, email format)
- [ ] Store submission with `source = 'book_demo'`
- [ ] Duplicate check: same email + `source = 'book_demo'` within 24h
- [ ] Push to CRM via `CRMService::push($lead)`

### Frontend
- [ ] Wire book-demo form to `POST /book-demo` via AJAX/fetch
- [ ] Show success pop-up on successful submission
- [ ] Show inline validation errors
- [ ] Show error + retry on failure
- [ ] Disable submit during request

---

## SUB-4 · Homepage & Project Onboarding (Logged-In)

### Database
- [ ] Create `projects` table: `id, user_id, domain, country, status, created_at, updated_at`
- [ ] Create `project_settings` table: `project_id, setting_key, setting_value`
- [ ] Create `project_integrations` table: `project_id, type (gsc|ga4|gbp), status, data(JSON), connected_at`

### Backend — Auth Guard
- [ ] Create `AuthFilter` in `app/Filters/` — redirect unauthenticated users to login
- [ ] Register filter for all `/dashboard*` routes in `Routes.php`

### Backend — Dashboard API
- [ ] Create `GET /dashboard` endpoint returning: user info, project count, recent activity summary
- [ ] Create `GET /projects` endpoint with search, status filter, sort, pagination
- [ ] Create `POST /projects` endpoint: validate domain, check duplicates, create project record
- [ ] Domain validation: check valid URL format + DNS resolution (no external request if blocked)
- [ ] Return quota utilisation data from user plan

### Backend — Project Modal
- [ ] Validate domain uniqueness per user
- [ ] Accept: `domain, target_country, ai_keywords (bool), ai_competitors (bool)`
- [ ] On success: return new project ID + trigger onboarding job (queue or sync stub)

### Frontend
- [ ] Build left navigation: Home, Technical Audit, AI Visibility, Content, Keywords, Reports, Authority
- [ ] Add global search bar (client-side filter across tools/projects)
- [ ] Build AI assistant prompt box (input → `POST /ai/chat`)
- [ ] Add Recommended Playbooks section with 3–4 action cards
- [ ] Build Active Projects list: card/table with domain, status, score
- [ ] Add search, status filter, sort controls to projects list
- [ ] Build "Create Project" modal: domain input, country selector, AI toggles, quota indicator
- [ ] Handle project loading/processing state (project created but data pending)
- [ ] Show empty state when no projects exist
- [ ] Handle success, validation, error, and duplicate project states

---

## SUB-5 · Create and Onboard an SEO Automation Site

### Database
- [ ] Create `crawl_runs` table: `id, project_id, status, pages_limit, pages_crawled, started_at, completed_at`
- [ ] Create `crawl_pages` table: `id, crawl_run_id, url, status_code, title, meta_desc, h1, canonical, schema(JSON), links(JSON), crawled_at`
- [ ] Create `crawl_errors` table: `id, crawl_run_id, url, error_type, message`

### Backend — Onboarding Flow
- [ ] `POST /projects/validate-domain` — check domain accessibility (HTTP HEAD, timeout 5s)
- [ ] `POST /projects/setup` (Step 2) — save GSC, GA4, GBP connection preferences
- [ ] `POST /projects/install` (Step 3) — generate unique pixel code (UUID-based), store against project
- [ ] `POST /projects/verify-pixel` — check if pixel snippet is live on the domain
- [ ] `POST /projects/complete` — finalise project, set status, queue initial crawl job
- [ ] Idempotency guard: prevent duplicate project creation (check by user_id + domain)
- [ ] Enqueue background jobs: `crawl_site`, `sync_search_console`, `generate_scores`

### Background Workers (DB queue + cron — no Redis/Supervisor needed)
- [ ] Create `app/Jobs/CrawlSiteJob.php` — crawl a domain, store pages + errors
- [ ] Create `app/Jobs/GenerateScoresJob.php` — compute scores from crawl data

### SEO Automation Dashboard
- [ ] Build dashboard table: domain, install status, engagement status, healthy pages, grader score, subscores, last analysis date
- [ ] Add scan action, automation toggle, project action menu per row
- [ ] Add search, status filter, sort, pagination controls
- [ ] Add processing/empty/failed/disconnected/retry states

### Frontend — 3-Step Onboarding
- [ ] Step 1: URL input + validation UI + crawl page limit selector
- [ ] Step 1: Show loading state while domain info is fetched
- [ ] Step 2: Display validated URL + connect GSC/GBP/GA4 buttons
- [ ] Step 2: Prefill available business info; show warnings for missing fields
- [ ] Step 3: Display generated pixel code + WordPress install instructions
- [ ] Step 3: Verify installation button + status indicator
- [ ] Step 3: Allow "Create with Not Installed" option
- [ ] Success screen with confirmation message

---

## SUB-9 · Site Audit Reports

### Database
- [ ] Add `issues` table: `id, crawl_run_id, url, type, severity (critical|high|medium|low), description, recommendation, status (open|ignored|resolved), created_at`
- [ ] Add `audit_reports` table: `id, project_id, crawl_run_id, health_score, total_pages, total_issues, generated_at`

### Backend — Crawl Engine
- [ ] Implement `CrawlEngine` library/service in `app/Libraries/` or `app/Services/`
- [ ] Crawl: fetch URL, follow links within domain, respect robots.txt, respect crawl limit
- [ ] Extract per-page: status code, title, meta description, H1–H6, canonical, robots meta, schema, internal/external links, image alt texts
- [ ] Store each page in `crawl_pages`

### Backend — Audit Analyzer
- [ ] Create `AuditAnalyzerService.php`
- [ ] Detect and store issues: missing/duplicate titles, missing/duplicate meta, missing H1, multiple H1, missing alt text, 404s, redirect chains, broken internal/external links, orphan pages, missing canonical, thin content (<300 words), missing schema
- [ ] Calculate and store overall health score
- [ ] Create `GET /audit/report/{project_id}` — return latest audit report + grouped issues
- [ ] Create `GET /audit/history/{project_id}` — list all past audit runs
- [ ] Create `POST /audit/start` — queue new crawl+audit job
- [ ] Create `PUT /audit/issue/{id}` — update issue status (open/ignored/resolved)

### Frontend — Audit Dashboard
- [ ] Health Score widget + issue count breakdown by severity
- [ ] Issue list with: name, severity badge, affected page count, status
- [ ] Filter by severity, issue type, status
- [ ] Expandable issue rows showing: description, affected pages, recommended fix
- [ ] Mark as resolved / ignore actions per issue
- [ ] Compare with previous audit view
- [ ] Audit history list
- [ ] Progress indicator during crawl
- [ ] Export audit report (CSV/PDF)

---

## SUB-6 · Content Scoring & Briefs

### Database
- [ ] Create `content_analyses` table: `id, user_id, project_id, keyword, content(TEXT), score, brief(JSON), suggestions(JSON), created_at`
- [ ] Create `content_briefs` table: `id, analysis_id, h1, h2s(JSON), h3s(JSON), word_count, keywords(JSON), topics(JSON), faqs(JSON), guidelines(TEXT), exported_at`

### Backend
- [ ] Create `POST /content/analyze` — accept keyword + content, return score + suggestions
- [ ] `ContentScorerService.php`: score content on keyword usage, readability, headings, word count, topic coverage, intent alignment (0–100)
- [ ] `ContentBriefService.php`: generate brief using OpenAI (GPT-4) — headings, keywords, topics, FAQs, word count recommendation
- [ ] Cache brief by keyword in DB (`content_briefs` table, keyed by keyword hash) to avoid repeat API calls
- [ ] Create `POST /content/brief` — generate/retrieve brief for a keyword
- [ ] Create `GET /content/brief/{id}/export` — return brief as JSON/HTML for download

### Frontend
- [ ] Build content editor with primary keyword input + paste/type area
- [ ] Display live Content Score gauge (0–100) updating as user types (debounced, 800ms)
- [ ] Show optimization suggestions panel (missing keywords, missing headings, readability, word count, FAQs)
- [ ] Display generated Content Brief: primary keyword, secondary keywords, recommended headings, word count, topics, FAQs, guidelines
- [ ] "Copy Brief" button
- [ ] "Export Brief" button (download JSON or formatted text)
- [ ] Handle loading and error states for AI generation

---

## SUB-21 · AI Content Writer

### Database
- [ ] Create `content_drafts` table: `id, user_id, project_id, brief_id, title, content(LONGTEXT), keyword, status (draft|approved|published), version, created_at, updated_at`

### Backend
- [ ] Create `POST /ai/write` — generate full article from brief using OpenAI, stream response or return full
- [ ] Create `POST /ai/rewrite-section` — rewrite a selected section (accepts: section_content, instructions)
- [ ] Create `POST /ai/expand` — expand a section
- [ ] Create `POST /ai/shorten` — shorten a section
- [ ] Create `POST /ai/change-tone` — change tone (Professional / Friendly / Conversational)
- [ ] Ensure generated content uses target keywords naturally, follows heading structure, avoids keyword stuffing
- [ ] Create `POST /content/draft/save` — save draft with version increment
- [ ] Create `GET /content/draft/{id}` — retrieve draft

### Frontend
- [ ] Build AI content editor interface: brief panel + content canvas
- [ ] "Generate Article" button — streams/inserts AI content
- [ ] Inline AI controls per section: Rewrite, Expand, Shorten, Simplify, Improve Readability, Change Tone
- [ ] Tone selector dropdown
- [ ] "Regenerate Article" button
- [ ] Manual editing of generated content
- [ ] "Save Draft" button
- [ ] Show loading/streaming states during generation

---

## SUB-7 · Keyword Research & Gap Analysis

### Database
- [ ] Create `keyword_searches` table: `id, user_id, keyword, country, results(JSON), created_at`
- [ ] Create `saved_keywords` table: `id, user_id, project_id, keyword, volume, difficulty, intent, cpc, created_at`
- [ ] Create `keyword_gap_runs` table: `id, user_id, our_domain, competitors(JSON), results(JSON), created_at`

### Backend — Keyword Research
- [ ] Create `POST /keywords/search` — accept keyword + country, call DataForSEO Keywords for Sites or Keyword Ideas API
- [ ] Map DataForSEO response to: keyword, monthly volume, difficulty, intent, CPC, competition, traffic potential
- [ ] Cache results in DB for 24h to reduce API calls
- [ ] Create `POST /keywords/save` — save selected keywords to `saved_keywords`
- [ ] Create `GET /keywords/saved` — list saved keywords with filter/sort/pagination

### Backend — Gap Analysis
- [ ] Create `POST /keywords/gap` — accept our domain + up to 5 competitor domains, call DataForSEO Domain vs Domain or keyword intersection API
- [ ] Classify results: missing (competitor ranks, we don't), shared, unique
- [ ] Store results in `keyword_gap_runs`
- [ ] Create `GET /keywords/gap/{id}` — retrieve gap analysis results

### Frontend — Keyword Research
- [ ] Keyword input + country selector
- [ ] Results table: keyword, volume, difficulty, intent badge, CPC, competition, traffic potential
- [ ] Filter panel: volume range, difficulty range, intent multi-select
- [ ] Sort by any column
- [ ] Checkbox to select keywords → Save / Send to Brief / Create Opportunity
- [ ] Export (CSV) button

### Frontend — Gap Analysis
- [ ] Domain input + up to 5 competitor inputs
- [ ] Results tabs: Missing / Shared / Unique
- [ ] Columns: keyword, rank, volume, difficulty, intent, traffic potential
- [ ] Filter by volume, difficulty, intent, position
- [ ] Select + save / export actions
- [ ] Error states: invalid domain, not found, duplicate competitor

---

## SUB-8 · Rank Tracking

### Database
- [ ] Create `tracked_keywords` table: `id, project_id, keyword, landing_page, country, device (desktop|mobile), search_engine, is_active, created_at`
- [ ] Create `keyword_rankings` table: `id, tracked_keyword_id, position, previous_position, change, crawled_at`
- [ ] Create `ranking_competitors` table: `id, project_id, domain`
- [ ] Create `competitor_rankings` table: `id, tracked_keyword_id, competitor_id, position, crawled_at`
- [ ] Create `ranking_alerts` table: `id, project_id, tracked_keyword_id, alert_type, triggered_at, sent_at`

### Backend
- [ ] Create `POST /rank-tracking/keywords` — add tracked keywords (manual + import from keyword research / gap / brief)
- [ ] Create `DELETE /rank-tracking/keywords/{id}` — remove/archive keyword
- [ ] Create `GET /rank-tracking/keywords` — list tracked keywords with current position, change, volume, intent
- [ ] Create `POST /rank-tracking/competitors` — add competitor domain
- [ ] Create `GET /rank-tracking/dashboard` — aggregate metrics: total, avg rank, top 3/10/20, improved, declined
- [ ] Create `GET /rank-tracking/history/{keyword_id}` — daily/weekly/monthly ranking history
- [ ] `RankCheckJob.php` — daily cron: query DataForSEO SERP API for each tracked keyword, store position
- [ ] Alert engine: detect Top 10 entry, Top 3 entry, significant drop (configurable), competitor overtake, lost ranking → insert `ranking_alerts`
- [ ] Create `GET /rank-tracking/export` — export rankings as CSV

### Frontend
- [ ] Dashboard metrics: Total, Avg Rank, Top 3, Top 10, Top 20, Improved, Declined, New, Lost
- [ ] Tracked keywords table: keyword, landing page, current rank, previous rank, change badge, volume, intent, last updated
- [ ] Filter panel: keyword search, position range, change direction, intent, device, location, competitor, date range
- [ ] Ranking history chart per keyword (line chart)
- [ ] Competitor comparison table per keyword
- [ ] Add keywords form (manual + import from other modules)
- [ ] Alerts list / notification badges
- [ ] Export button

---

## SUB-10 · AI Content Changes

### Database
- [ ] Create `content_change_jobs` table: `id, project_id, page_url, status, current_content(LONGTEXT), ai_suggestions(JSON), accepted_changes(JSON), draft_content(LONGTEXT), created_at, updated_at`
- [ ] Create `content_change_audit` table: `id, change_job_id, user_id, action (accepted|rejected|edited|submitted), change_type, before(TEXT), after(TEXT), timestamp`

### Backend
- [ ] Create `POST /content-changes/analyze` — accept page URL, fetch current page content, call OpenAI for SEO improvements
- [ ] `AIContentChangeService.php`: analyze and return structured suggestions per element (title, meta, H1-H6, paragraphs, FAQs, internal links, schema)
- [ ] Each suggestion includes: type, reason, expected SEO impact, confidence level, supporting evidence
- [ ] Create `POST /content-changes/save-draft` — save accepted/edited changes as draft
- [ ] Create `POST /content-changes/submit` — submit approved draft to publishing workflow
- [ ] Log all accept/reject/edit actions to `content_change_audit`
- [ ] Handle: empty page content, unsupported format, AI timeout, missing business evidence

### Frontend
- [ ] Page selector (from project's crawled pages)
- [ ] Side-by-side diff view: Current Content | AI Suggested (highlight added/removed/modified)
- [ ] Per-suggestion action row: Accept / Reject / Edit manually
- [ ] Suggestion detail: type, reason, expected impact, confidence
- [ ] "Accept All" button
- [ ] "Save as Draft" button
- [ ] "Submit for Publishing" button
- [ ] Audit trail / history view for the page
- [ ] Loading, error, empty states

---

## Infrastructure Tasks

> ⚠️ No sudo access — Redis and Supervisor cannot be installed. All background processing uses a **DB-based queue + cron jobs** instead.

### Queue (DB-based — no Redis needed)
- [ ] Create `queue_jobs` table: `id, type, payload(JSON), status (pending|running|done|failed), attempts, run_at, created_at, updated_at`
- [ ] Create `QueueService.php`: `push(type, payload)` → inserts row; `pop(type)` → claims next pending row with row-level lock (`SELECT ... FOR UPDATE`)
- [ ] Create `app/Workers/BaseWorker.php` — CLI worker: polls `queue_jobs`, dispatches to job class, marks done/failed, retries up to 3x
- [ ] Create individual job classes: `CrawlSiteJob`, `GenerateScoresJob`, `GoogleSyncJob`, `KeywordJob`, `ReportJob`

### Cron Jobs (no Supervisor needed)
- [ ] Set up user-level crontab (`crontab -e`) — no sudo required
- [ ] Add cron: `* * * * * php /var/www/.../spark queue:work >> writable/logs/worker.log 2>&1` (process queue every minute)
- [ ] Add cron: `0 2 * * * php /var/www/.../spark rank:check` (daily rank tracking)
- [ ] Create `spark queue:work` CLI command in CodeIgniter (`app/Commands/QueueWork.php`)
- [ ] Create `spark rank:check` CLI command (`app/Commands/RankCheck.php`)

### Permissions
- [ ] Ensure `writable/` subdirs are writable by web user (no sudo needed if already owned by same user)

---

## Shared / Cross-Cutting Tasks

- [ ] Create `CRMService.php` — HubSpot/Freshdesk integration stub (confirm vendor)
- [ ] Create `GoogleService.php` — OAuth2 + GSC / GA4 / GBP API wrappers
- [ ] Create `AIService.php` — OpenAI GPT-4 wrapper with retry + error handling
- [ ] Create `DataForSEOService.php` — HTTP client wrapper for DataForSEO API
- [ ] Add environment config: `OPENAI_API_KEY`, `DATAFORSEO_LOGIN`, `DATAFORSEO_PASSWORD`, `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `HUBSPOT_API_KEY`
- [ ] Set up `.env` file with all keys (do not commit to version control)
- [ ] Write DB migrations for all new tables above
- [ ] Add CSRF protection to all POST endpoints
- [ ] Add rate limiting filter for AI and external API endpoints
- [ ] Add input sanitization/output encoding across all views