# SUB-4 Task Breakdown & Planning
## WebCrawlers Homepage & Project Onboarding

**Ticket ID:** SUB-4  
**Status:** Planning  
**Created:** 2026-08-17  
**Estimated Effort:** 8-10 weeks

---

## 📋 Overview

Develop the logged-in homepage for WebCrawlers providing:
- Navigation menu for key modules
- Global search functionality
- AI assistant interface
- Recommended playbooks/workflows
- Active Projects management with filtering, sorting, and creation
- Project creation modal with domain validation and onboarding flow

---

## 🎯 Main Components

### 1. **Left Navigation Menu** 
### 2. **Global Search Bar**
### 3. **AI Assistant Prompt Box**
### 4. **Recommended Playbooks Section**
### 5. **Active Projects Section**
### 6. **Project Creation Modal**
### 7. **Empty State Management**

---

## 📝 Detailed Task Breakdown

### PHASE 1: Foundation & Navigation (Weeks 1-2)

#### Task 1.1: Design & Create Dashboard Layout
- **Description:** Create the homepage template with left sidebar navigation
- **Subtasks:**
  - Create `app/Controllers/DashboardController.php` (if not exists)
  - Create `app/Views/dashboard/home.php` layout
  - Design responsive grid system (2-3 columns)
  - Implement sticky header and navigation
- **Dependencies:** None
- **Acceptance Criteria:**
  - Layout loads without errors
  - Responsive on mobile/tablet/desktop
  - Navigation sidebar visible and functional
  - All main sections have placeholder areas

#### Task 1.2: Build Left Navigation Menu
- **Description:** Create navigation menu for key modules
- **Subtasks:**
  - Build menu structure with links to:
    - Home
    - Technical Audit → Crawl Monitoring, Page Inventory, Issues
    - AI Visibility
    - Content → AI Content, Scoring, Briefs
    - Keywords → Keywords, Rank Tracking, Research
    - Reports
    - Site Overview (Authority)
  - Active module highlighting
  - Collapse/expand functionality for submenus
  - Mobile menu hamburger icon
  - Icon assets for each module
- **Dependencies:** Task 1.1
- **Acceptance Criteria:**
  - All modules accessible from navigation
  - Active state correctly highlights current page
  - Submenu expansion/collapse works
  - Mobile hamburger menu functional

#### Task 1.3: Create Route & Controller for Homepage
- **Description:** Set up backend for homepage
- **Subtasks:**
  - Add route in `app/Config/Routes.php`: `GET /dashboard → Dashboard::index`
  - Create `DashboardController::index()` method
  - Load user data from session
  - Load user's websites list
  - Render dashboard view with user context
  - Apply auth filter
- **Dependencies:** Task 1.1
- **Acceptance Criteria:**
  - Route accessible at `/dashboard`
  - User data available in view
  - Websites list loads
  - Auth filter prevents unauthorized access

---

### PHASE 2: Global Search & AI Assistant (Weeks 2-3)

#### Task 2.1: Design & Build Global Search Bar
- **Description:** Create searchable interface for tools, projects, keywords, domains
- **Subtasks:**
  - Design search input with icon
  - Create search dropdown/modal
  - Implement search categories (Projects, Keywords, Domains, Tools)
  - Styling with Tailwind CSS
  - Keyboard shortcuts (⌘K or Ctrl+K)
  - Clear button for mobile
  - Search history (optional)
- **Dependencies:** Task 1.2
- **Acceptance Criteria:**
  - Search bar visible in header
  - Categories displayed
  - Responsive on all screen sizes
  - Keyboard navigation works

#### Task 2.2: Create Search API Endpoint
- **Description:** Backend search functionality
- **Subtasks:**
  - Create `Api/V1/SearchController.php`
  - Add route: `POST /api/v1/search`
  - Search websites by domain
  - Search keywords by term
  - Search reports/audits
  - Pagination (10 items per category)
  - User permission filtering
- **Dependencies:** Task 2.1, existing models
- **Acceptance Criteria:**
  - API returns matching results
  - Results filtered by user_id
  - Pagination works
  - Response time < 200ms

#### Task 2.3: Build AI Assistant Prompt Box
- **Description:** Integrate AI assistant interface
- **Subtasks:**
  - Design prompt input box
  - Create AI assistant icon/branding
  - Suggested prompts carousel:
    - "Audit website for technical issues"
    - "Fix critical on-page issues"
    - "Discover content opportunities"
  - Text input with send button
  - Loading state for responses
  - Response display area (or modal)
  - Clear chat history option
- **Dependencies:** Task 1.1, existing AI service
- **Acceptance Criteria:**
  - Prompt box displays correctly
  - Suggested prompts clickable
  - Send button functional
  - Loading state shown
  - Responses display

#### Task 2.4: Create AI Assistant API Endpoint
- **Description:** Backend for AI request processing
- **Subtasks:**
  - Create `Api/V1/AiAssistantController.php`
  - Add route: `POST /api/v1/ai/chat`
  - Parse user prompt
  - Route to appropriate workflow:
    - "audit" → Trigger crawl
    - "fix issues" → Open Issues page with filters
    - "discover content" → Open Opportunities page
  - Generate response text
  - Store chat history (optional)
  - Handle errors gracefully
- **Dependencies:** Task 2.3, existing services
- **Acceptance Criteria:**
  - API receives prompt
  - Routes correctly to workflows
  - Response is contextual
  - Error handling implemented

---

### PHASE 3: Recommended Playbooks (Weeks 3-4)

#### Task 3.1: Design Recommended Playbooks Section
- **Description:** Create visual playbook cards
- **Subtasks:**
  - Design card layout with:
    - Icon/image
    - Title
    - Description
    - "Start" CTA button
    - Difficulty level badge
  - Create 3 default playbooks:
    - Technical SEO Audit
    - Fix Critical Issues
    - Content Opportunity Discovery
  - Responsive grid (1-3 columns)
  - Optional: User-specific recommendations based on audit data
- **Dependencies:** Task 1.1
- **Acceptance Criteria:**
  - Cards display correctly
  - CTAs clickable
  - Opens correct workflow
  - Responsive layout

#### Task 3.2: Link Playbooks to Workflows
- **Description:** Wire playbooks to actual features
- **Subtasks:**
  - Technical Audit → Opens crawl creation
  - Fix Issues → Opens Issues page with critical filter
  - Discover Content → Opens Opportunities page
  - Add tracking (analytics event when clicked)
  - Add hover effects
- **Dependencies:** Task 3.1
- **Acceptance Criteria:**
  - Clicking playbook opens correct page
  - Correct filters applied
  - Analytics tracked

---

### PHASE 4: Active Projects Section (Weeks 4-6)

#### Task 4.1: Design Active Projects Grid/Table
- **Description:** Display user's websites/projects
- **Subtasks:**
  - Create table/card view toggle
  - Display columns:
    - Project name/domain (with favicon)
    - Status badge (active/paused/onboarding)
    - Last audit date
    - Overall health score (visual indicator)
    - Actions dropdown (scan, settings, delete)
  - Responsive design (cards on mobile, table on desktop)
  - Loading skeleton
  - Empty state ("No projects" message)
- **Dependencies:** Task 1.1, existing WebsiteModel
- **Acceptance Criteria:**
  - Projects load from database
  - All columns display correctly
  - View toggle works
  - Empty state shows when appropriate
  - Responsive on all devices

#### Task 4.2: Implement Search & Filter for Projects
- **Description:** Allow filtering and sorting of projects
- **Subtasks:**
  - Add search input (by domain name)
  - Filter by status:
    - All Projects
    - Active
    - Paused
    - Onboarding
  - Filter by health score:
    - Excellent (90-100)
    - Good (70-89)
    - Fair (50-69)
    - Poor (<50)
  - Sort options:
    - Recently audited (default)
    - Alphabetical
    - Highest score
    - Lowest score
  - Pagination (10/25/50 per page)
  - State persistence in URL (optional)
- **Dependencies:** Task 4.1
- **Acceptance Criteria:**
  - Search filters domains
  - Status filter works
  - Score filter works
  - Sort changes order
  - Pagination works
  - Responsive filter UI

#### Task 4.3: Build Projects API Endpoint
- **Description:** Backend for project listing and filtering
- **Subtasks:**
  - Create/extend `Api/V1/WebsitesController.php`
  - Add route: `GET /api/v1/websites` (already exists, may need enhancement)
  - Return projects with:
    - id, domain, status, score, last_audit_date
    - favicon URL
    - connection status
  - Support filters:
    - search_query (domain search)
    - status (active/paused/onboarding)
    - score_min, score_max
  - Support sorting
  - Support pagination
  - User permission filtering
- **Dependencies:** Task 4.2
- **Acceptance Criteria:**
  - API returns correct data
  - Filters work correctly
  - Sorting works
  - Pagination works
  - Response < 200ms

#### Task 4.4: Add Project Actions (Scan, Settings, Delete)
- **Description:** Context menu for project management
- **Subtasks:**
  - Create action dropdown menu
  - "Scan/Audit Now" → Opens audit configuration or queues scan
  - "View Reports" → Navigate to reports page for this project
  - "Settings" → Opens project settings modal
  - "Pause" → Disables automated audits
  - "Delete" → Confirmation dialog and deletion
  - Disable actions based on user permissions
  - Toast notifications for successful actions
- **Dependencies:** Task 4.1
- **Acceptance Criteria:**
  - Dropdown menu appears
  - All actions work as expected
  - Confirmation for destructive actions
  - Success notifications shown
  - Permissions respected

#### Task 4.5: Implement "Create Project" CTA
- **Description:** Add button to start project creation
- **Subtasks:**
  - Add "Create Project" button in header
  - Button styling (primary color, icon)
  - Placeholder text or icon when no projects
  - Click handler to open create modal
  - Visual state (hover, active, disabled)
- **Dependencies:** Task 4.1
- **Acceptance Criteria:**
  - Button visible and clickable
  - Opens create modal on click
  - Styling consistent with design system

---

### PHASE 5: Project Creation Modal (Weeks 6-8)

#### Task 5.1: Design Create Project Modal - Step 1 (Domain Selection)
- **Description:** First step of project onboarding
- **Subtasks:**
  - Modal title: "Create New Project"
  - Step indicator: "1/3"
  - Input field for domain URL
  - Domain validation feedback:
    - Visual checkmark/error on input
    - "Domain is available" / "Domain already in use" message
  - Loading state while validating
  - "Next" button (enabled only if domain valid)
  - "Cancel" button
  - Styling with form validations
  - Error messages for:
    - Invalid domain format
    - Unreachable domain
    - Domain already exists
- **Dependencies:** Task 4.5
- **Acceptance Criteria:**
  - Modal displays correctly
  - Domain validation works
  - Error messages helpful
  - Next button enabled when valid
  - Cancel closes modal

#### Task 5.2: Create Domain Validation API
- **Description:** Backend domain validation
- **Subtasks:**
  - Create endpoint: `POST /api/v1/websites/validate`
  - Input: domain URL
  - Validate domain format
  - Check if domain already owned by user
  - Attempt HTTP HEAD request to domain
  - Return status + metadata:
    - Valid (true/false)
    - Message (string)
    - Available (true/false)
    - Metadata (title, description, etc. if available)
  - Handle errors gracefully
  - Rate limit validation requests
- **Dependencies:** Task 5.1
- **Acceptance Criteria:**
  - API validates domain format
  - Checks domain reachability
  - Prevents duplicates
  - Returns helpful error messages
  - Performance optimized

#### Task 5.3: Design Create Project Modal - Step 2 (Configuration)
- **Description:** Configure crawl and integrations
- **Subtasks:**
  - Step indicator: "2/3"
  - Display validated domain
  - Crawl configuration:
    - "Max pages to crawl" selector (based on plan limit)
    - "Crawl frequency" option:
      - Daily
      - Weekly
      - Monthly
    - Preview of estimated crawl cost/quota
  - Data source connections:
    - Google Search Console (Connect / Already Connected)
    - Google Analytics 4 (Connect / Already Connected)
    - Google Business Profile (Connect / Already Connected)
  - Business information form:
    - Business name
    - Business description
    - Primary service category
    - Target locations (multi-select)
  - Prefilled fields (if available from domain)
  - "Back" and "Next" buttons
  - Validation (required fields)
- **Dependencies:** Task 5.1
- **Acceptance Criteria:**
  - Form displays all fields
  - Validation prevents invalid submission
  - Back button works
  - Next button advances to Step 3
  - Data persists across steps

#### Task 5.4: Create Project Configuration API
- **Description:** Backend for project setup
- **Subtasks:**
  - Create endpoint: `POST /api/v1/websites/configure`
  - Input: domain, crawl_limit, crawl_frequency, integrations
  - Create WebsiteModel record with status "configuring"
  - Store configuration in website_settings or similar
  - Return website_id for next step
  - Validate user quota
  - Queue configuration jobs (if needed)
- **Dependencies:** Task 5.3
- **Acceptance Criteria:**
  - API creates website record
  - Configuration saved
  - Returns website_id
  - Quota validated

#### Task 5.5: Design Create Project Modal - Step 3 (Pixel Installation)
- **Description:** Install WebCrawlers pixel tracking code
- **Subtasks:**
  - Step indicator: "3/3"
  - Instructions header: "Install WebCrawlers Pixel"
  - Display:
    - Installation method selector:
      - Manual (paste code in header)
      - WordPress Plugin (if WordPress detected)
      - Google Tag Manager
    - Copy-to-clipboard code snippet
    - Visual guide (screenshots)
  - "Verify Installation" button
  - Loading state during verification
  - Success message when pixel detected
  - "Complete" button (creates project even if pixel not installed)
  - "Project created" success state:
    - Website name
    - Status: "Not Installed" or "Installed"
    - "Go to Dashboard" button
  - Error handling for verification failures
- **Dependencies:** Task 5.4
- **Acceptance Criteria:**
  - Instructions clear and accurate
  - Code snippet copyable
  - Verification works
  - Success state shows
  - Completion creates project and redirects

#### Task 5.6: Create Pixel Installation & Verification API
- **Description:** Backend for pixel tracking setup
- **Subtasks:**
  - Create endpoint: `POST /api/v1/websites/generate-pixel`
  - Input: website_id
  - Generate unique pixel code:
    - Unique identifier per website
    - Embed script URL
    - Data collection parameters
  - Create endpoint: `POST /api/v1/websites/verify-pixel`
  - Input: website_id
  - Query pixel event logs for this website
  - Return: pixel_detected (boolean), last_seen_at (timestamp)
  - Create endpoint: `POST /api/v1/websites/complete-onboarding`
  - Input: website_id
  - Update website status from "configuring" → "active"
  - Queue initial crawl job
  - Queue integration sync jobs
  - Return success with next steps
- **Dependencies:** Task 5.5
- **Acceptance Criteria:**
  - Pixel code generated correctly
  - Verification detects pixel firing
  - Completion status changes
  - Background jobs queued

#### Task 5.7: Wire Modal Steps Together
- **Description:** Connect all modal steps
- **Subtasks:**
  - Create modal state management (step, data)
  - Implement step navigation (back/next)
  - Persist form data across steps
  - Pass data between steps correctly
  - Handle modal close on completion
  - Redirect after completion
  - Error handling and retry logic
  - Loading states for API calls
- **Dependencies:** Tasks 5.1-5.6
- **Acceptance Criteria:**
  - Modal flow works end-to-end
  - Data persists
  - All steps accessible
  - Completion flows correctly

---

### PHASE 6: Integration & Testing (Weeks 8-10)

#### Task 6.1: Implement Project Onboarding Jobs
- **Description:** Queue background jobs after project creation
- **Subtasks:**
  - Initial crawl job:
    - Queue crawl with configured max_pages
    - Update project status to "crawling"
  - Integration sync jobs:
    - Sync Google Search Console data
    - Sync Google Analytics data
    - Sync Google Business Profile
  - Analysis & scoring jobs:
    - Run technical audit analysis
    - Calculate health scores
    - Generate summary
  - Job queuing system (CodeIgniter queue or similar)
  - Job failure handling and retries
  - Notification to user when jobs complete
- **Dependencies:** Task 5.6
- **Acceptance Criteria:**
  - Jobs queue after creation
  - Jobs execute in order
  - Project status updates
  - User notified of completion

#### Task 6.2: Test All Workflows
- **Description:** Comprehensive testing of SUB-4 features
- **Subtasks:**
  - Unit tests for API endpoints
  - Integration tests for modal flow
  - End-to-end tests:
    - Create new project flow
    - Search projects
    - Filter projects
    - Project actions (scan, delete)
    - Playbook workflows
    - AI assistant
  - Browser testing (Chrome, Firefox, Safari, Edge)
  - Mobile responsive testing
  - Performance testing (load times)
  - Security testing (SQL injection, XSS)
- **Dependencies:** All previous tasks
- **Acceptance Criteria:**
  - All tests passing
  - No broken workflows
  - Performance acceptable
  - Security validated

#### Task 6.3: Performance Optimization
- **Description:** Optimize homepage and features
- **Subtasks:**
  - Profile page load times
  - Optimize database queries:
    - Eager load relationships
    - Add missing indexes
    - Cache frequently accessed data
  - Optimize API responses:
    - Pagination defaults
    - Only return needed fields
    - Compress responses
  - Frontend optimization:
    - Code splitting for modal
    - Lazy load playbook images
    - Minify CSS/JS
  - Caching strategy:
    - Cache website list (5 min TTL)
    - Cache project scores (1 hour TTL)
  - Monitor and log slow queries
- **Dependencies:** Task 6.2
- **Acceptance Criteria:**
  - Homepage loads < 2 seconds
  - API responses < 500ms
  - No N+1 queries
  - Performance metrics tracked

#### Task 6.4: Documentation & Handoff
- **Description:** Create documentation for feature
- **Subtasks:**
  - API documentation (endpoints, parameters, responses)
  - User documentation (how to use features)
  - Architecture documentation
  - Code comments for complex logic
  - Database schema updates documented
  - Configuration requirements
  - Deployment instructions
- **Dependencies:** All previous tasks
- **Acceptance Criteria:**
  - Documentation complete
  - Examples provided
  - Clear and accurate

---

## 🗓️ Timeline Summary

| Phase | Weeks | Tasks |
|-------|-------|-------|
| **1: Foundation** | 1-2 | Layout, Navigation, Routes |
| **2: Search & AI** | 2-3 | Global Search, AI Assistant |
| **3: Playbooks** | 3-4 | Recommended Workflows |
| **4: Projects** | 4-6 | Listing, Filtering, Actions |
| **5: Create Modal** | 6-8 | 3-Step Onboarding Flow |
| **6: Integration** | 8-10 | Jobs, Testing, Optimization |

---

## ✅ Acceptance Criteria Checklist

- [ ] Users can access all primary modules from homepage
- [ ] Global search finds projects, keywords, domains, tools
- [ ] AI assistant prompt works and routes to workflows
- [ ] Recommended playbooks display and link to features
- [ ] Projects list shows all user's websites
- [ ] Projects can be searched, filtered, and sorted
- [ ] Project actions (scan, settings, delete) work
- [ ] Create Project button opens modal
- [ ] Modal validates domain correctly
- [ ] Modal collects all required configuration
- [ ] Pixel installation code displayed and copyable
- [ ] Pixel verification detects installations
- [ ] Project creation completes successfully
- [ ] New project appears in project list immediately
- [ ] Onboarding jobs queue and execute
- [ ] Initial crawl begins after project creation
- [ ] No duplicate projects created
- [ ] All workflows tested end-to-end
- [ ] Responsive on mobile/tablet/desktop
- [ ] Performance metrics met
- [ ] Documentation complete

---

## 📊 Estimated Effort

| Component | Effort | Notes |
|-----------|--------|-------|
| Design & Layout | 1 week | Straightforward implementation |
| Search & AI | 1 week | May need AI service integration |
| Playbooks | 0.5 week | Mostly routing |
| Projects Listing | 2 weeks | Filtering/sorting complexity |
| Create Modal | 2.5 weeks | Multi-step, validation, pixel |
| Integration & Testing | 2 weeks | Jobs, testing, optimization |
| **Total** | **9 weeks** | Could be 10-12 with edge cases |

---

## 🚀 Success Metrics

1. **Homepage loads < 2 seconds**
2. **All 8 acceptance criteria met**
3. **Zero critical bugs in QA**
4. **API endpoints respond < 500ms**
5. **Mobile responsive (pass Lighthouse)**
6. **100% test coverage for APIs**
7. **User satisfaction feedback positive**

---

## 📞 Dependencies & Risks

### Dependencies
- Existing models: WebsiteModel, UserModel, AuditModel
- Existing services: integration APIs (GSC, GA4, GBP)
- Pixel tracking system must be functional
- Background job system (CodeIgniter queue)

### Risks
1. **Domain validation complexity** — Third-party API dependencies
2. **Integration syncing** — Multiple external APIs to coordinate
3. **Performance at scale** — Large project lists could be slow
4. **Mobile responsiveness** — Complex modal on small screens

### Mitigations
- Implement proper error handling and retries
- Cache integration data aggressively
- Use pagination for large datasets
- Test mobile thoroughly early