# Advanced Scraper V3.5 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Upgrade V3.4 scraping so transient failures and varied publisher markup do not silently miss news.

**Architecture:** Add a dedicated advanced scraper module and additive retry/statistics schema, then wire the existing collection loop and admin UI into it. Keep RSS, classification, district mapping and public APIs backward compatible.

**Tech Stack:** PHP 8+, PDO/MySQL, DOMDocument/DOMXPath, cURL, existing JSON/config helpers.

**Spec:** `docs/superpowers/specs/2026-09-14-advanced-scraper-v35-design.md`

## Global Constraints
- Backend only.
- Additive database migration only.
- Transient article failures must not be marked seen.
- Existing V3.4 tests remain green.
- No new external PHP dependencies.

---

### Task 1: Advanced scraper core
**Files:** Create `includes/advanced_scraper.php`; create `tests/test_advanced_scraper.php`.
**Produces:** URL normalization, retry classification, multi-strategy article extraction and link discovery helpers.
- [ ] Write failing tests for URL normalization, retry classification, JSON-LD extraction, lazy image extraction and pagination link discovery.
- [ ] Run tests and confirm RED.
- [ ] Implement minimal helpers.
- [ ] Run tests and confirm GREEN.

### Task 2: Retry persistence and source diagnostics
**Files:** Create `database/migration_advanced_scraper_v35.sql`; modify `includes/functions.php`; create `tests/test_advanced_scraper_contracts.php`.
**Produces:** retry queue helpers and persisted source metrics/settings.
- [ ] Write failing schema/contract tests.
- [ ] Confirm RED.
- [ ] Add migration and persistence wiring.
- [ ] Confirm GREEN.

### Task 3: Collection orchestration regression fix
**Files:** Modify `includes/functions.php`; modify/create tests.
**Produces:** run loop that only marks terminal URLs processed and queues transient failures.
- [ ] Write failing regression test for failed-link-not-seen and retry scheduling.
- [ ] Confirm RED.
- [ ] Wire advanced discovery/extraction/retry into `run_link_scrape_collection()`.
- [ ] Confirm GREEN.

### Task 4: Admin controls and Test Scraper
**Files:** Modify `admin/scrape_sources.php`, `admin/settings.php`, `cron/scrape_fetch.php`.
**Produces:** source controls, diagnostics and read-only test action.
- [ ] Write failing admin contract tests.
- [ ] Confirm RED.
- [ ] Add controls and diagnostics UI.
- [ ] Confirm GREEN.

### Task 5: Regression and packaging
**Files:** Update docs; create V3.5 ZIP.
- [ ] Run all PHP tests.
- [ ] Lint all PHP files.
- [ ] Validate migration and ZIP contents.
- [ ] Package as `BD-News-Control-Center-V3.5-Advanced-Scraper.zip`.