# Solution Design Document: POS Data Catchup System

## Validation Checklist

- [x] All required sections are complete
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] All context sources are listed with relevance ratings
- [x] Project commands are discovered from actual project files
- [x] Constraints → Strategy → Design → Implementation path is logical
- [x] Architecture pattern is clearly stated with rationale
- [x] Every component in diagram has directory mapping
- [x] Every interface has specification
- [x] Error handling covers all error types
- [x] Quality requirements are specific and measurable
- [x] Every quality requirement has test coverage
- [x] **All architecture decisions confirmed by user**
- [x] Component names consistent across diagrams
- [x] A developer could implement from this design

---

## Constraints

- **CON-1 Framework**: PHP 8.x, Slim 2.6.2, Twig 1.44.8. Must use existing TaskEngine infrastructure and Ably REST PHP SDK (v1.1)
- **CON-2 Database**: MariaDB/MySQL with migration system. Central DB (`kiosk_buykiosk`) for gap tracking, store DBs for `drsDailySFileData`. Sales/Buys in `kiosk_sales`. NEVER modify tables directly — always use migration system
- **CON-3 Ably Limits**: 50 messages/sec per channel. Must use existing `AblyPublishThrottle`. Desktop app already listens on `{typeNum}` channel
- **CON-4 TaskEngine**: Per-store jobs get `Store` context automatically. 300-second default timeout. Must use `BaseJob` pattern with `progress()`, `checkpoint()`, `info()` helpers
- **CON-5 No Deployment**: Local dev served via ngrok to `dev2.buyerkiosk.com`. Code changes are live immediately after saving
- **CON-6 Existing APIs**: Desktop app posts to existing endpoints (`/api/{typeNum}/sales/salesDay/{date}`, `/api/{typeNum}/buys/buysDay`, `/api/{typeNum}/daily-close`). These are upsert-safe and must not change

## Implementation Context

### Required Context Sources

- ICO-1 General Application Context
```yaml
- doc: CLAUDE.md
  relevance: HIGH
  why: "Core architecture patterns, route conventions, migration system, PSR-4 namespacing"

- doc: docs/specs/043-pos-data-catchup/product-requirements.md
  relevance: CRITICAL
  why: "PRD defining all requirements, business rules, edge cases, and acceptance criteria"
```

- ICO-2 TaskEngine System
```yaml
- file: userfrosting/src/BuyerKiosk/TaskEngine/Domain/Job/BaseJob.php
  relevance: CRITICAL
  why: "Base class our job extends — provides progress(), checkpoint(), info(), getStore(), getStoreDb()"

- file: userfrosting/src/BuyerKiosk/TaskEngine/Domain/Job/JobInterface.php
  relevance: HIGH
  why: "Interface contract: getName(), getScope(), getQueue(), getTimeout(), handle()"

- file: userfrosting/src/BuyerKiosk/TaskEngine/Jobs/BackfillMockDataJob.php
  relevance: HIGH
  why: "Reference implementation for backfill pattern with lookback, chunking, progress reporting"

- file: userfrosting/src/BuyerKiosk/TaskEngine/Commands/TaskCommandFactory.php
  relevance: HIGH
  sections: [registerJobs method, lines 234-301]
  why: "Where new jobs must be registered"

- file: userfrosting/src/BuyerKiosk/TaskEngine/Application/JobDispatcher.php
  relevance: MEDIUM
  why: "Dispatch and idempotency patterns"
```

- ICO-3 Ably Integration
```yaml
- file: userfrosting/src/BuyerKiosk/Core/AblyPublishThrottle.php
  relevance: HIGH
  why: "Must use this throttle for all Ably publishes"

- file: userfrosting/src/BuyerKiosk/Workbook/WorkbookAbly.php
  relevance: HIGH
  sections: [constructor pattern, publish method]
  why: "Reference for Ably publisher class pattern with throttle integration"

- file: userfrosting/models/BaseModel.php
  relevance: MEDIUM
  sections: [getAblyClient(), getRedisClient(), validateAPIKey()]
  why: "Ably client creation, Redis access, API key validation"

- file: userfrosting/routes/groups/drs.php
  relevance: MEDIUM
  sections: [sendEncodedData function, lines 1866-1886]
  why: "Existing DRS Ably publish pattern"
```

- ICO-4 POS Data APIs
```yaml
- file: userfrosting/src/BuyerKiosk/Sales/Controllers/SalesController.php
  relevance: MEDIUM
  why: "getOldestRecordDate() pattern for querying sales date ranges"

- file: userfrosting/src/BuyerKiosk/QuickBooks/Controllers/DailyCloseApiController.php
  relevance: MEDIUM
  sections: [getOldestRecordDate, lines 238-280]
  why: "Oldest date API pattern for daily close data"

- file: userfrosting/src/BuyerKiosk/Core/Store.php
  relevance: HIGH
  sections: [drsApiKey getter/setter, timezone, dbName, active flag]
  why: "Store entity with DRS configuration"
```

### Implementation Boundaries

- **Must Preserve**: All existing DRS API endpoints, Ably event format conventions, TaskEngine job registration pattern, migration system conventions
- **Can Modify**: Store.php to add new getters for earliest available date column, TaskCommandFactory to register new job
- **Must Not Touch**: Existing DRS route handlers, existing Ably publisher classes, existing sales/buys/daily-close controllers

### External Interfaces

#### System Context Diagram

```mermaid
graph TB
    Scheduler[TaskEngine Scheduler] --> CatchupJob[POS Data Catchup Job]
    CLI[CLI Manual Dispatch] --> CatchupJob

    CatchupJob --> CentralDB[(kiosk_buykiosk<br/>Gap Tracking + Store Config)]
    CatchupJob --> SalesDB[(kiosk_sales<br/>Sales & Buys Data)]
    CatchupJob --> StoreDB[(kiosk_{typeNum}<br/>buyQueue + drsDailySFileData)]
    CatchupJob --> Ably[Ably Channel<br/>{typeNum}]

    Ably --> DesktopApp[Desktop DRS App]

    DesktopApp --> ExistingAPIs[Existing POS Data APIs<br/>/api/{typeNum}/sales/...<br/>/api/{typeNum}/buys/...<br/>/api/{typeNum}/daily-close]
    DesktopApp --> BoundaryAPI[Earliest Date API<br/>POST /api/{typeNum}/drs/catchup/boundary]

    ExistingAPIs --> SalesDB
    ExistingAPIs --> StoreDB
    BoundaryAPI --> CentralDB
```

#### Interface Specifications

```yaml
# Inbound Interfaces
inbound:
  - name: "TaskEngine Scheduler"
    type: Internal (cron → PHP CLI)
    format: Job dispatch via Redis queue
    authentication: None (internal process)
    data_flow: "Triggers per-store gap detection on schedule"

  - name: "CLI Manual Dispatch"
    type: CLI
    format: "php userfrosting/bin/task job:dispatch pos-data-catchup --store=pc00 --payload='{...}'"
    authentication: Server access
    data_flow: "Manual trigger with optional date range override"

  - name: "Desktop App Earliest Date Report"
    type: HTTPS POST
    format: JSON
    authentication: DRS API Key (60-char, validated via validateAPIKey())
    data_flow: "Desktop app reports the earliest date it has POS data for"

# Outbound Interfaces
outbound:
  - name: "Ably Catchup Commands"
    type: Ably REST publish
    format: JSON payload
    authentication: ABLY_KEY env var
    data_flow: "Commands sent to desktop app via store channel"
    criticality: HIGH

# Data Interfaces
data:
  - name: "Central Database (kiosk_buykiosk)"
    type: MySQL/MariaDB
    connection: PDO via dbConnectByName()
    data_flow: "Gap tracking table, store config, job definitions"

  - name: "Sales Database (kiosk_sales)"
    type: MySQL/MariaDB
    connection: PDO via dbConnectByName('kiosk_sales')
    data_flow: "Query sales and buys date existence"

  - name: "Store Databases (kiosk_{typeNum})"
    type: MySQL/MariaDB
    connection: PDO via $this->getStoreDb()
    data_flow: "Query buyQueue activity and drsDailySFileData"

  - name: "Redis"
    type: Redis via Predis
    connection: getRedisClient()
    data_flow: "Ably publish throttle counters"
```

### Cross-Component Boundaries

- **API Contracts**: The Ably catchup command payload (defined below) is a NEW contract between server and desktop app. Desktop team must implement the listener
- **Team Ownership**: Server-side (this spec) owned by BuyerKiosk web team. Desktop app listener owned by DRS team
- **Shared Resources**: `{typeNum}` Ably channel is shared between DRS sync events, workbook events, and catchup commands. New action prefix `drs:catchup:*` prevents collision
- **Breaking Change Policy**: Ably command format is versioned. Changes require desktop team coordination

### Project Commands

```bash
# Testing
./test.sh --testsuite unit                    # Run all unit tests
cd userfrosting && ./vendor/bin/phpunit --filter "PosDataCatchup"  # Targeted tests

# Database Migrations
php userfrosting/conductor run                 # Run all pending migrations

# TaskEngine
php userfrosting/bin/task job:list             # Verify job registered
php userfrosting/bin/task job:dispatch pos-data-catchup --store=pc00  # Manual dispatch
php userfrosting/bin/task job:dispatch pos-data-catchup --store=pc00 --payload='{"startDate":"2025-04-01","endDate":"2025-04-30"}'

# Worker
php userfrosting/bin/task worker:start --queues=default  # Start worker to pick up job

# PHPStan
cd userfrosting && ./vendor/bin/phpstan analyse src/BuyerKiosk/PosDataCatchup/
```

## Solution Strategy

- **Architecture Pattern**: Per-store TaskEngine job following the established `BaseJob` pattern. The job is self-contained — it detects gaps, publishes commands, and tracks state, all within a single job execution per store
- **Integration Approach**: Extends the existing Ably `{typeNum}` channel with new `drs:catchup:*` action types. Uses a dedicated publisher class following the `WorkbookAbly` pattern. New gap tracking table in central DB
- **Justification**: The TaskEngine already handles per-store scheduling, retries, progress reporting, and abort signals. Building on this avoids reinventing infrastructure. The Ably channel is already established between server and desktop
- **Key Decisions**: Per-store scope (not global), dedicated tracking table (not TaskEngine execution data), publisher class (not inline publishes), chunked backfill (not single massive request)

## Building Block View

### Components

```mermaid
graph LR
    subgraph TaskEngine
        Job[PosDataCatchupJob<br/>extends BaseJob]
        Job --> Detector[GapDetector<br/>Service]
        Job --> Publisher[CatchupAblyPublisher]
        Job --> Tracker[GapTracker<br/>Repository]
    end

    subgraph DataSources
        Detector --> SalesDB[(kiosk_sales)]
        Detector --> StoreDB[(kiosk_{typeNum})]
        Detector --> HoursDB[(storeOperatingHours)]
    end

    subgraph Tracking
        Tracker --> GapTable[(posDataGaps<br/>in kiosk_buykiosk)]
    end

    subgraph Messaging
        Publisher --> Ably[Ably Channel<br/>{typeNum}]
        Publisher --> Throttle[AblyPublishThrottle]
    end

    subgraph ExternalAPI
        BoundaryCtrl[CatchupBoundaryController] --> GapTable
    end
```

### Directory Map

```
userfrosting/src/BuyerKiosk/PosDataCatchup/           # NEW: Feature namespace
├── Jobs/
│   └── PosDataCatchupJob.php                          # NEW: Main TaskEngine job
├── Services/
│   ├── GapDetector.php                                # NEW: Gap detection logic
│   └── CatchupAblyPublisher.php                       # NEW: Ably command publisher
├── Persistence/
│   └── GapTracker.php                                 # NEW: Gap CRUD operations
└── Controllers/
    └── CatchupBoundaryController.php                  # NEW: API for desktop boundary report

userfrosting/src/BuyerKiosk/TaskEngine/Commands/
└── TaskCommandFactory.php                             # MODIFY: Register new job (line ~300)

userfrosting/src/BuyerKiosk/Core/
└── Store.php                                          # MODIFY: Add drsEarliestDataDate getter/setter

userfrosting/routes/groups/
└── catchup.php                                        # NEW: Route definitions for catchup API

userfrosting/routes/
└── api.php                                            # MODIFY: Include catchup route group

userfrosting/migrations/input/
├── 20260401_043_001_pos_data_gaps_table.json           # NEW: Gap tracking table
├── 20260401_043_002_stores_earliest_data_date.json     # NEW: Column on stores
└── 20260401_043_003_pos_data_catchup_job_def.json      # NEW: Job definition

tests/Unit/PosDataCatchup/
├── GapDetectorTest.php                                # NEW: Unit tests for gap detection
├── CatchupAblyPublisherTest.php                       # NEW: Unit tests for command formatting
├── GapTrackerTest.php                                 # NEW: Unit tests for gap CRUD
└── PosDataCatchupJobTest.php                          # NEW: Integration test for job
```

### Interface Specifications

#### Data Storage Changes

**Table: `posDataGaps` (NEW — in `kiosk_buykiosk` central DB)**

```sql
CREATE TABLE `posDataGaps` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `typeNum` varchar(10) NOT NULL COMMENT 'Store identifier',
  `gapDate` date NOT NULL COMMENT 'The date with missing data',
  `missingSales` tinyint(1) NOT NULL DEFAULT 1 COMMENT 'Sales data missing',
  `missingBuys` tinyint(1) NOT NULL DEFAULT 1 COMMENT 'Buys data missing',
  `missingDailyClose` tinyint(1) NOT NULL DEFAULT 1 COMMENT 'Daily close S-File missing',
  `verificationSource` enum('buyQueue','storeHours','manual') NOT NULL DEFAULT 'storeHours' COMMENT 'How we determined the store was open',
  `status` enum('detected','commanded','resolved','persistent','unavailable') NOT NULL DEFAULT 'detected',
  `commandCount` int(10) unsigned NOT NULL DEFAULT 0 COMMENT 'Number of catchup commands sent',
  `lastCommandSentAt` datetime DEFAULT NULL,
  `lastRequestId` varchar(64) DEFAULT NULL COMMENT 'UUID of most recent catchup command',
  `resolvedAt` datetime DEFAULT NULL,
  `firstDetectedAt` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updatedAt` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_store_date` (`typeNum`, `gapDate`),
  KEY `idx_status` (`status`),
  KEY `idx_typenum_status` (`typeNum`, `status`),
  KEY `idx_persistent` (`status`, `firstDetectedAt`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

**Table: `stores` (MODIFY — add column)**

```sql
ALTER TABLE `stores` ADD COLUMN `drsEarliestDataDate` date DEFAULT NULL
  COMMENT 'Earliest date the desktop DRS app has POS data for (reported by desktop)'
  AFTER `drsApiKey`;
```

**Table: `task_job_definitions` (INSERT — new job definition)**

```sql
INSERT INTO `task_job_definitions`
  (`name`, `displayName`, `className`, `schedule`, `queue`, `scope`, `timeout`, `maxRetries`, `retryBackoff`, `config`, `isEnabled`, `notifyOnFailure`, `notifyEmails`)
VALUES
  ('pos-data-catchup', 'POS Data Catchup',
   'BuyerKiosk\\PosDataCatchup\\Jobs\\PosDataCatchupJob',
   '0 3 * * *', 'default', 'per_store', 600, 2, 300,
   '{"lookbackDays": 365, "batchChunkDays": 30, "bufferDays": 1, "persistentThresholdDays": 7}',
   1, 1, NULL);
```

#### Internal API Changes — Earliest Date Boundary Endpoint

```yaml
Endpoint: Report Earliest Available Data Date
  Method: POST
  Path: /api/{typeNum}/drs/catchup/boundary
  Authentication: DRS API Key (X-API-Key header or "api" POST param)
  Request:
    earliestDate: string (YYYY-MM-DD) REQUIRED — oldest date the POS system has data for
  Response:
    success:
      success: true
      message: "Earliest data date recorded"
      typeNum: string
      earliestDate: string (YYYY-MM-DD)
      previousDate: string|null (YYYY-MM-DD) — previous value if updating
    error:
      success: false
      error: string — error description
  Notes:
    - Updates stores.drsEarliestDataDate
    - Gaps before this date are auto-resolved as "unavailable"
    - Idempotent — can be called multiple times
```

#### Ably Catchup Command Contract

This is the NEW contract between server and desktop app. Published to `{typeNum}` channel.

**Action: `drs:catchup:request`**

```json
{
  "action": "drs:catchup:request",
  "category": "pc00",
  "timestamp": 1743494400,
  "source": "catchup",
  "requestId": "uuid-v4-string",
  "version": 1,
  "dataTypes": ["sales", "buys", "dailyClose"],
  "dateRange": {
    "startDate": "2025-10-01",
    "endDate": "2025-10-31"
  },
  "reportBoundary": true,
  "endpoints": {
    "sales": "/api/{typeNum}/sales/salesDay/{date}",
    "buys": "/api/{typeNum}/buys/buysDay",
    "dailyClose": "/api/{typeNum}/daily-close",
    "boundary": "/api/{typeNum}/drs/catchup/boundary"
  }
}
```

**Field descriptions:**

| Field | Type | Description |
|-------|------|-------------|
| `action` | string | Always `drs:catchup:request` |
| `category` | string | Store typeNum (Ably channel name) |
| `timestamp` | int | Unix timestamp of command |
| `source` | string | Always `catchup` (for throttle tracking) |
| `requestId` | string | UUID v4 for tracking this specific request |
| `version` | int | Contract version (start at 1) |
| `dataTypes` | string[] | Which data types are needed: `sales`, `buys`, `dailyClose` |
| `dateRange.startDate` | string | Start of range (YYYY-MM-DD, inclusive) |
| `dateRange.endDate` | string | End of range (YYYY-MM-DD, inclusive) |
| `reportBoundary` | bool | If true, desktop should also POST its earliest available date to the boundary endpoint |
| `endpoints` | object | Reminder of which endpoints to POST data to (convenience for desktop devs) |

**Desktop app expected behavior:**
1. Receive `drs:catchup:request` on the store's Ably channel
2. For each date in `dateRange`, fetch data from local POS system
3. POST data to the listed endpoints (same format as existing sync)
4. If `reportBoundary` is true AND earliest date hasn't been reported yet, POST to the boundary endpoint
5. If data is unavailable for a date (store wasn't open, POS system doesn't go back that far), skip it silently

#### Application Data Models

```pseudocode
ENTITY: PosDataCatchupJob (NEW) extends BaseJob
  STATIC:
    getName(): 'pos-data-catchup'
    getDisplayName(): 'POS Data Catchup'
    getQueue(): 'default'
    getScope(): 'per_store'
    getTimeout(): 600

  BEHAVIORS:
    handle(): JobResult
      — Orchestrates gap detection, command publishing, and tracking

ENTITY: GapDetector (NEW)
  CONSTRUCTOR: (PDO $centralDb, PDO $salesDb, PDO $storeDb, string $typeNum, string $timezone)

  BEHAVIORS:
    detectGaps(DateTimeInterface $from, DateTimeInterface $to): GapResult[]
      — Returns dates with missing data within the window
    getOperationalDays(DateTimeInterface $from, DateTimeInterface $to): DateTimeInterface[]
      — Combines buyQueue activity + storeOperatingHours
    hasSalesData(string $typeNum, string $date): bool
    hasBuysData(string $typeNum, string $date): bool
    hasDailyCloseData(string $date): bool
    getBuyQueueDates(DateTimeInterface $from, DateTimeInterface $to): string[]
    getScheduledOpenDays(string $typeNum, DateTimeInterface $from, DateTimeInterface $to): string[]

ENTITY: CatchupAblyPublisher (NEW)
  CONSTRUCTOR: (string $typeNum)

  BEHAVIORS:
    publishCatchupRequest(array $dataTypes, string $startDate, string $endDate, string $requestId, bool $reportBoundary): bool
      — Publishes drs:catchup:request to Ably channel
      — Returns false if throttled

ENTITY: GapTracker (NEW)
  CONSTRUCTOR: (PDO $centralDb)

  BEHAVIORS:
    upsertGap(string $typeNum, string $date, array $missingTypes, string $verificationSource): int
      — INSERT ON DUPLICATE KEY UPDATE. Returns gap ID
    markCommanded(string $typeNum, string $date, string $requestId): void
    resolveGap(string $typeNum, string $date): void
    markUnavailable(string $typeNum, string $datesBefore): int
      — Bulk-resolve gaps before earliest available date. Returns count
    getUnresolvedGaps(string $typeNum): array
    getPersistentGaps(string $typeNum, int $thresholdDays): array
    getGapStats(string $typeNum): array
      — Returns summary: total, detected, commanded, resolved, persistent
    cleanupOldResolved(int $retentionDays): int
      — Archive/delete resolved gaps older than retention period

ENTITY: CatchupBoundaryController (NEW)
  CONSTRUCTOR: (Store $store, \Slim\Slim $app)

  BEHAVIORS:
    reportEarliestDate(): void
      — POST handler: validates date, updates stores.drsEarliestDataDate
      — Auto-resolves gaps before the new earliest date
```

### Implementation Examples

#### Example: Gap Detection Algorithm

**Why this example**: The core algorithm combining buyQueue verification with store hours is the most complex piece and needs clarity.

```php
// Example: GapDetector::detectGaps() core logic
// This demonstrates the expected approach, not exact implementation

public function detectGaps(\DateTimeInterface $from, \DateTimeInterface $to): array
{
    // Step 1: Get dates where buyQueue had activity (independent verifier)
    $buyQueueDates = $this->getBuyQueueDates($from, $to);

    // Step 2: Get dates store was scheduled to be open
    $scheduledOpenDates = $this->getScheduledOpenDays($this->typeNum, $from, $to);

    // Step 3: Union = all dates we expect POS data for
    // buyQueue takes precedence (Rule 2: store was actually open)
    $expectedDates = array_unique(array_merge($buyQueueDates, $scheduledOpenDates));
    sort($expectedDates);

    // Step 4: For each expected date, check what's missing
    $gaps = [];
    foreach ($expectedDates as $date) {
        $missing = [];
        if (!$this->hasSalesData($this->typeNum, $date)) {
            $missing[] = 'sales';
        }
        if (!$this->hasBuysData($this->typeNum, $date)) {
            $missing[] = 'buys';
        }
        if (!$this->hasDailyCloseData($date)) {
            $missing[] = 'dailyClose';
        }

        if (!empty($missing)) {
            $source = in_array($date, $buyQueueDates) ? 'buyQueue' : 'storeHours';
            $gaps[] = new GapResult($date, $missing, $source);
        }
    }

    return $gaps;
}
```

#### Example: Chunked Backfill Command Publishing

**Why this example**: Shows how large gaps are split into batches to avoid overwhelming the desktop app.

```php
// Example: PosDataCatchupJob chunk logic
// Large gaps are split into configurable batch sizes

$batchChunkDays = (int) $this->config('batchChunkDays', 30);

// Group unresolved gaps by contiguous date ranges
$dateRanges = $this->groupIntoRanges($unresolvedGapDates);

foreach ($dateRanges as $range) {
    // Split ranges longer than batchChunkDays
    $chunks = $this->chunkDateRange($range['start'], $range['end'], $batchChunkDays);

    foreach ($chunks as $chunk) {
        $requestId = \Ramsey\Uuid\Uuid::uuid4()->toString();

        $published = $this->publisher->publishCatchupRequest(
            dataTypes: $chunk['dataTypes'],
            startDate: $chunk['start'],
            endDate: $chunk['end'],
            requestId: $requestId,
            reportBoundary: !$this->hasEarliestDate()
        );

        if ($published) {
            // Mark all gaps in this chunk as "commanded"
            foreach ($chunk['dates'] as $date) {
                $this->tracker->markCommanded($this->getTypeNum(), $date, $requestId);
            }
            $commandsSent++;
        }

        $this->checkpoint(); // Heartbeat + abort check
    }
}
```

#### Example: Ably Publisher Pattern

**Why this example**: Shows how to follow the established WorkbookAbly pattern with throttle integration.

```php
// Example: CatchupAblyPublisher following WorkbookAbly pattern

class CatchupAblyPublisher
{
    private ?\Ably\AblyRest $ably = null;
    private string $typeNum;
    private bool $enabled;
    private ?AblyPublishThrottle $throttle;

    public function __construct(string $typeNum)
    {
        $this->typeNum = $typeNum;
        $this->enabled = false;

        if (empty($_ENV['ABLY_KEY'])) {
            return;
        }

        try {
            $this->ably = \getAblyClient();
            $this->enabled = true;
        } catch (\Exception $e) {
            error_log("CatchupAblyPublisher: Failed to init - " . $e->getMessage());
        }

        try {
            $redis = \getRedisClient();
        } catch (\Throwable $e) {
            $redis = null;
        }
        $this->throttle = new AblyPublishThrottle($redis);
    }

    public function publishCatchupRequest(
        array $dataTypes, string $startDate, string $endDate,
        string $requestId, bool $reportBoundary
    ): bool {
        if (!$this->enabled) return false;

        $action = 'drs:catchup:request';

        if ($this->throttle && !$this->throttle->shouldPublish($this->typeNum, $action, 'catchup')) {
            return false; // Throttled — caller will retry next cycle
        }

        $payload = [
            'action' => $action,
            'category' => $this->typeNum,
            'timestamp' => time(),
            'source' => 'catchup',
            'requestId' => $requestId,
            'version' => 1,
            'dataTypes' => $dataTypes,
            'dateRange' => [
                'startDate' => $startDate,
                'endDate' => $endDate,
            ],
            'reportBoundary' => $reportBoundary,
            'endpoints' => [
                'sales' => "/api/{$this->typeNum}/sales/salesDay/{date}",
                'buys' => "/api/{$this->typeNum}/buys/buysDay",
                'dailyClose' => "/api/{$this->typeNum}/daily-close",
                'boundary' => "/api/{$this->typeNum}/drs/catchup/boundary",
            ],
        ];

        try {
            $channel = $this->ably->channel($this->typeNum);
            $channel->publish($action, $payload);
            return true;
        } catch (\Exception $e) {
            error_log("CatchupAblyPublisher: publish failed for {$this->typeNum} - " . $e->getMessage());
            return false;
        }
    }
}
```

## Runtime View

### Primary Flow: Scheduled Gap Detection and Command

```mermaid
sequenceDiagram
    participant Scheduler as TaskEngine Scheduler
    participant Job as PosDataCatchupJob
    participant Detector as GapDetector
    participant StoreDB as Store DB
    participant SalesDB as kiosk_sales
    participant Tracker as GapTracker
    participant Publisher as CatchupAblyPublisher
    participant Ably as Ably Channel
    participant Desktop as Desktop DRS App

    Scheduler->>Job: Dispatch per store (typeNum=pc00)
    Job->>Job: Load config (lookbackDays=365, chunkDays=30)
    Job->>Job: Check stores.drsEarliestDataDate

    Job->>Detector: detectGaps(fromDate, toDate)
    Detector->>StoreDB: SELECT DISTINCT DATE(timeEntered) FROM buyQueue WHERE...
    Detector->>StoreDB: SELECT closingDate FROM drsDailySFileData WHERE...
    Detector->>SalesDB: SELECT DISTINCT salesDate FROM sales WHERE typeNum=...
    Detector->>SalesDB: SELECT DISTINCT buyDate FROM buys WHERE typeNum=...
    Detector-->>Job: GapResult[] (dates + missing types)

    Job->>Tracker: upsertGap() for each detected gap
    Job->>Job: Group gaps into date ranges, chunk into 30-day batches

    loop For each chunk
        Job->>Publisher: publishCatchupRequest(chunk)
        Publisher->>Ably: publish('drs:catchup:request', payload)
        Ably->>Desktop: Catchup command received
        Job->>Tracker: markCommanded(dates, requestId)
    end

    Job->>Tracker: Check previously commanded gaps for resolution
    Tracker->>Detector: Re-check if data now exists
    Detector-->>Tracker: Some gaps resolved
    Job->>Tracker: resolveGap() for resolved dates

    Job-->>Scheduler: JobResult::success({gapsFound, commandsSent, resolved})
```

### Secondary Flow: Desktop Reports Earliest Date

```mermaid
sequenceDiagram
    participant Desktop as Desktop DRS App
    participant API as CatchupBoundaryController
    participant Store as stores table
    participant Tracker as GapTracker

    Desktop->>API: POST /api/pc00/drs/catchup/boundary {earliestDate: "2025-04-01"}
    API->>API: Validate API key + date format
    API->>Store: UPDATE stores SET drsEarliestDataDate = '2025-04-01'
    API->>Tracker: markUnavailable('pc00', '2025-04-01')
    Note over Tracker: All gaps before 2025-04-01 → status='unavailable'
    API-->>Desktop: {success: true, earliestDate: "2025-04-01"}
```

### Error Handling

- **Ably unavailable**: Publisher returns `false`. Job logs warning, continues to next chunk. Commands deferred to next cycle
- **Database connection failure**: Job throws exception, TaskEngine handles retry with exponential backoff (configured: 2 retries, 300s backoff)
- **Store DB missing**: Job skips store, logs error, returns partial success
- **Invalid API key on boundary endpoint**: 401 response, no state change
- **Invalid date format on boundary endpoint**: 400 response with descriptive error
- **Gap tracking table write failure**: Job logs error, continues (gap detection still works, just not tracked)
- **Throttled by Ably**: Chunk not sent, logged. Next scan cycle will retry
- **Job timeout approaching**: Use `checkpoint()` to detect and return partial results before timeout

### Complex Logic: Gap Detection Window Calculation

```
ALGORITHM: Calculate Gap Detection Window
INPUT: store, lookbackDays, bufferDays, earliestDataDate
OUTPUT: fromDate, toDate

1. toDate = today - bufferDays (default: yesterday)
2. fromDate = today - lookbackDays (default: 365 days ago)
3. IF store.drsEarliestDataDate IS NOT NULL:
     fromDate = MAX(fromDate, store.drsEarliestDataDate)
4. IF store has ANY existing POS data:
     firstDataDate = MIN(oldest sales date, oldest buys date, oldest close date)
     // Don't go before the first data ever received unless this is a new store
5. IF store has ZERO POS data records (new store):
     // Full backfill scenario — use the full window
     // Set reportBoundary = true on catchup command
6. VALIDATE: fromDate < toDate (else no window to check)
7. RETURN fromDate, toDate
```

## Deployment View

### Single Application Deployment

- **Environment**: PHP 8.x on local dev machine, served via ngrok to `dev2.buyerkiosk.com`
- **Configuration**: No new env vars needed. Uses existing `ABLY_KEY`. Job config stored in `task_job_definitions.config` JSON column
- **Dependencies**: Existing — Ably PHP SDK, Redis (for throttle), MySQL (for gap tracking)
- **Performance**: Job runs daily at 3 AM per store. Each store execution should complete in <60 seconds. Date existence queries use indexed columns

### Deployment Steps

1. Run migrations: `php userfrosting/conductor run` (creates `posDataGaps` table, adds `drsEarliestDataDate` column, inserts job definition)
2. Code changes are live immediately (local dev via ngrok)
3. Verify job registered: `php userfrosting/bin/task job:list | grep catchup`
4. Test manually: `php userfrosting/bin/task job:dispatch pos-data-catchup --store=pc00`

## Cross-Cutting Concepts

### Pattern Documentation

```yaml
- pattern: BaseJob + per-store execution pattern
  relevance: CRITICAL
  why: "Must follow established TaskEngine job patterns"

- pattern: WorkbookAbly publisher pattern
  relevance: HIGH
  why: "Must follow established Ably publishing with throttle"

- pattern: Migration JSON format
  relevance: HIGH
  why: "All schema changes via migration system"

- pattern: DRS API key validation
  relevance: HIGH
  why: "Boundary endpoint uses existing validateAPIKey()"
```

### System-Wide Patterns

- **Security**: DRS API key validation for boundary endpoint. No new auth mechanisms
- **Error Handling**: Fail-safe — if any component fails, the job continues and retries next cycle. No data loss possible because existing APIs are upsert-safe
- **Performance**: Use `EXISTS` subqueries for date checks (faster than `COUNT(*)` for large tables). Batch gap detection into single queries per data type
- **Logging**: Use TaskEngine's built-in `$this->info()`, `$this->warning()`, `$this->error()` for execution logs visible in dashboard

### Implementation Patterns

#### Code Patterns and Conventions

- PSR-4 autoloading: `BuyerKiosk\PosDataCatchup\*` namespace
- camelCase for column names and table names (per project convention)
- Constructor injection for dependencies (PDO connections, store reference)
- Static factory methods for job interface (`getName()`, `getScope()`, etc.)

#### State Management Patterns

- Gap state is persistent in `posDataGaps` table (survives between job runs)
- Job is stateless between runs — all state comes from database
- No Redis state beyond Ably throttle counters (auto-expiring)

#### Performance Characteristics

- **Date existence checks**: Use `SELECT 1 FROM sales WHERE typeNum = :t AND salesDate = :d LIMIT 1` (indexed, fast)
- **buyQueue activity**: `SELECT DISTINCT DATE(timeEntered) as d FROM buyQueue WHERE DATE(timeEntered) BETWEEN :from AND :to` (single query for entire window)
- **Daily close check**: `SELECT closingDate FROM drsDailySFileData WHERE closingDate BETWEEN :from AND :to` (single query, unique index on closingDate)
- **Store hours**: `SELECT dayOfWeek, isClosed FROM storeOperatingHours WHERE typeNum = :t` (7 rows max, cached in memory)
- **Batch operations**: Group gap upserts into multi-row INSERT...ON DUPLICATE KEY UPDATE where possible

#### Integration Patterns

- **Ably publish**: Fire-and-forget with throttle check. If throttled or failed, job continues — next cycle will retry
- **Desktop response**: Desktop posts to existing endpoints. No new callback/webhook needed. Server detects resolution by re-checking data existence on next scan

## Architecture Decisions

- [x] **ADR-1 Job Scope: per_store (not global)**
  - Rationale: Per-store scope gives each store its own execution with timeout, retry, and progress tracking. The scheduler automatically handles store iteration. A global job would need to manually iterate stores and risk timeout for large store counts
  - Trade-offs: More execution records in the database (one per store per run), but better isolation and debuggability
  - User confirmed: **Yes (2026-04-01)**

- [x] **ADR-2 Gap Tracking: Dedicated table (not TaskEngine execution data)**
  - Rationale: TaskEngine execution data is per-run and gets archived. Gap state needs to persist across runs (a gap detected on Monday needs to be checked again on Tuesday). A dedicated `posDataGaps` table with `UNIQUE(typeNum, gapDate)` gives us persistent, queryable gap state
  - Trade-offs: New table to maintain, but gap lifecycle (detected → commanded → resolved) is cleanly modeled
  - User confirmed: **Yes (2026-04-01)**

- [x] **ADR-3 Ably Action Prefix: `drs:catchup:*`**
  - Rationale: Uses colon-separated namespace matching existing patterns (`workbook:task:complete`, `comeback_cash.settings_updated`). The `drs:` prefix ties it to the DRS system. Desktop app can filter on action prefix
  - Trade-offs: Slightly verbose, but prevents collision with existing ~20 action types on the channel
  - User confirmed: **Yes (2026-04-01)**

- [x] **ADR-4 Boundary Reporting: HTTP POST (not Ably reverse publish)**
  - Rationale: HTTP POST is the established pattern for desktop → server communication (all existing DRS endpoints are POST). Ably reverse-publish would require the desktop to have publish permissions and would need a separate listener on the server side — unnecessary complexity
  - Trade-offs: Desktop must be online to report boundary, but that's already a requirement for receiving catchup commands
  - User confirmed: **Yes (2026-04-01)**

- [x] **ADR-5 Backfill Chunking: 30-day batches (configurable)**
  - Rationale: A 365-day backfill request as a single command could overwhelm the desktop app and the server's ingestion endpoints. 30-day chunks balance throughput with manageability. Only ONE chunk per scan cycle per gap range — the next chunk is sent after the previous one resolves
  - Trade-offs: Full year backfill takes ~12 scan cycles (12 days at daily runs). This is acceptable — backfill is not urgent
  - Alternative considered: Send all chunks at once. Rejected because it could create a data storm
  - User confirmed: **Yes (2026-04-01)**

- [x] **ADR-6 Gap Detection Queries: Per-store DB + Central DB (multi-source)**
  - Rationale: Daily close data lives in per-store DBs. Sales and buys data lives in `kiosk_sales` (central). buyQueue is per-store. This means the gap detector needs connections to both. The `BaseJob` provides `$this->getStoreDb()` for per-store, and we use `dbConnectByName('kiosk_sales')` for central
  - Trade-offs: Two DB connections per store execution, but this is standard practice in the codebase
  - User confirmed: **Yes (2026-04-01)**

## Quality Requirements

- **Performance**: Each store's gap detection must complete in <60 seconds (including all DB queries). Full 365-day window with 3 data types = ~1095 date checks. Batch queries reduce this to 3-4 queries total
- **Reliability**: Job must be idempotent — running twice produces the same result. `UNIQUE(typeNum, gapDate)` on gap table + `INSERT ON DUPLICATE KEY UPDATE` ensures this
- **Data Integrity**: Never modify or delete POS data. Only read data for gap detection. Gap tracking is append-only (gaps can only progress forward in status, never backward except admin override)
- **Observability**: All gaps tracked in queryable table. TaskEngine execution logs capture per-store results. Job reports summary metrics in `JobResult.data`

## Risks and Technical Debt

### Known Technical Issues

- No existing index on `kiosk_sales.sales.salesDate` for the typeNum+date combination — may need composite index for performance at scale
- `storeOperatingHours` tables exist but are unused in production code — first real consumer, may discover data quality issues

### Technical Debt

- The `sendEncodedData()` function in `drs.php` is a route-level function, not a class. Our `CatchupAblyPublisher` class is the cleaner pattern and could eventually replace inline publishing in DRS routes
- Multiple "oldest date" endpoint patterns exist (`SalesController`, `DailyCloseApiController`) with slightly different response formats. Our boundary endpoint follows the newer pattern

### Implementation Gotchas

- **PDO named param reuse**: Cannot reuse `:param` name in a single query (MariaDB PDO gotcha). Use unique names like `:typeNum1`, `:typeNum2`
- **Store timezone**: All date comparisons must use `stores.timezone`. Use `new DateTimeZone($store->getTimezone())` for all `DateTime` operations
- **buyQueue table name**: The buys table in store databases is named `buyQueue` NOT `buys` (per CLAUDE.md)
- **Daily close is per-store DB**: `drsDailySFileData` is in `kiosk_{typeNum}`, NOT in `kiosk_sales`. Sales/buys data is in `kiosk_sales`
- **Migration `check_query` pattern**: Use `SHOW COLUMNS` for ALTER TABLE, `SELECT id FROM...` for INSERT operations
- **INT column comparisons**: Use `> 0` not `<> ''` for INT columns to avoid MariaDB strict mode errors

## Test Specifications

### Critical Test Scenarios

**Scenario 1: Store with gaps detected and commands sent**
```gherkin
Given: Store pc00 has drsApiKey set and has been syncing data
And: Sales data exists for all dates except 2026-03-15 and 2026-03-20
And: buyQueue had activity on both 2026-03-15 and 2026-03-20
When: PosDataCatchupJob runs for pc00
Then: Two gaps are detected (2026-03-15, 2026-03-20)
And: Gaps are recorded in posDataGaps with status "commanded"
And: One Ably catchup command is published with both dates
And: JobResult shows gapsFound=2, commandsSent=1
```

**Scenario 2: New store with no data (full backfill)**
```gherkin
Given: Store se01 has drsApiKey set but ZERO POS data records
And: No drsEarliestDataDate is set
When: PosDataCatchupJob runs for se01
Then: Job requests full 365-day backfill
And: Backfill is chunked into 30-day batches
And: First chunk (most recent 30 days) is published
And: reportBoundary is true in the command
And: Job does NOT flag all 365 days as gaps (waits for boundary)
```

**Scenario 3: Desktop reports earliest available date**
```gherkin
Given: Store pc00 has 50 unresolved gaps before 2025-06-01
And: Desktop reports earliestDate = "2025-06-01"
When: POST /api/pc00/drs/catchup/boundary with earliestDate
Then: stores.drsEarliestDataDate is set to 2025-06-01
And: All 50 gaps before 2025-06-01 are marked "unavailable"
And: Response includes previousDate=null and new earliestDate
```

**Scenario 4: Gap resolved on next scan**
```gherkin
Given: Gap exists for pc00 on 2026-03-15 with status "commanded"
And: Desktop app has since posted sales data for 2026-03-15
When: PosDataCatchupJob runs again for pc00
Then: Gap for 2026-03-15 transitions to status "resolved"
And: resolvedAt is set to current timestamp
```

**Scenario 5: Store closed day not flagged**
```gherkin
Given: Store pc00 has storeOperatingHours with Sunday (dayOfWeek=0) as isClosed=1
And: buyQueue has NO activity on Sunday 2026-03-29
When: PosDataCatchupJob runs for pc00
Then: 2026-03-29 is NOT detected as a gap
```

**Scenario 6: buyQueue overrides store hours**
```gherkin
Given: Store pc00 has storeOperatingHours with Sunday as isClosed=1
But: buyQueue HAS activity on Sunday 2026-03-29 (store was actually open)
And: No sales data exists for 2026-03-29
When: PosDataCatchupJob runs for pc00
Then: 2026-03-29 IS detected as a gap with verificationSource="buyQueue"
```

### Test Coverage Requirements

- **Business Logic**: Gap detection algorithm (all date combinations), chunking logic, window calculation, boundary handling
- **Integration Points**: Ably publish with throttle, multi-DB queries (central + store), API key validation on boundary endpoint
- **Edge Cases**: Empty stores, stores with no drsApiKey (skipped), timezone boundary dates, stores with drsEarliestDataDate, partial data (some types present, others missing)
- **Performance**: Verify batch queries used (not per-date queries), verify `EXISTS` over `COUNT(*)` pattern
- **Idempotency**: Running job twice produces same gap state, boundary endpoint is idempotent

---

## Glossary

### Domain Terms

| Term | Definition | Context |
|------|------------|---------|
| Gap | A date where a store was expected to have POS data but doesn't | Core concept tracked in posDataGaps table |
| Catchup Command | An Ably message telling the desktop app to fetch and post missing data | Published with action `drs:catchup:request` |
| Backfill | The process of retrieving and posting historical POS data | Triggered by catchup commands for date ranges |
| Earliest Available Date | The oldest date the desktop POS system has data for | Reported by desktop, stored in stores.drsEarliestDataDate |
| Operational Day | A date when the store was open (confirmed by buyQueue or store hours) | Used to determine which dates should have POS data |

### Technical Terms

| Term | Definition | Context |
|------|------------|---------|
| typeNum | Store identifier string matching `[a-z]{2}\d+` (e.g., `pc00`, `ou00`) | Used as Ably channel name and store DB prefix |
| DRS | Desktop Resale Software — the POS integration desktop application | The system that syncs POS data to our server |
| S-File | A daily closing report from the POS system with 135+ financial fields | Stored in per-store `drsDailySFileData` table |
| buyQueue | The per-store table tracking customer buy transactions | Used as independent verifier of store operation |

### API/Interface Terms

| Term | Definition | Context |
|------|------------|---------|
| `drs:catchup:request` | Ably action type for catchup commands | Published to `{typeNum}` channel |
| Boundary Endpoint | POST `/api/{typeNum}/drs/catchup/boundary` | Desktop reports earliest available date |
| reportBoundary | Boolean flag in catchup command | Tells desktop to report its earliest date |
| requestId | UUID v4 tracking a specific catchup command | Links commands to gap records |