# Product Requirements Document: POS Data Catchup System ## Validation Checklist - [x] All required sections are complete - [x] No [NEEDS CLARIFICATION] markers remain - [x] Problem statement is specific and measurable - [x] Problem is validated by evidence (not assumptions) - [x] Context → Problem → Solution flow makes sense - [x] Every persona has at least one user journey - [x] All MoSCoW categories addressed (Must/Should/Could/Won't) - [x] Every feature has testable acceptance criteria - [x] Every metric has corresponding tracking events - [x] No feature redundancy (check for duplicates) - [x] No contradictions between sections - [x] Technical references limited to system names and table identifiers (appropriate for backend-infrastructure feature) - [x] A new team member could understand this PRD --- ## Product Overview ### Vision Ensure every store has at least one full year of complete POS transaction data (sales, buys, and daily close reports) by automatically detecting missing data and commanding the desktop DRS app to backfill gaps — enabling accurate year-over-year goals, reliable analytics, and eliminating silent data loss. ### Problem Statement When the desktop DRS app experiences connectivity issues, crashes, or misses syncing data for any reason, the server has **no way to detect the gap and no way to request the missing data**. Currently: - **No gap detection exists** for sales or buys transaction data — there is no audit log, no last-sync timestamp, and no completeness check for these data streams - **Daily close S-File data** has an `oldestDate` API but no automated monitoring - **The server cannot command the desktop app** to re-fetch data — all sync is desktop-initiated (pull-only) - **No "last seen" tracking** exists for desktop app connectivity — the server doesn't know if a desktop is online or offline - **Downstream impact is severe**: Missing POS data corrupts analytics aggregations (`storeMetricsDaily`, `storeStatsAggregate`, homepage stats), causes incorrect KPI reporting, and leads to bad business decisions based on incomplete data - **Year-over-year goals are impossible** without at least 12 months of historical data — new stores and stores with sync gaps can't set realistic sales/buy targets - **New store onboarding has no backfill mechanism** — when a store is added, there's no way to pull historical POS data from the desktop app The result is that operators and managers unknowingly rely on incomplete data, can't set data-driven goals, and have no visibility into what's missing or when it went missing. ### Value Proposition This system provides **automatic, hands-off data integrity** by: 1. **Proactively detecting gaps** — the server knows what data is missing before anyone asks 2. **Commanding backfill** — the desktop app receives targeted instructions to re-fetch specific dates 3. **Verifying completion** — the system confirms data arrived and marks gaps as resolved 4. **Providing visibility** — operators can see data completeness status per store Unlike manual re-sync (which requires someone to notice the problem), this system catches gaps automatically and resolves them without human intervention, running as a scheduled background process via the existing TaskEngine. ## User Personas ### Primary Persona: Store Operations Manager - **Demographics:** 30-55, manages daily store operations, moderate technical expertise - **Goals:** Ensure all daily reports and analytics are accurate and complete. Make data-driven decisions about staffing, inventory, and pricing - **Pain Points:** Discovers missing data days/weeks later when reports look wrong. Has no way to tell the system to "catch up" on missing days. Wastes time manually checking if data looks right ### Secondary Personas #### System Administrator - **Demographics:** Technical staff managing infrastructure and integrations - **Goals:** Monitor system health, troubleshoot sync failures, ensure all stores are reporting data correctly - **Pain Points:** No centralized view of data completeness across stores. No automated alerting when a store stops syncing. Manual troubleshooting requires checking multiple tables per store #### Desktop App Developer (DRS Team) - **Demographics:** Software developer maintaining the desktop POS integration app - **Goals:** Implement the Ably listener for catchup commands, ensure the desktop app can respond to backfill requests reliably - **Pain Points:** No documented contract for server-to-desktop commands. No standard error response format for "data unavailable for that date" ## User Journey Maps ### Primary User Journey: Automatic Gap Detection and Resolution 1. **Awareness:** A scheduled TaskEngine job runs daily (or configurable), scanning each DRS-enabled store for missing POS data within a configurable lookback window 2. **Detection:** The job cross-references expected data days (verified by `buyQueue` activity as an independent indicator of store operation) against actual POS data received (sales, buys, daily close records) 3. **Command:** For each detected gap, the system publishes an Ably event to the store's channel instructing the desktop app to fetch and post data for specific missing dates 4. **Backfill:** The desktop app receives the command, retrieves the data from its local POS system, and POSTs it to the existing API endpoints (which already handle upserts) 5. **Verification:** On next scan, the job re-checks and confirms the gap is filled. If still missing after retries, the gap is escalated (logged as persistent) 6. **Visibility:** Data completeness status is available through the TaskEngine dashboard and execution logs ### Secondary User Journey: Manual Catchup Trigger 1. **Awareness:** An operator notices incorrect reporting or a store manager reports missing data 2. **Trigger:** The operator manually dispatches the catchup job for a specific store and date range via CLI or admin dashboard 3. **Command:** Same Ably command flow as automatic — targeted dates published to the store channel 4. **Verification:** Operator can check execution logs to see if the desktop app responded and data was received ### Secondary User Journey: Desktop App Offline Handling 1. **Detection:** The catchup job publishes an Ably command, but the desktop app is offline 2. **Persistence:** The gap remains unfilled on next scan. **Note**: We do NOT rely on Ably message history or replay — Ably messages are ephemeral. The server persists gap state in the tracking table and re-publishes commands on each scan cycle 3. **Retry:** The system re-publishes the catchup command on subsequent scan runs (one command per gap per cycle — no exponential backoff needed since the scan itself is daily) 4. **Recovery:** When the desktop app comes back online and the next scan cycle runs, it receives the freshly-published catchup command and begins backfilling 5. **Escalation:** If a gap persists beyond a configurable threshold (default: 7 days unfilled), the gap status transitions to "persistent" for visibility and potential alerting ## Feature Requirements ### Must Have Features #### Feature 1: POS Data Gap Detection Job - **User Story:** As a system operator, I want the system to automatically detect missing POS data for each DRS-enabled store so that I don't have to manually check data completeness - **Acceptance Criteria:** - [ ] A TaskEngine job runs on a configurable schedule (default: daily at 3 AM) for each active store with a `drsApiKey` - [ ] The job checks for missing data across three data types: sales transactions, buys transactions, and daily close S-File reports - [ ] The job uses `buyQueue` activity as an independent verifier — if a store had buys in `buyQueue` on a given day, that day is expected to have POS data - [ ] Store operating hours (`storeOperatingHours`, `storeHolidayHours`) are consulted to avoid flagging closed days as gaps - [ ] Lookback window is configurable per store (default: 365 days — one full year of historical data for goal-setting) - [ ] The gap detection window respects the store's "earliest available date" if the desktop app has reported one (see Feature 5) - [ ] For stores that have never synced any data, the job requests a full backfill from the desktop app up to the lookback limit - [ ] Detected gaps are logged with store, date, and data type(s) missing - [ ] Job completes within 600-second timeout even for stores with a full 365-day lookback window (achieved via batch date queries, not per-date queries) #### Feature 2: Ably Catchup Command Publishing - **User Story:** As the system, I want to publish targeted commands to the desktop app via Ably so that it knows exactly which dates and data types to re-fetch - **Acceptance Criteria:** - [ ] Catchup commands are published to the store's `{typeNum}` Ably channel using a new action type (e.g., `drs:catchup:request`) - [ ] Each command specifies: the data types needed, the date or date range, and a unique request ID for tracking - [ ] Commands respect Ably rate limits via existing `AblyPublishThrottle` - [ ] If multiple dates are missing, commands are batched into date ranges of up to 30 days (configurable) — not one Ably message per date - [ ] Maximum of one Ably command per contiguous date range per store per scan cycle - [ ] Command payload format is documented as a versioned contract for the desktop app team #### Feature 3: Gap Tracking and Status - **User Story:** As a system administrator, I want to see the status of detected gaps and whether they've been resolved so that I can identify persistent sync problems - **Acceptance Criteria:** - [ ] Each detected gap is tracked with: store, date, data type, first detected timestamp, last command sent timestamp, status (detected/commanded/resolved/persistent) - [ ] Gaps transition to "resolved" when the expected data appears on subsequent scans - [ ] Gaps transition to "persistent" after remaining unresolved for a configurable threshold (default: 7 days since first detection) - [ ] An additional status "unavailable" exists for gaps before the store's earliest available date (auto-resolved, not treated as failures) - [ ] Gap history is queryable by store, date range, and status - [ ] TaskEngine execution logs capture gap counts per store per run #### Feature 4: Manual Catchup Dispatch - **User Story:** As an operator, I want to manually trigger a catchup for a specific store and date range so that I can handle ad-hoc data recovery - **Acceptance Criteria:** - [ ] A CLI command allows dispatching the catchup job for a specific store: `php userfrosting/bin/task job:dispatch pos-data-catchup --store=pc00 --payload='{"startDate":"2026-03-01","endDate":"2026-03-15"}'` - [ ] Manual dispatch bypasses the gap detection logic and directly commands the desktop app for the specified dates - [ ] Manual dispatch uses the same Ably command format as automatic runs - [ ] Execution is logged with `triggerType = 'manual'` #### Feature 5: Store Data Boundary (Earliest Available Date) - **User Story:** As the system, I want the desktop app to report the earliest date it has POS data for so that I stop requesting data that doesn't exist and avoid persistent false-positive gaps - **Acceptance Criteria:** - [ ] A new API endpoint accepts the desktop app's "earliest available date" per store - [ ] The earliest available date is stored per-store and used to bound the gap detection window - [ ] If the desktop app has not yet reported an earliest date, the system requests the full lookback window (365 days) — the desktop app is expected to respond with its boundary - [ ] The earliest available date can be updated (e.g., if the POS system is replaced and older data becomes unavailable) - [ ] Gaps detected for dates before the earliest available date are automatically resolved as "unavailable" (not treated as persistent failures) - [ ] The Ably catchup command includes a field asking the desktop to report its earliest available date if it hasn't already ### Should Have Features #### Feature 6: Catchup Progress Reporting - **User Story:** As an operator, I want to see real-time progress of a catchup operation so that I know it's working and can estimate time remaining - **Acceptance Criteria:** - [ ] The TaskEngine job reports progress percentage based on phases completed within the scan (e.g., gap detection 0-40%, command publishing 40-80%, resolution checks 80-100%) - [ ] Progress is visible in the TaskEngine dashboard - [ ] Per-store results are included in the job result: `gapsFound`, `commandsSent`, `gapsResolved`, `gapsPersistent`, `scanDuration` ### Could Have Features #### Feature 7: Admin Dashboard Widget - **User Story:** As a store operations manager, I want a dashboard view showing data completeness across stores so I can spot problems at a glance - **Acceptance Criteria:** - [ ] A widget displays per-store data completeness percentage for the last N days - [ ] Stores with persistent gaps are highlighted - [ ] Clicking a store shows the specific missing dates and their status #### Feature 8: Automated Alerting - **User Story:** As a system administrator, I want to be notified when a store has persistent data gaps so I can investigate before it impacts reporting - **Acceptance Criteria:** - [ ] Email notification sent when a gap has been unresolved for more than N days (configurable) - [ ] Notification includes store, date(s), data type(s), and number of catchup attempts - [ ] Uses existing `FailureNotifier` pattern from TaskEngine #### Feature 9: Desktop App Acknowledgment - **User Story:** As the system, I want to know if the desktop app received and is processing a catchup command so that I can distinguish between "offline" and "failed" - **Acceptance Criteria:** - [ ] The desktop app can POST an acknowledgment to a new API endpoint confirming receipt of a catchup command (keyed by request ID) - [ ] Acknowledgment updates the gap status from "commanded" to "acknowledged" - [ ] Lack of acknowledgment within a configurable window is logged as a potential offline scenario ### Won't Have (This Phase) - **Automatic POS data generation/estimation** — We won't fabricate missing data; only request it from the source - **Desktop app installation or update management** — Out of scope; desktop team manages their own releases - **Metadata catchup (brands, categories, etc.)** — Metadata changes rarely and doesn't have date-based gaps; excluded from Phase 1 - **Real-time sync monitoring** — This system is batch-oriented (scheduled scans), not real-time event-driven - **Multi-POS system support** — Assumes one desktop DRS app per store; multi-system scenarios are future scope - **Record-level completeness verification** — Phase 1 checks date-level presence only ("any records on this date?"), not transaction-count accuracy ("all 847 items posted?") ## Detailed Feature Specifications ### Feature: POS Data Gap Detection Job (Most Complex) **Description:** A per-store TaskEngine job that scans for missing POS transaction data by cross-referencing expected operational days (determined by `buyQueue` activity and store hours) against actual POS data received. Publishes Ably commands for missing dates. **User Flow:** 1. Scheduler triggers the job for each active store with `drsApiKey` set 2. Job checks if the store has an "earliest available date" on record - If NOT: This is a new/unconfigured store. Job sends a full backfill request (up to 365 days) AND asks the desktop app to report its earliest available date - If YES: Job uses the earliest available date as the lower bound for gap detection 3. Job determines the gap detection window: from `max(earliestAvailableDate, today - lookbackDays)` to `yesterday` (today excluded per Rule 4) 4. Within that window, job determines expected operational days using: - a. `buyQueue` records — if there are buys on a date, the store was open (independent verifier) - b. `storeOperatingHours` / `storeHolidayHours` — for days with no buyQueue activity, check if the store was scheduled to be open 5. For each expected operational day, job checks if POS data exists: - Sales: any records in `kiosk_sales.sales` for that `typeNum` and `salesDate` - Buys: any records in `kiosk_sales.buys` for that `typeNum` and `buyDate` - Daily Close: a record in store DB `drsDailySFileData` for that `closingDate` 6. Missing dates are collected and grouped by data type 7. For large gaps (30+ days), dates are chunked into batch-sized ranges (default: 30-day chunks) 8. For each batch of missing dates, an Ably catchup command is published 9. All gaps are recorded in the tracking system with status "commanded" 10. Job reports summary: stores scanned, gaps found, commands sent, new stores detected **Business Rules:** - Rule 1: A day is "expected" to have POS data if EITHER the `buyQueue` has activity for that date OR the store was scheduled to be open per `storeOperatingHours` - Rule 2: The `buyQueue` check takes precedence — if a store had buys but was supposedly "closed" per hours config, the day is still expected (the store was actually open) - Rule 3: If a day has partial data (e.g., sales exist but daily close is missing), only the missing data types are requested - Rule 4: Today's date is excluded from gap checks — data for today may not be complete yet - Rule 5: The most recent 1 day beyond today is also excluded (configurable buffer) to account for end-of-day processing - Rule 6: If the desktop app has never synced any data (no records at all), the system requests a FULL BACKFILL up to the lookback limit (default 365 days). This is the "new store onboarding" scenario — we need historical data for goal-setting - Rule 7: The gap detection window is bounded by the store's "earliest available date" (reported by the desktop app via Feature 5). The system never requests data before this date. If no earliest date has been reported yet, the system requests the full lookback window and waits for the desktop to report its boundary - Rule 8: Catchup commands for the same gap should not be re-sent more frequently than once per scan cycle to avoid flooding - Rule 9: When a store with `drsApiKey` has integration enabled, ALL three data types (sales, buys, daily close) are expected. There is no per-data-type opt-out - Rule 10: For large backfill requests (e.g., 365 days of missing data), commands should be chunked into manageable date ranges (e.g., 30-day batches) to avoid overwhelming the desktop app **Edge Cases:** - **Store just onboarded, no historical data:** → The job requests a FULL BACKFILL up to 365 days. The desktop app reports its "earliest available date" and the system adjusts its window accordingly. Backfill commands are chunked into 30-day batches to be manageable - **New store with less than 1 year of POS history:** → Desktop app reports its earliest available date (e.g., store opened 6 months ago). The system records this boundary per-store and stops requesting data before that date. No persistent gap escalation for dates before the boundary - **Desktop app offline for extended period:** → Gaps accumulate. Commands are re-sent each scan cycle. After configurable threshold (e.g., 7 days of persistent gaps), status escalates to "persistent" - **Store temporarily closed (holiday/renovation):** → `storeHolidayHours.isClosed = 1` marks the day as closed. If `buyQueue` also has no activity, day is not expected. No false positive - **POS system replaced mid-period:** → Historical data may be unavailable from new system. Desktop app reports new "earliest available date" reflecting the replacement date. Old gaps before the boundary are auto-resolved as "unavailable" - **Partial day data (some sales but not all):** → We can only detect date-level presence, not completeness within a day. The system checks "any records exist?" not "all records present?" This is an acceptable limitation for Phase 1 - **Timezone mismatch between server and POS:** → Store timezone (`stores.timezone`) must be used for all date calculations. A 10 PM sale in Pacific time is the same day even if it's next day in UTC - **DST transitions:** → On spring-forward day (23 hours) and fall-back day (25 hours), date boundaries are determined by the store's timezone. Gap detection operates on calendar dates, not hour counts — a "day" is always midnight-to-midnight in the store's local time regardless of DST - **Race condition: desktop app syncing while job runs:** → Non-issue because gap detection is read-only and commands are idempotent (re-posting data uses upsert) - **Multiple desktop app instances per store:** → Only one should respond to catchup commands. The desktop team should handle deduplication on their side (first responder wins) - **Ably rate limits exceeded:** → `AblyPublishThrottle` prevents exceeding limits. If throttled, commands are deferred to next cycle. Not a data loss scenario — just delayed - **Store switches from DRS to non-DRS:** → Removing `drsApiKey` from the store excludes it from future scans. Existing gap records remain for historical reference - **Large backfill overwhelms desktop app:** → Backfill is chunked into configurable batch sizes (default: 30-day ranges). The next batch is only sent on the next scan cycle after the previous batch is resolved, preventing the desktop from being overloaded ## Success Metrics ### Key Performance Indicators - **Data Completeness Rate:** Percentage of expected data-days that have POS data, per store and across all stores. Target: >98% within 48 hours of the operational day - **Historical Coverage:** Percentage of DRS-enabled stores with 365+ days of complete POS data (or data back to their earliest available date). Target: >95% within 30 days of system launch - **Gap Resolution Time:** Average time between gap detection and data receipt. Target: <24 hours for stores with online desktop apps - **Persistent Gap Rate:** Percentage of detected gaps that remain unresolved after 7 days. Target: <2% (indicating desktop app or configuration issues) - **False Positive Rate:** Percentage of flagged gaps where the store was actually closed (no data expected). Target: <5% - **Earliest Date Reporting:** Percentage of DRS-enabled stores that have reported their earliest available date. Target: 100% within 7 days of system launch - **Job Reliability:** Percentage of scheduled job runs that complete successfully. Target: >99% ### Tracking Requirements | Event | Properties | Purpose | |-------|------------|---------| | `catchup.scan.completed` | `typeNum`, `gapsFound`, `commandsSent`, `duration`, `lookbackDays` | Measure scan effectiveness and performance | | `catchup.gap.detected` | `typeNum`, `date`, `dataTypes[]`, `verificationSource` (buyQueue/storeHours) | Track what's missing and why we think it should exist | | `catchup.command.sent` | `typeNum`, `requestId`, `dateRange`, `dataTypes[]` | Track commands issued to desktop apps | | `catchup.command.acknowledged` | `typeNum`, `requestId`, `acknowledgedAt` | Confirm desktop received the command | | `catchup.gap.resolved` | `typeNum`, `date`, `dataTypes[]`, `resolutionTime` | Measure time-to-resolution | | `catchup.gap.persistent` | `typeNum`, `date`, `dataTypes[]`, `daysSinceDetected`, `attemptCount` | Identify systemic problems | | `catchup.store.skipped` | `typeNum`, `reason` (no_drs_key/inactive) | Understand coverage | | `catchup.store.backfill_requested` | `typeNum`, `dateRange`, `isNewStore`, `totalDaysRequested` | Track full backfill requests for new/unconfigured stores | | `catchup.store.earliest_date_reported` | `typeNum`, `earliestDate`, `reportedAt` | Track when desktop apps report their data boundary | | `catchup.store.earliest_date_updated` | `typeNum`, `previousDate`, `newDate`, `reason` | Track boundary changes (e.g., POS system replacement) | --- ## Dependencies | Dependency | Owner | Status | Notes | |------------|-------|--------|-------| | Desktop DRS app implements Ably catchup listener | DRS Team | Pending | Server side is independently testable. Commands are logged. Desktop can implement when ready | | Desktop DRS app implements boundary reporting (earliest date POST) | DRS Team | Pending | Server provides the API endpoint; desktop team calls it | | Ably service availability | External (Ably) | Active | Existing dependency; publisher fails gracefully if unavailable | | TaskEngine scheduler running | Infrastructure | Active | Already running for existing jobs | | `storeOperatingHours` / `storeHolidayHours` data maintained | Store Managers | Assumed | If not maintained, system falls back to buyQueue-only verification | | Existing POS data API endpoints remain upsert-safe | Server Team (us) | Active | Must not change — these are existing contracts | ## Constraints and Assumptions ### Constraints - **Existing API endpoints must not change** — The desktop app already posts to `/api/{typeNum}/sales/salesDay/{date}`, `/api/{typeNum}/buys/buysDay`, and `/api/{typeNum}/daily-close`. These endpoints use upsert logic and are safe for re-posting - **Ably message rate limits** — 50 messages/sec per channel. Catchup commands must be batched, not one per date - **TaskEngine per-store timeout** — Default 300 seconds per store execution. Gap detection queries must be efficient - **Desktop app Ably listener already exists** — We are extending an existing listener, not building from scratch. New action types must not conflict with existing ones - **Store-level database isolation** — Daily close data is in per-store DBs, while sales/buys are in the central `kiosk_sales` DB. Gap detection must query both ### Assumptions - **Desktop app team will implement their side** — This PRD covers the server side. The desktop team receives a documented Ably command contract and builds the response logic - **Ably messages are ephemeral (no history/replay relied upon)** — We do NOT depend on Ably message history or replay features. The server persists gap state in the database and re-publishes commands on each scan cycle. If the desktop is offline when a command is published, it will receive the next command on the next scan cycle after it comes back online - **Desktop apps reconnect to Ably after going offline** — Ably's built-in reconnection handles temporary outages. For extended offline periods, the server re-publishes commands on the next scan cycle - **buyQueue data is reliable as an operational indicator** — If a store processed buys through our system, POS data should exist for that day - **Store operating hours are maintained** — The `storeOperatingHours` and `storeHolidayHours` tables are kept current by store managers - **One desktop DRS app per store** — Each store has at most one DRS instance listening on its Ably channel - **POS data is available on the desktop app's local system** — The desktop can retrieve historical data from its local POS for any date back to its "earliest available date" - **Desktop app can report its earliest available date** — The desktop team will implement the API call to report the oldest date the POS has data for. This is the boundary beyond which no data exists (e.g., store opened 6 months ago) - **All DRS-enabled stores sync all three data types** — If a store has `drsApiKey` set, it is expected to provide sales, buys, AND daily close data. No partial integrations ## Risks and Mitigations | Risk | Impact | Likelihood | Mitigation | |------|--------|------------|------------| | Desktop app doesn't implement listener in time | High — no catchup possible | Medium | Server side is fully testable independently. Commands are logged. Desktop can catch up when ready | | buyQueue has no activity for a day that should have POS data (store open but no buys) | Medium — false negative (gap not detected) | Low | Fall back to `storeOperatingHours` as secondary check. Day is flagged if hours say open even without buyQueue activity | | Gap detection queries are slow for stores with large data volumes | Medium — job timeouts | Low | Use date-indexed queries with bounded lookback. Use `EXISTS` subqueries instead of `COUNT(*)` | | Ably channel overloaded with catchup commands during mass outage recovery | Medium — throttling delays | Low | Batch dates into single commands. Stagger store processing. Use existing throttle mechanism | | False positives on days store was open but POS system was offline (buyQueue had activity but POS didn't) | Low — unnecessary commands sent | Medium | Acceptable — desktop app responds with "no data available" and gap is marked accordingly. No harm in asking | | Desktop app sends duplicate data on retry | Low — data integrity | Low | All existing API endpoints use `ON DUPLICATE KEY UPDATE` (upsert). Re-posting is idempotent | | Store timezone calculation errors | High — wrong dates checked | Low | Use `stores.timezone` column consistently. All date comparisons in store-local time | | Gap tracking table grows unbounded | Low — storage/performance | Medium | Archive resolved gaps older than configurable retention period (default: 90 days) | | Full year backfill overwhelms desktop app or server | High — performance/stability | Medium | Chunk backfill into 30-day batches. Only send next batch after previous is resolved. Rate-limit via scan cycle frequency | | Desktop app never reports earliest available date | Medium — persistent false gaps | Low | System defaults to full lookback window. Catchup commands include a flag requesting the boundary. Escalate to persistent after threshold | | Desktop app reports incorrect earliest date (too recent) | Medium — missing data not requested | Low | Allow admin override of earliest available date per store. Log when desktop reports a date and allow correction | ## Open Questions - [x] Which POS data types to include? → **Decision: Sales, Buys, Daily Close (not metadata)** - [x] How to identify DRS-enabled stores? → **Decision: Stores with `drsApiKey` populated** - [x] Lookback window? → **Decision: Configurable per store, default 365 days (1 year for goal-setting)** - [x] Store hours awareness? → **Decision: Use both `buyQueue` (independent verifier) and `storeOperatingHours`/`storeHolidayHours`** - [x] What about new stores with no data? → **Decision: Request full backfill. Desktop app reports its earliest available date so we know where to stop** - [x] Desktop acknowledgment priority? → **Decision: Moved to "Could Have" — just re-send commands each cycle. Desktop may be closed and won't respond** - [x] What is the Ably action name convention for catchup commands? → **Defined in SDD (ADR-3): `drs:catchup:request` with colon-separated namespace** - [x] What is the gap tracking storage mechanism — new DB table or TaskEngine execution data? → **Defined in SDD (ADR-2): Dedicated `posDataGaps` table in central DB** - [x] Exact API endpoint design for desktop to report earliest available date → **Defined in SDD: POST `/api/{typeNum}/drs/catchup/boundary` with DRS API key auth** - [x] Batch chunk size for large backfills (30 days default — confirm with desktop team) → **Defined in SDD (ADR-5): 30-day configurable chunks, one chunk per scan cycle** --- ## Supporting Research ### Competitive Analysis Data gap detection and automated backfill is a common pattern in data pipeline systems (e.g., Airflow backfill, dbt incremental models). Our approach is simpler because we have a single data source (POS) and a single consumer (our server), with an existing real-time communication channel (Ably) to signal the source. ### User Research Operators have reported discovering missing POS data days or weeks after the fact, typically when reviewing reports that "look wrong." The current workaround is to manually ask the desktop app team to re-export data — a process that can take hours and requires coordination between 2-3 people. ### Market Data POS data integrity is critical for the business: missing a single day of daily close data can cause $10K+ discrepancies in monthly financial reconciliation. For a multi-store operation, undetected gaps multiply this impact proportionally.