# Product Requirements Document ## Validation Checklist - [x] All required sections are complete - [x] No [NEEDS CLARIFICATION] markers remain - [x] Problem statement is specific and measurable - [x] Problem is validated by evidence (not assumptions) - [x] Context → Problem → Solution flow makes sense - [x] Every persona has at least one user journey - [x] All MoSCoW categories addressed (Must/Should/Could/Won't) - [x] Every feature has testable acceptance criteria - [x] Every metric has corresponding tracking events - [x] No feature redundancy (check for duplicates) - [x] No contradictions between sections - [x] No technical implementation details included - [x] A new team member could understand this PRD --- ## Product Overview ### Vision A single, unified task scheduling and queue processing system that provides complete visibility into all background operations across the BuyerKiosk platform, enabling reliable automated execution of business-critical processes. ### Problem Statement BuyerKiosk currently operates **5+ independent scheduling and queue systems** with different technologies (Beanstalkd, Redis, Jobby, manual cron), different monitoring approaches, and inconsistent patterns. This fragmentation causes: 1. **Operational Complexity**: DevOps must maintain multiple queue technologies (Beanstalkd for FiveStars, Redis for SMS, manual cron for others) 2. **Lack of Visibility**: No unified dashboard to see job status across all systems - requires checking multiple log files and Redis CLI 3. **Missing Critical Scheduling**: The Event Management system has phase transition logic (`EventPhaseProcessor`) but it is **never called on a schedule** - events don't automatically transition phases 4. **Inconsistent Reliability**: SMS system has sophisticated retry logic; FiveStars has basic error recording; other scripts have no retry mechanism 5. **Difficult Debugging**: Job failures require checking different log locations per system; no centralized execution history **Evidence**: Codebase analysis found: - FiveStars: `/FSRunner/` using Beanstalkd daemon - SMS: `/tasker/` using Redis + Jobby + Worker Pool - Employee Sync: `/userfrosting/scripts/employee-sync.php` (manual cron) - QuickBooks: `/userfrosting/workers/quickbooks-sync-worker.php` (manual cron) - Events: `EventPhaseProcessor` class exists but has no scheduling mechanism ### Value Proposition The Unified Task Engine will: - **Reduce operational complexity** by consolidating on a single queue technology (Redis) - **Provide complete visibility** through an admin dashboard showing all job status, history, and worker health - **Enable reliable execution** of the Event system phase transitions (currently broken) - **Standardize reliability** with consistent retry logic, failure notifications, and audit trails - **Simplify debugging** with centralized execution logs and real-time monitoring ## User Personas ### Primary Persona: System Administrator / DevOps Engineer - **Demographics:** Technical staff responsible for system operations, high technical expertise - **Goals:** - Ensure all background jobs run reliably without manual intervention - Quickly identify and resolve job failures before they impact business - Add or modify job schedules without code deployments - Monitor system health and capacity planning - **Pain Points:** - Currently must check 5+ different locations to verify all jobs are running - No alerting when jobs fail - discovers problems reactively - Changing a job schedule requires code changes and deployment - Cannot see execution history or trends ### Secondary Personas #### Store Manager - **Demographics:** Non-technical store operator, uses admin dashboard daily - **Goals:** - Understand why certain automated processes (SMS, events) aren't working - See if scheduled marketing campaigns were sent successfully - **Pain Points:** - No visibility into whether background jobs affecting their store are running - Must contact technical support to debug job issues #### Developer - **Demographics:** Software engineer adding new features, high technical expertise - **Goals:** - Add new scheduled jobs easily following established patterns - Debug job failures during development - Test jobs manually before deploying to production - **Pain Points:** - Multiple different patterns for scheduling make it confusing which to follow - No standard way to test scheduled jobs locally ## User Journey Maps ### Primary User Journey: Job Failure Investigation 1. **Awareness:** Admin receives email notification that a job has failed 2. **Consideration:** Admin opens Task Engine dashboard to see failure details, reviews error message and stack trace 3. **Adoption:** Admin uses dashboard retry button to re-run the failed job with one click 4. **Usage:** Admin monitors real-time execution status, sees job complete successfully 5. **Retention:** Admin trusts the system to notify them of issues, checks dashboard for trends ### Secondary User Journeys #### Adding a New Scheduled Job 1. **Awareness:** Developer needs to add new background task (e.g., daily report generation) 2. **Consideration:** Developer reviews existing job patterns in Task Engine 3. **Adoption:** Developer creates new Job class following BaseJob template 4. **Usage:** Developer adds job definition via dashboard or migration, tests with manual dispatch 5. **Retention:** Job runs on schedule automatically, developer monitors via dashboard #### Capacity Planning 1. **Awareness:** Admin notices jobs taking longer to complete 2. **Consideration:** Admin reviews queue depths and worker utilization in dashboard 3. **Adoption:** Admin increases worker count via configuration 4. **Usage:** Admin monitors throughput improvements in real-time 5. **Retention:** Admin uses dashboard metrics for ongoing capacity planning ## Feature Requirements ### Must Have Features #### Feature 1: Unified Job Scheduler - **User Story:** As a system administrator, I want a single scheduler daemon that checks all job schedules every minute so that I only need one cron entry to manage all scheduled tasks - **Acceptance Criteria:** - [ ] Scheduler reads job definitions from database - [ ] Scheduler evaluates cron expressions to determine which jobs are due - [ ] Scheduler dispatches jobs to appropriate Redis queues - [ ] Scheduler supports both "global" jobs (run once) and "per-store" jobs (run for each active store) - [ ] Single cron entry `* * * * *` runs the scheduler #### Feature 2: Worker Pool Management - **User Story:** As a system administrator, I want workers that pull jobs from Redis queues and execute them reliably so that background tasks process without manual intervention - **Acceptance Criteria:** - [ ] Workers register heartbeats to track health - [ ] Worker manager maintains target number of worker processes - [ ] Workers automatically restart if they crash - [ ] Workers support graceful shutdown on SIGTERM - [ ] Workers process jobs with configurable timeouts #### Feature 3: Database-Driven Job Definitions - **User Story:** As a system administrator, I want to add or modify job schedules in the database so that I don't need code deployments to change schedules - **Acceptance Criteria:** - [ ] Job definitions stored in MySQL table with name, schedule, queue, class, config - [ ] Jobs can be enabled/disabled without removing the definition - [ ] Job configuration (batch size, etc.) stored as JSON - [ ] Schedule changes take effect on next scheduler run (within 1 minute) #### Feature 4: Job Execution History - **User Story:** As a system administrator, I want to see a history of all job executions so that I can audit what ran and debug failures - **Acceptance Criteria:** - [ ] Every job execution creates a record with status, timestamps, duration - [ ] Failed executions include error message and stack trace - [ ] Execution records link to specific job definition and store (if per-store) - [ ] History retained for at least 30 days #### Feature 5: Admin Dashboard - **User Story:** As a system administrator, I want a web dashboard to monitor job status so that I have visibility into all background operations - **Acceptance Criteria:** - [ ] Dashboard shows summary cards: active workers, pending jobs, running jobs, failed jobs (24h) - [ ] Dashboard shows queue depths for each queue type - [ ] Dashboard lists recent executions with status, duration, timestamps - [ ] Dashboard lists all scheduled jobs with next run time - [ ] Dashboard requires admin permission to access #### Feature 6: Failure Notifications - **User Story:** As a system administrator, I want to receive email notifications when jobs fail so that I can respond to issues promptly - **Acceptance Criteria:** - [ ] Email sent when job fails after all retry attempts exhausted - [ ] Email includes job name, store, error message, dashboard link - [ ] Notification recipients configurable per job or globally - [ ] Rate limiting to prevent notification floods #### Feature 7: Automatic Retry with Backoff - **User Story:** As a system administrator, I want failed jobs to retry automatically with exponential backoff so that transient failures resolve without manual intervention - **Acceptance Criteria:** - [ ] Jobs retry up to configurable max attempts (default 3) - [ ] Retry delay increases exponentially (1min, 2min, 4min, etc.) - [ ] Retry attempts tracked in execution record - [ ] Job marked as permanently failed only after max retries exhausted ### Should Have Features #### Feature 8: Manual Job Dispatch - **User Story:** As a developer, I want to manually trigger a job from the dashboard or CLI so that I can test jobs without waiting for scheduled time - **Acceptance Criteria:** - [ ] Dashboard "Run Now" button dispatches job immediately - [ ] CLI command `task:dispatch --store=` available - [ ] Manual dispatches appear in execution history with "manual" trigger type #### Feature 9: Job Execution Logs - **User Story:** As a developer, I want to see detailed logs for each job execution so that I can debug issues - **Acceptance Criteria:** - [ ] Jobs can log messages at debug/info/warning/error levels - [ ] Logs stored per execution in database - [ ] Logs viewable in dashboard execution detail view - [ ] Log retention configurable (default 7 days) #### Feature 10: Worker Health Monitoring - **User Story:** As a system administrator, I want to see worker status and health so that I can ensure adequate processing capacity - **Acceptance Criteria:** - [ ] Dashboard shows list of active workers with status (idle/busy) - [ ] Dashboard shows current job for busy workers - [ ] Dashboard shows jobs processed and failed counts per worker - [ ] Stale workers (no heartbeat > 60s) highlighted as unhealthy #### Feature 11: Abort Running Tasks - **User Story:** As a system administrator, I want to cancel a job that's currently executing so that I can stop runaway or stuck jobs - **Acceptance Criteria:** - [ ] Dashboard shows "Abort" button for running jobs - [ ] Abort sends SIGTERM to worker process handling the job - [ ] Job marked as "cancelled" in execution history - [ ] Worker gracefully handles abort and returns to idle state - [ ] CLI command `task:abort ` available #### Feature 12: Progress Support - **User Story:** As a system administrator, I want to see progress percentage for long-running jobs so that I can monitor their completion status - **Acceptance Criteria:** - [ ] Jobs can report progress (0-100%) during execution - [ ] Progress stored in execution record and Redis for real-time updates - [ ] Dashboard shows progress bar for running jobs with progress data - [ ] Progress updates visible without page refresh #### Feature 13: Job Lifecycle Hooks - **User Story:** As a developer, I want to define before/after/failed callbacks for jobs so that I can perform setup, cleanup, and error handling - **Acceptance Criteria:** - [ ] Jobs can implement `beforeHandle()` method called before execution - [ ] Jobs can implement `afterHandle()` method called after successful execution - [ ] Jobs can implement `failed()` method called when job fails (already planned) - [ ] Hooks execute in worker context with access to job payload and config #### Feature 14: Orphan Job Detection - **User Story:** As a system administrator, I want the system to automatically detect jobs stuck in "running" state so that crashed jobs don't block the system - **Acceptance Criteria:** - [ ] Scheduler checks for jobs in "running" state longer than 2x their timeout - [ ] Orphaned jobs automatically marked as "failed" with "orphan_detected" error - [ ] Orphaned jobs requeued for retry if retries remaining - [ ] Dashboard shows orphan detection events in execution history - [ ] Alert sent when orphan jobs detected (via existing notification system) ### Could Have Features #### Feature 15: Job Dependencies - **User Story:** As a developer, I want to define job dependencies so that complex workflows execute in correct order - **Acceptance Criteria:** - [ ] Job can depend on another job (runs after success) - [ ] Chained jobs trigger automatically when dependency completes - [ ] Circular dependencies prevented #### Feature 16: Queue Prioritization - **User Story:** As a system administrator, I want high-priority jobs to process before low-priority jobs so that critical tasks complete faster - **Acceptance Criteria:** - [ ] Jobs have configurable priority (0-100) - [ ] Higher priority jobs dequeued first within same queue - [ ] Priority visible in dashboard #### Feature 17: Daily Digest Email - **User Story:** As a system administrator, I want a daily summary email of job statistics so that I can monitor trends without checking dashboard - **Acceptance Criteria:** - [ ] Email sent daily at configurable time - [ ] Includes: jobs run, success rate, failures, slowest jobs - [ ] Configurable recipients ### Won't Have (This Phase) 1. **Distributed Scheduler**: Will use single scheduler daemon, not distributed/HA scheduler 2. **External Queue Backend**: Will use Redis only, not support for RabbitMQ/SQS 3. **Real-time Streaming Logs**: Logs stored and viewed after execution, not streamed live 4. **Job Rate Limiting**: Per-job rate limits not in initial scope 5. **Multi-tenant Isolation**: All jobs share same Redis and worker pool 6. **Web-based Job Code Editor**: Jobs defined as PHP classes, not editable via UI ## Detailed Feature Specifications ### Feature: Admin Dashboard **Description:** A web-based monitoring interface that provides real-time visibility into all scheduled jobs, active workers, queue depths, and execution history. The dashboard serves as the single source of truth for all background job operations. **User Flow:** 1. User navigates to `/admin/:typeNum/tasks` 2. System displays dashboard with summary cards showing key metrics 3. User views queue status section showing pending jobs per queue 4. User views recent executions section showing last 10 completed/failed jobs 5. User clicks on failed execution to view details 6. System shows execution detail with error message, stack trace, and retry button 7. User clicks retry button 8. System dispatches job and shows new execution in pending state **Business Rules:** - Rule 1: Dashboard only accessible to users with `uri_task_engine` permission - Rule 2: Summary metrics refresh every 30 seconds automatically - Rule 3: Failed jobs count only includes last 24 hours - Rule 4: Queue depths calculated from Redis LIST lengths - Rule 5: Worker status determined by heartbeat within last 60 seconds **Edge Cases:** - Scenario 1: Redis unavailable → Expected: Dashboard shows error banner, cached data displayed - Scenario 2: No active workers → Expected: Warning banner displayed, queue depths still shown - Scenario 3: Job execution takes > 10 minutes → Expected: Still shows as "running" with duration updating - Scenario 4: User without permission accesses dashboard → Expected: 403 Forbidden response ## Success Metrics ### Key Performance Indicators - **Adoption:** 100% of existing scheduled tasks migrated to Task Engine within 3 months - **Engagement:** Dashboard accessed by admin users at least 5x per week - **Quality:** 99.9% scheduled job execution success rate (excluding legitimate business errors) - **Business Impact:** Zero missed Event phase transitions due to scheduling issues ### Tracking Requirements | Event | Properties | Purpose | |-------|------------|---------| | job_dispatched | job_name, queue, store, trigger_type | Track job volume and scheduling accuracy | | job_completed | job_name, queue, store, duration_ms, attempt | Measure success rate and performance | | job_failed | job_name, queue, store, error_type, attempt | Identify failure patterns | | job_retried | job_name, queue, store, attempt_number | Track retry effectiveness | | job_aborted | job_name, store, user_id, reason | Track manual cancellations | | job_progress | job_name, store, progress_pct | Monitor long-running job progress | | job_orphaned | job_name, store, stuck_duration_s | Track orphan detection frequency | | dashboard_viewed | user_id, page | Measure dashboard adoption | | manual_dispatch | job_name, store, user_id | Track manual intervention frequency | | worker_started | worker_id, hostname | Monitor worker pool size | | worker_stopped | worker_id, reason | Track worker lifecycle | --- ## Constraints and Assumptions ### Constraints - **Technology:** Must use Redis as queue backend (already deployed and operational) - **Infrastructure:** Must run on existing server infrastructure, no new services - **Compatibility:** Must support PHP 8.x environment with Slim 2.6.2 framework - **Migration:** Must migrate existing jobs without disrupting current operations - **Permission:** Dashboard access must integrate with existing permission system ### Assumptions - Redis server has sufficient capacity for additional queue workload - PHP PCNTL extension available for worker signal handling - Email service (existing SMTP configuration) available for notifications - Existing job logic (SMS, FiveStars, etc.) can be encapsulated in Job classes without rewrite - Store database isolation pattern (`kiosk_{typeNum}`) continues unchanged ## Risks and Mitigations | Risk | Impact | Likelihood | Mitigation | |------|--------|------------|------------| | Migration disrupts existing SMS processing | High | Medium | Run new system in parallel, gradual cutover with rollback plan | | FiveStars integration breaks during Beanstalkd migration | High | Low | Extensive testing in staging, maintain Beanstalkd as fallback for 1 week | | Worker memory leaks cause crashes | Medium | Medium | Implement worker auto-restart after N jobs processed | | Redis queue fills up if workers stop | High | Low | Monitor queue depths, alert if exceeds threshold, auto-scale workers | | Dashboard performance degrades with large execution history | Low | Medium | Implement pagination, retention policy to purge old records | ## Open Questions - [x] Should we implement job priorities within queues? → **Yes, as "Could Have" feature** - [x] Do we need job chaining/dependencies for complex workflows? → **Yes, as "Could Have" feature** - [x] Should failed jobs be automatically requeued after a delay? → **Yes, exponential backoff in "Must Have"** - [x] Do we need separate queues per store for isolation? → **No, shared queues with per-store job instances sufficient** --- ## Supporting Research ### Competitive Analysis Industry-standard PHP job queue solutions analyzed: - **Laravel Queue**: Framework-integrated, supports multiple backends (Redis, SQS, database) - **Symfony Messenger**: Message bus architecture, good for decoupled systems - **Beanstalkd + Pheanstalk**: Lightweight, already in use for FiveStars - **Custom Redis**: What SMS system currently uses successfully **Conclusion**: Build custom solution modeled on Laravel Queue patterns but tailored for existing BuyerKiosk architecture, leveraging existing Redis infrastructure. ### User Research Based on analysis of existing systems and developer feedback: - Current pain point: No visibility into job status without SSH access and log checking - Current pain point: Changing schedules requires code deployment - Current pain point: Event system phase transitions not running automatically - Desired state: Single dashboard to see all background operations ### Market Data - Redis is the most popular queue backend for PHP applications - Database-driven job definitions are standard in modern job queue systems - Admin dashboards are essential for production job queue management (Laravel Horizon, Sidekiq Web)