# S001 - Unified Task Engine Specification

**Status:** Research Complete
**Created:** 2025-12-11
**Author:** Claude Code Analysis
**Priority:** Research Phase

---

## Executive Summary

This specification defines a unified task scheduler/runner system to consolidate BuyerKiosk's existing disparate scheduling and queue systems into a single, cohesive architecture.

### Problem Statement

Currently, BuyerKiosk has multiple independent scheduling and queue systems:
- **FiveStars (FSRunner):** Beanstalkd-based daemon for loyalty points sync
- **SMS Marketing:** Redis-based worker pool for SMS queue processing
- **Various Scripts:** Manual cron entries for employee sync, QuickBooks, stats aggregation
- **Event System:** Has `EventPhaseProcessor` but NO scheduling mechanism

This fragmentation causes:
- Operational complexity (multiple technologies to maintain)
- Lack of visibility (no unified dashboard)
- Inconsistent patterns (different approaches per system)
- Missing scheduling (Event system not scheduled at all)

### Solution Overview

Build a **Unified Task Engine** that:
- Consolidates all job queues on Redis
- Provides database-driven job definitions
- Offers admin dashboard for monitoring
- Uses specialized workers per queue type
- Runs from a single cron entry point

---

## Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Queue Backend | **Redis Only** | Consolidate on existing Redis infrastructure, migrate FiveStars from Beanstalkd |
| Schedule Storage | **Database-Driven** | Modify schedules without code deployments |
| Monitoring | **Admin Dashboard + Email** | Web UI for status, email alerts for critical failures |
| Module Location | **New Feature Module** | `src/BuyerKiosk/TaskEngine/` as standalone feature |
| Store Distribution | **Any Worker** | Better load balancing, simpler operations |
| Failure Alerts | **Dashboard + Email** | Critical failures trigger email notifications |

---

## Current State Analysis

### Existing Systems Inventory

#### 1. FiveStars / FSRunner
- **Technology:** Beanstalkd + Pheanstalk
- **Location:** `/FSRunner/`
- **Trigger:** Daemon with infinite loop (60-second sleep)
- **Purpose:** Sync loyalty points to FiveStars API
- **Queue:** `fsOutgoing` Beanstalk tube

**Data Flow:**
```
Sales POS → API → dailySalesData → Points calc → fsOutgoing table
    ↓
Beanstalk Queue (fsOutgoing tube)
    ↓
FSRunner Daemon (60-second loop)
    ↓
Worker → FiveStars API
```

#### 2. SMS Marketing System
- **Technology:** Redis + Worker Pool
- **Location:** `/tasker/`, `/userfrosting/workers/`
- **Trigger:** Jobby scheduler + Worker manager daemon
- **Purpose:** Process SMS marketing queue
- **Queues:** `sms:queue:pending`, `sms:queue:failed`

**Architecture:**
```
Triggers/Blasts → seller_marketing_queue (MySQL)
    ↓
Queue Pusher (cron: every minute)
    ↓
Redis Queue (sms:queue:pending)
    ↓
Worker Pool (4-16 workers)
    ↓
Twilio/Vonage API
```

#### 3. Jobby Scheduler
- **Technology:** hellogerard/jobby PHP library
- **Location:** `/userfrosting/config/jobby-sms.php`
- **Jobs Configured:**
  - SMS Queue Full: Every hour
  - SMS Queue Pending: Every 15 min
  - SMS Queue Retry: Every 30 min
  - SMS Queue Blasts: Every 5 min
  - SMS Queue Cleanup: Every 2 hours
  - SMS Queue Health: Every 10 min

#### 4. Standalone Scripts (Manual Cron)
| Script | Purpose | Suggested Schedule |
|--------|---------|-------------------|
| `quickbooks-sync-worker.php` | QB daily close sync | Nightly 10 PM |
| `employee-sync.php` | WhenIWork/Homebase sync | Every 15 min |
| `aggregate-store-stats.php` | Store stats rollups | Daily 5 AM |
| `process-triggers.php` | SMS trigger processing | Hourly |

#### 5. Event System (NOT SCHEDULED)
- **Technology:** PHP classes, no scheduler
- **Location:** `/userfrosting/src/BuyerKiosk/EventManagement/`
- **Problem:** `EventPhaseProcessor` exists but is never called on a schedule
- **Needs:**
  - Daily phase transition detection
  - Integration activation when phases change
  - Relative timing support (N days before/after event)

---

## Proposed Architecture

### High-Level Diagram

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         UNIFIED TASK ENGINE                                  │
│                      "BuyerKiosk Task Engine (BKTE)"                        │
└─────────────────────────────────────────────────────────────────────────────┘

                                    │
          ┌─────────────────────────┼─────────────────────────┐
          │                         │                         │
          ▼                         ▼                         ▼
   ┌─────────────┐         ┌─────────────┐         ┌─────────────┐
   │ SCHEDULER   │         │   QUEUE     │         │   ADMIN     │
   │  DAEMON     │         │  WORKERS    │         │  DASHBOARD  │
   └─────────────┘         └─────────────┘         └─────────────┘
          │                         │                         │
          │                         │                         │
          ▼                         ▼                         ▼
   ┌─────────────────────────────────────────────────────────────────┐
   │                       REDIS MESSAGE BROKER                       │
   │   Queues: sms, fivestars, events, email, sync, default          │
   └─────────────────────────────────────────────────────────────────┘
          │                         │                         │
          │                         │                         │
          ▼                         ▼                         ▼
   ┌─────────────────────────────────────────────────────────────────┐
   │                       MySQL (Job Registry)                       │
   │   scheduled_jobs | job_executions | worker_status | job_logs    │
   └─────────────────────────────────────────────────────────────────┘
```

### Component Overview

1. **Scheduler Daemon** - Reads job definitions from MySQL, checks cron schedules, dispatches jobs to Redis
2. **Queue Workers** - Pull jobs from Redis queues, execute job classes, report results
3. **Worker Manager** - Maintains pool of workers, health checks, auto-restart
4. **Admin Dashboard** - Web UI for monitoring, manual triggers, log viewing
5. **Job Registry** - MySQL tables storing job definitions, executions, logs

---

## Database Schema

### Core Tables (Central Database: `kiosk_buykiosk`)

```sql
-- Job Definitions (what can be scheduled)
CREATE TABLE scheduled_jobs (
    id INT PRIMARY KEY AUTO_INCREMENT,
    name VARCHAR(100) NOT NULL UNIQUE,
    displayName VARCHAR(150) NOT NULL,
    description TEXT,
    jobClass VARCHAR(200) NOT NULL,        -- PHP class to execute
    queue VARCHAR(50) DEFAULT 'default',   -- Redis queue name
    schedule VARCHAR(50) NULL,             -- Cron expression (NULL = queue only)
    isEnabled TINYINT(1) DEFAULT 1,
    priority INT DEFAULT 0,                -- Higher = more important
    timeout INT DEFAULT 300,               -- Seconds before timeout
    maxRetries INT DEFAULT 3,
    retryDelay INT DEFAULT 60,             -- Seconds between retries
    config JSON,                           -- Job-specific configuration
    scope ENUM('global', 'per-store') DEFAULT 'global',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    INDEX idx_schedule (schedule),
    INDEX idx_queue (queue),
    INDEX idx_enabled (isEnabled)
);

-- Job Executions (history of runs)
CREATE TABLE job_executions (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    jobId INT NOT NULL,
    typeNum VARCHAR(10) NULL,              -- Store context (if per-store)
    status ENUM('pending', 'running', 'completed', 'failed', 'timeout', 'cancelled') DEFAULT 'pending',
    startedAt TIMESTAMP NULL,
    completedAt TIMESTAMP NULL,
    duration DECIMAL(10,3) NULL,           -- Seconds
    workerId VARCHAR(100) NULL,
    attempt INT DEFAULT 1,
    result JSON,                           -- Output/return data
    errorMessage TEXT NULL,
    errorTrace TEXT NULL,
    queuedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_job_status (jobId, status),
    INDEX idx_status_queued (status, queuedAt),
    INDEX idx_worker (workerId),
    INDEX idx_typenum (typeNum),
    FOREIGN KEY (jobId) REFERENCES scheduled_jobs(id) ON DELETE CASCADE
);

-- Worker Status (heartbeats)
CREATE TABLE worker_status (
    id INT PRIMARY KEY AUTO_INCREMENT,
    workerId VARCHAR(100) NOT NULL UNIQUE,
    workerType VARCHAR(50) NOT NULL,       -- 'sms', 'fivestars', 'events', etc.
    hostname VARCHAR(100),
    pid INT,
    status ENUM('starting', 'idle', 'busy', 'stopping', 'stopped') DEFAULT 'starting',
    currentJobId BIGINT NULL,
    jobsProcessed INT DEFAULT 0,
    jobsFailed INT DEFAULT 0,
    lastHeartbeat TIMESTAMP NULL,
    startedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    memoryUsage INT NULL,                  -- Bytes
    INDEX idx_type_status (workerType, status),
    INDEX idx_heartbeat (lastHeartbeat),
    FOREIGN KEY (currentJobId) REFERENCES job_executions(id) ON DELETE SET NULL
);

-- Job Logs (detailed execution logs)
CREATE TABLE job_logs (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    executionId BIGINT NOT NULL,
    level ENUM('debug', 'info', 'warning', 'error') DEFAULT 'info',
    message TEXT NOT NULL,
    context JSON,
    loggedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_execution (executionId),
    INDEX idx_level_time (level, loggedAt),
    FOREIGN KEY (executionId) REFERENCES job_executions(id) ON DELETE CASCADE
);

-- Job Dependencies (for complex workflows)
CREATE TABLE job_dependencies (
    id INT PRIMARY KEY AUTO_INCREMENT,
    jobId INT NOT NULL,
    dependsOnJobId INT NOT NULL,
    dependencyType ENUM('after_success', 'after_any', 'chain') DEFAULT 'after_success',
    UNIQUE KEY (jobId, dependsOnJobId),
    FOREIGN KEY (jobId) REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
    FOREIGN KEY (dependsOnJobId) REFERENCES scheduled_jobs(id) ON DELETE CASCADE
);
```

---

## Directory Structure

```
userfrosting/src/BuyerKiosk/TaskEngine/
├── Contracts/
│   ├── JobInterface.php
│   ├── SchedulerInterface.php
│   └── WorkerInterface.php
├── Jobs/
│   ├── BaseJob.php
│   ├── SmsQueueProcessorJob.php
│   ├── FiveStarsPointsSyncJob.php
│   ├── EventPhaseProcessorJob.php
│   ├── EmployeeSyncJob.php
│   ├── QuickBooksSyncJob.php
│   ├── StoreStatsAggregationJob.php
│   └── SmsTriggerProcessorJob.php
├── Services/
│   ├── Scheduler.php
│   ├── Dispatcher.php
│   ├── WorkerManager.php
│   ├── Worker.php
│   ├── JobRegistry.php
│   └── ExecutionLogger.php
├── Models/
│   ├── ScheduledJob.php
│   ├── JobExecution.php
│   ├── WorkerStatus.php
│   └── JobLog.php
├── Controllers/
│   ├── DashboardController.php
│   ├── JobController.php
│   ├── ExecutionController.php
│   └── WorkerController.php
├── Notifications/
│   ├── JobFailedEmail.php
│   └── DailyDigestEmail.php
└── Console/
    ├── SchedulerCommand.php
    ├── WorkerCommand.php
    └── JobCommand.php

userfrosting/routes/task-engine/
├── api.php
└── pages.php

userfrosting/templates/themes/default/task-engine/
├── dashboard.html
├── jobs/
│   ├── index.html
│   └── edit.html
├── executions/
│   ├── index.html
│   └── detail.html
├── workers/
│   └── index.html
└── partials/
    └── *.html

userfrosting/migrations/input/
├── 20251211_001_task_engine_jobs.json
├── 20251211_002_task_engine_executions.json
├── 20251211_003_task_engine_workers.json
└── 20251211_004_task_engine_logs.json
```

---

## Core Components

### Job Interface

```php
namespace BuyerKiosk\TaskEngine\Contracts;

interface JobInterface
{
    public function handle(array $payload): mixed;
    public function failed(\Throwable $exception): void;
    public function getQueue(): string;
    public function getTimeout(): int;
    public function getMaxRetries(): int;
}
```

### Base Job Class

```php
namespace BuyerKiosk\TaskEngine\Jobs;

abstract class BaseJob implements JobInterface
{
    protected string $queue = 'default';
    protected int $timeout = 300;
    protected int $maxRetries = 3;
    protected ?string $typeNum = null;
    protected array $config = [];
    protected LoggerInterface $logger;

    abstract public function handle(array $payload): mixed;

    public function withStore(string $typeNum): self
    {
        $this->typeNum = $typeNum;
        return $this;
    }

    public function withConfig(array $config): self
    {
        $this->config = $config;
        return $this;
    }

    protected function getStoreDb(): PDO
    {
        if (!$this->typeNum) {
            throw new \RuntimeException('Store context not set');
        }
        return dbConnectByName('kiosk_' . $this->typeNum);
    }

    public function log(string $level, string $message, array $context = []): void
    {
        $this->logger->log($level, "[{$this->typeNum}] {$message}", $context);
    }

    public function failed(\Throwable $exception): void
    {
        $this->log('error', 'Job failed: ' . $exception->getMessage());
    }

    public function getQueue(): string { return $this->queue; }
    public function getTimeout(): int { return $this->timeout; }
    public function getMaxRetries(): int { return $this->maxRetries; }
}
```

### Scheduler Service

```php
namespace BuyerKiosk\TaskEngine\Services;

class Scheduler
{
    private PDO $db;
    private RedisClient $redis;
    private LoggerInterface $logger;

    public function run(): void
    {
        $now = new DateTime();
        $jobs = $this->getEnabledJobs();

        foreach ($jobs as $job) {
            if ($this->isDue($job, $now)) {
                if ($job['scope'] === 'per-store') {
                    $this->dispatchPerStore($job);
                } else {
                    $this->dispatch($job);
                }
            }
        }
    }

    private function dispatch(array $job, ?string $typeNum = null): void
    {
        $executionId = $this->createExecution($job, $typeNum);

        $payload = [
            'executionId' => $executionId,
            'jobClass' => $job['jobClass'],
            'config' => json_decode($job['config'], true) ?? [],
            'typeNum' => $typeNum,
            'timeout' => $job['timeout'],
            'maxRetries' => $job['maxRetries'],
            'attempt' => 1,
            'queuedAt' => time(),
        ];

        $this->redis->rpush("tasks:{$job['queue']}", json_encode($payload));
        $this->logger->info("Dispatched: {$job['name']}", ['executionId' => $executionId]);
    }

    private function dispatchPerStore(array $job): void
    {
        $stores = $this->getActiveStores();
        foreach ($stores as $store) {
            $this->dispatch($job, $store['typeNum']);
        }
    }

    private function isDue(array $job, DateTime $now): bool
    {
        if (empty($job['schedule'])) return false;
        $cron = new CronExpression($job['schedule']);
        return $cron->isDue($now);
    }
}
```

### Worker Service

```php
namespace BuyerKiosk\TaskEngine\Services;

class Worker
{
    private string $workerId;
    private array $queues;
    private RedisClient $redis;
    private PDO $db;
    private bool $running = true;

    public function run(): void
    {
        $this->register();
        $this->setupSignalHandlers();

        while ($this->running) {
            $this->heartbeat();
            $job = $this->fetchJob();

            if ($job) {
                $this->process($job);
            }
        }

        $this->unregister();
    }

    private function fetchJob(): ?array
    {
        $queues = array_map(fn($q) => "tasks:{$q}", $this->queues);
        $result = $this->redis->blpop($queues, 30);
        return $result ? json_decode($result[1], true) : null;
    }

    private function process(array $payload): void
    {
        $this->updateStatus('busy', $payload['executionId']);

        try {
            $job = $this->instantiateJob($payload['jobClass']);
            $job->withConfig($payload['config'] ?? []);

            if ($payload['typeNum']) {
                $job->withStore($payload['typeNum']);
            }

            $startTime = microtime(true);
            $result = $job->handle($payload);
            $duration = microtime(true) - $startTime;

            $this->markCompleted($payload['executionId'], $result, $duration);

        } catch (\Throwable $e) {
            $this->handleFailure($payload, $e);
        }

        $this->updateStatus('idle', null);
    }

    private function handleFailure(array $payload, \Throwable $e): void
    {
        $attempt = $payload['attempt'] ?? 1;
        $maxRetries = $payload['maxRetries'] ?? 3;

        if ($attempt < $maxRetries) {
            $delay = pow(2, $attempt) * 60; // Exponential backoff
            $payload['attempt'] = $attempt + 1;
            $this->scheduleRetry($payload, $delay);
            $this->markRetrying($payload['executionId'], $e->getMessage(), $attempt);
        } else {
            $this->markFailed($payload['executionId'], $e->getMessage(), $e->getTraceAsString());
            $this->sendFailureNotification($payload, $e);
        }
    }
}
```

---

## Job Definitions

### Initial Jobs to Implement

```sql
INSERT INTO scheduled_jobs (name, displayName, jobClass, queue, schedule, scope, timeout, maxRetries, config) VALUES
('sms_queue_processor', 'SMS Queue Processor', 'BuyerKiosk\\TaskEngine\\Jobs\\SmsQueueProcessorJob', 'sms', '*/5 * * * *', 'per-store', 600, 3, '{"batchSize": 100}'),
('fivestars_points_sync', 'FiveStars Points Sync', 'BuyerKiosk\\TaskEngine\\Jobs\\FiveStarsPointsSyncJob', 'fivestars', '* * * * *', 'global', 60, 5, '{}'),
('event_phase_processor', 'Event Phase Processor', 'BuyerKiosk\\TaskEngine\\Jobs\\EventPhaseProcessorJob', 'events', '0 * * * *', 'per-store', 300, 3, '{}'),
('employee_sync', 'Employee Sync', 'BuyerKiosk\\TaskEngine\\Jobs\\EmployeeSyncJob', 'sync', '*/15 * * * *', 'per-store', 300, 3, '{}'),
('quickbooks_sync', 'QuickBooks Daily Sync', 'BuyerKiosk\\TaskEngine\\Jobs\\QuickBooksSyncJob', 'sync', '0 22 * * *', 'per-store', 600, 3, '{}'),
('store_stats_aggregation', 'Store Stats Aggregation', 'BuyerKiosk\\TaskEngine\\Jobs\\StoreStatsAggregationJob', 'sync', '0 5 * * *', 'per-store', 300, 3, '{}'),
('sms_trigger_processor', 'SMS Trigger Processor', 'BuyerKiosk\\TaskEngine\\Jobs\\SmsTriggerProcessorJob', 'sms', '0 * * * *', 'per-store', 600, 3, '{}');
```

---

## Admin Dashboard

### Routes

```
/admin/:typeNum/tasks              → Dashboard overview
/admin/:typeNum/tasks/jobs         → Job definitions list
/admin/:typeNum/tasks/jobs/:id     → Job detail/edit
/admin/:typeNum/tasks/executions   → Execution history
/admin/:typeNum/tasks/workers      → Worker status
/admin/:typeNum/tasks/logs         → Log viewer
/admin/:typeNum/tasks/settings     → System settings
```

### Dashboard Wireframe

```
┌────────────────────────────────────────────────────────────────────────┐
│  TASK ENGINE DASHBOARD                                    [Refresh ↻] │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │
│  │   WORKERS    │  │   PENDING    │  │   RUNNING    │  │   FAILED   │ │
│  │      8       │  │     24       │  │      3       │  │     2      │ │
│  │   ● Active   │  │   in queue   │  │   jobs now   │  │  last 24h  │ │
│  └──────────────┘  └──────────────┘  └──────────────┘  └────────────┘ │
│                                                                        │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │ QUEUE STATUS                                           View All │  │
│  ├─────────────────────────────────────────────────────────────────┤  │
│  │ sms        ████████████████████░░░░░░  156 pending │ 8 workers  │  │
│  │ fivestars  ██████░░░░░░░░░░░░░░░░░░░░   24 pending │ 2 workers  │  │
│  │ events     ░░░░░░░░░░░░░░░░░░░░░░░░░░    0 pending │ 2 workers  │  │
│  │ sync       ██░░░░░░░░░░░░░░░░░░░░░░░░    4 pending │ 1 worker   │  │
│  └─────────────────────────────────────────────────────────────────┘  │
│                                                                        │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │ RECENT EXECUTIONS                                      View All │  │
│  ├─────────────────────────────────────────────────────────────────┤  │
│  │ ● SmsQueueProcessor      ou00    Completed    2.3s    12:45:23  │  │
│  │ ● EventPhaseProcessor    pa00    Completed    0.8s    12:45:01  │  │
│  │ ✗ FiveStarsPointsSync    wi00    Failed       -       12:44:58  │  │
│  │ ● EmployeeSync           ou00    Running...   -       12:44:45  │  │
│  │ ● SmsQueueProcessor      pa00    Completed    1.9s    12:44:30  │  │
│  └─────────────────────────────────────────────────────────────────┘  │
│                                                                        │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │ SCHEDULED JOBS                              [+ Add Job] [↻ Run] │  │
│  ├─────────────────────────────────────────────────────────────────┤  │
│  │ ☑ SMS Queue Processing     */5 * * * *     sms      Per-Store   │  │
│  │ ☑ FiveStars Points Sync    * * * * *       fivestars Global     │  │
│  │ ☑ Event Phase Processor    0 * * * *       events   Per-Store   │  │
│  │ ☑ Employee Sync            */15 * * * *    sync     Per-Store   │  │
│  │ ☑ QuickBooks Sync          0 22 * * *      sync     Per-Store   │  │
│  │ ☑ Store Stats Aggregation  0 5 * * *       sync     Per-Store   │  │
│  └─────────────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────┘
```

---

## CLI Commands

```bash
# Start the scheduler (runs every minute via cron)
php userfrosting/conductor task:schedule

# Start a worker (long-running daemon)
php userfrosting/conductor task:work --queue=sms,fivestars,events,sync

# Start worker manager (manages pool of workers)
php userfrosting/conductor task:manager --workers=8

# Manually dispatch a job
php userfrosting/conductor task:dispatch event_phase_processor --store=ou00

# List scheduled jobs
php userfrosting/conductor task:list

# View job status
php userfrosting/conductor task:status

# Retry failed jobs
php userfrosting/conductor task:retry --job=fivestars_points_sync
```

---

## Cron Setup

Replace all existing cron entries with a single entry:

```bash
# Single cron entry for Task Engine scheduler
* * * * * cd /path/to/buyerkiosk-web && php userfrosting/conductor task:schedule >> /var/log/task-scheduler.log 2>&1
```

---

## Email Notifications

### Job Failed Email Template

```
Subject: [BuyerKiosk] Job Failed: {job_name} on {store}

A scheduled job has failed and requires attention.

Job: {job_name}
Store: {store}
Execution ID: {execution_id}
Failed At: {timestamp}
Attempt: {attempt} of {max_retries}

Error Message:
{error_message}

Stack Trace:
{stack_trace}

View in Dashboard: {dashboard_url}

---
BuyerKiosk Task Engine
This is an automated notification.
```

---

## Migration Plan

### Phase 1: Foundation (Week 1-2)
- [ ] Create database migrations
- [ ] Build Job interface and BaseJob class
- [ ] Build Scheduler and Worker services
- [ ] Create CLI commands
- [ ] Unit tests for core components

### Phase 2: SMS Migration (Week 3-4)
- [ ] Create SmsQueueProcessorJob
- [ ] Migrate from existing Jobby/workers
- [ ] Run in parallel for validation
- [ ] Cut over to new system
- [ ] Retire old SMS worker scripts

### Phase 3: FiveStars Migration (Week 5-6)
- [ ] Create FiveStarsPointsSyncJob
- [ ] Migrate from Beanstalkd to Redis
- [ ] Update FSRunner logic to new Job class
- [ ] Test thoroughly
- [ ] Retire Beanstalkd

### Phase 4: Events & Other Jobs (Week 7-8)
- [ ] Create EventPhaseProcessorJob
- [ ] Create EmployeeSyncJob
- [ ] Create QuickBooksSyncJob
- [ ] Create StoreStatsAggregationJob
- [ ] Migrate all remaining scripts

### Phase 5: Dashboard (Week 9-10)
- [ ] Build admin dashboard pages
- [ ] Implement monitoring views
- [ ] Add email notifications
- [ ] Documentation and training

---

## Key Benefits

| Benefit | Description |
|---------|-------------|
| **Single Entry Point** | One cron job manages all scheduled tasks |
| **Database-Driven** | Add/modify jobs without deployments |
| **Unified Queue** | Redis for all job types, simpler operations |
| **Visibility** | Admin dashboard shows all job status in real-time |
| **Scalability** | Add workers dynamically, scale per queue |
| **Reliability** | Automatic retries with exponential backoff |
| **Auditability** | Full execution history and logs |
| **Per-Store Support** | Jobs can run globally or per-store |
| **Extensibility** | Easy to add new job types |

---

## Redis Queue Structure

### Keys

| Key Pattern | Type | Purpose |
|-------------|------|---------|
| `tasks:{queue}` | LIST | Pending jobs for queue |
| `tasks:{queue}:delayed` | ZSET | Delayed/retry jobs (scored by timestamp) |
| `tasks:failed` | LIST | Permanently failed jobs |
| `worker:{id}:heartbeat` | STRING (TTL: 30s) | Worker health tracking |
| `tasks:stats` | HASH | Global statistics |

### Statistics Hash Fields

```
total_queued      - Total jobs dispatched
total_processed   - Total jobs successfully completed
total_failed      - Total jobs that permanently failed
total_retried     - Total retry attempts
```

---

## Performance Considerations

### Expected Throughput

| Queue | Expected Volume | Workers Needed |
|-------|-----------------|----------------|
| sms | 1000-5000/hour | 4-8 |
| fivestars | 100-500/hour | 2 |
| events | 50-200/hour | 2 |
| sync | 10-50/hour | 1-2 |

### Memory/CPU

- Each worker: ~50-100MB memory
- CPU: Mostly I/O bound (database, API calls)
- Total for 12 workers: ~1.2GB memory

---

## Security Considerations

1. **Job Class Validation**: Only allow registered job classes to be instantiated
2. **Input Sanitization**: Validate all job payloads before processing
3. **Permission Checks**: Dashboard requires `uri_task_engine` permission
4. **Audit Trail**: All job modifications logged with employee ID
5. **Rate Limiting**: Prevent runaway job dispatching

---

## Testing Strategy

1. **Unit Tests**: Job classes, Scheduler, Worker logic
2. **Integration Tests**: End-to-end job dispatch and processing
3. **Load Tests**: Verify performance under expected volume
4. **Chaos Tests**: Worker crashes, Redis failures, network issues

---

## Open Questions

1. Should we implement job priorities within queues?
2. Do we need job chaining/dependencies for complex workflows?
3. Should failed jobs be automatically requeued after a delay?
4. Do we need separate queues per store for isolation?

---

## References

- [Laravel Queues Documentation](https://laravel.com/docs/12.x/queues)
- [Symfony Messenger Documentation](https://symfony.com/doc/current/messenger.html)
- [Ecotone Message Processing](https://blog.ecotone.tech/message-processing-in-php-symfony-laravel-ecotone/)
- [Redis Data Structures for Queues](https://redis.io/docs/data-types/)
- [Existing SMS Worker System](../systems/redis-worker-system.md)

---

## Appendix A: Existing Systems Documentation

See related documentation:
- `/docs/systems/redis-worker-system.md` - Current SMS Redis worker architecture
- `/userfrosting/workers/README.md` - Current worker documentation
- `/FSRunner/` - FiveStars daemon codebase

---

## Revision History

| Date | Author | Changes |
|------|--------|---------|
| 2025-12-11 | Claude Code | Initial specification created from analysis |