# Solution Design Document

## Validation Checklist

- [x] All required sections are complete
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] All context sources are listed with relevance ratings
- [x] Project commands are discovered from actual project files
- [x] Constraints → Strategy → Design → Implementation path is logical
- [x] Architecture pattern is clearly stated with rationale
- [x] Every component in diagram has directory mapping
- [x] Every interface has specification
- [x] Error handling covers all error types
- [x] Quality requirements are specific and measurable
- [x] Every quality requirement has test coverage
- [x] **All architecture decisions confirmed by user**
- [x] Component names consistent across diagrams
- [x] A developer could implement from this design

---

## Constraints

**CON-1 Technology Stack**
- PHP 8.x runtime environment
- Slim 2.6.2 framework with Twig 1.44.8 templating
- MySQL (multi-store architecture with `kiosk_{typeNum}` pattern)
- Redis-compatible server (Valkey preferred; Redis OSS acceptable) via Predis client
- PCNTL extension required for worker signal handling

**CON-2 Operational Requirements**
- Must run on existing server infrastructure (no new services)
- Single cron entry `* * * * *` invokes one scheduler cycle; scheduler uses a Redis lock to prevent overlapping runs
- Workers must support graceful shutdown (SIGTERM)
- Dashboard must integrate with existing permission system
- No disruption to production during migration

**CON-3 Architecture Constraints**
- PSR-4 autoloading under `BuyerKiosk\TaskEngine` namespace
- Follow existing controller patterns (BaseController, ApiController)
- Use existing design tokens and Bootstrap 5 for UI
- Central database (`kiosk_buykiosk`) for job definitions and execution history
- Store databases for per-store job data access

---

## Implementation Context

### Required Context Sources

**ICO-1 Application Architecture**
```yaml
- doc: docs/patterns/architecture-overview.md
  relevance: HIGH
  why: "Multi-store architecture, database connection patterns"

- doc: docs/patterns/namespace-structure.md
  relevance: HIGH
  why: "Namespace conventions for new feature modules"

- doc: docs/patterns/controller-patterns.md
  relevance: HIGH
  why: "BaseController pattern for page and API controllers"
```

**ICO-2 Database Access Patterns**
```yaml
- file: userfrosting/models/BaseModel.php
  relevance: HIGH
  sections: [dbConnectByName, getStoreFromID]
  why: "Standard database connection helpers"

- file: userfrosting/src/BuyerKiosk/Core/Store.php
  relevance: MEDIUM
  why: "Store entity and configuration access"
```

**ICO-3 Redis Infrastructure**
```yaml
- doc: docs/systems/redis-worker-system.md
  relevance: MEDIUM
  why: "Understanding existing Redis patterns (for reference, not reuse)"

- file: userfrosting/src/BuyerKiosk/Auth/Services/RateLimiter.php
  relevance: LOW
  why: "Example of clean Redis usage with Predis"
```

**ICO-4 Admin Dashboard Patterns**
```yaml
- file: userfrosting/src/BuyerKiosk/Core/Controllers/BaseController.php
  relevance: HIGH
  why: "Base controller for all page controllers"

- file: userfrosting/routes/admin/comeback-cash.php
  relevance: HIGH
  why: "Route registration pattern for admin pages"

- file: userfrosting/templates/themes/default/admin/comeback-cash/index.html
  relevance: MEDIUM
  why: "Admin dashboard template structure"
```

### Implementation Boundaries

- **Must Preserve**: Existing store access patterns, permission system, database connection helpers
- **Can Modify**: Nothing - this is a net-new feature module
- **Must Not Touch**: Existing SMS/FiveStars/QuickBooks implementations (these will be replaced by new job implementations in phase 2)

### External Interfaces

#### System Context Diagram

```mermaid
	graph TB
	    subgraph "BuyerKiosk Application"
	        Admin[Admin Dashboard]
	        API[Task Engine API]
	        Scheduler[Scheduler (Cron)]
	        Workers[Worker Pool]
	    end

    subgraph "Data Stores"
        MySQL[(MySQL Central DB)]
        StoreDB[(Store Databases)]
        Redis[(Redis Queue)]
    end

    subgraph "External Services"
        Twilio[Twilio SMS]
        Vonage[Vonage SMS]
        QuickBooks[QuickBooks API]
        FiveStars[FiveStars API]
        WhenIWork[WhenIWork API]
    end

    DevOps[DevOps Admin] --> Admin
    Developer[Developer] --> API

    Admin --> API
    API --> MySQL

    Scheduler --> MySQL
    Scheduler --> Redis

    Workers --> Redis
    Workers --> MySQL
    Workers --> StoreDB
    Workers --> Twilio
    Workers --> Vonage
    Workers --> QuickBooks
    Workers --> FiveStars
    Workers --> WhenIWork

    Cron[System Cron] -->|"* * * * *"| Scheduler
```

#### Interface Specifications

```yaml
# Inbound Interfaces
inbound:
  - name: "Admin Dashboard"
    type: HTTPS
    format: HTML/REST
    authentication: Session-based (existing)
    permission: uri_task_engine
    data_flow: "Job monitoring, manual dispatch, configuration"

  - name: "CLI Commands"
    type: Shell
    format: PHP CLI
    authentication: None (server access required)
    data_flow: "Scheduler start, worker management, manual job dispatch"

# Outbound Interfaces
outbound:
  - name: "Redis Queue"
    type: TCP
    format: Redis Protocol
    authentication: Optional password
    connection: Predis Client
    data_flow: "Job payloads, worker heartbeats, queue operations"
    criticality: HIGH

  - name: "MySQL Central"
    type: TCP
    format: MySQL Protocol
    connection: PDO
    data_flow: "Job definitions, execution history, worker registration"
    criticality: HIGH

  - name: "External APIs"
    type: HTTPS
    format: REST/JSON
    data_flow: "Job-specific integrations (SMS, sync, etc.)"
    criticality: MEDIUM
```

### Project Commands

```bash
# Environment Setup
cd userfrosting && composer install

# Testing Commands
./test.sh                           # Run all tests
./test.sh --testsuite unit          # Run unit tests only
./test.sh --testsuite integration   # Run integration tests only

# Code Quality
cd userfrosting && ./vendor/bin/phpstan analyse

# Database Migration
php userfrosting/conductor run

# Task Engine Commands (NEW)
php userfrosting/bin/task scheduler:run     # Run one scheduler cycle (cron-invoked)
php userfrosting/bin/task worker:start      # Start single worker
php userfrosting/bin/task worker:manager    # Start worker manager
php userfrosting/bin/task job:dispatch <job> [--store=<typeNum>]  # Manual dispatch
php userfrosting/bin/task job:list          # List all job definitions
php userfrosting/bin/task queue:status      # Show queue depths
```

---

## Solution Strategy

### Architecture Pattern

**Layered Architecture with Command Pattern**

The Task Engine uses a layered architecture with clear separation of concerns:

1. **Presentation Layer**: Admin dashboard (Twig templates + REST API)
2. **Application Layer**: Controllers, Commands, Dispatcher
3. **Domain Layer**: Job definitions, Scheduler, Worker logic
4. **Infrastructure Layer**: Redis queue, MySQL persistence, External APIs

Jobs follow the **Command Pattern** where each job is a self-contained unit of work with:
- `handle()` method containing business logic
- Configuration (queue, timeout, retries)
- Optional lifecycle hooks (beforeHandle, afterHandle, failed)

### Integration Approach

The Task Engine integrates as a new feature module under `BuyerKiosk\TaskEngine`:

- **Routes**: `/admin/:typeNum/tasks/` for dashboard, `/api/:typeNum/tasks/` for API
- **Database**: New tables in `kiosk_buykiosk` central database
- **Permissions**: New `uri_task_engine` permission
- **No dependencies** on existing job systems - completely independent

### Key Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Queue Backend | Redis-compatible (Valkey preferred) pending + processing lists | Already deployed; uses BRPOPLPUSH reserve/ack for at-least-once delivery |
| Job Storage | MySQL + Redis-compatible | MySQL for definitions/history (durability), Redis-compatible for queue (speed) |
| Worker Model | Process-per-worker | Isolation, simple restart, PCNTL signal handling |
| Job Scope | Global + Per-Store | Flexibility to run jobs once or per-store with single scheduler |
| Admin Scope | Store-scoped | Consistent with other admin features, permission isolation |

---

## Building Block View

### Components

```mermaid
graph TB
    subgraph "Presentation Layer"
        Dashboard[TaskDashboardController]
        API[TaskApiController]
    end

    subgraph "Application Layer"
        Dispatcher[JobDispatcher]
        CLI[TaskCommand]
    end

    subgraph "Domain Layer"
        Scheduler[Scheduler]
        WorkerManager[WorkerManager]
        Worker[Worker]
        JobRegistry[JobRegistry]
    end

    subgraph "Infrastructure Layer"
        RedisQueue[RedisQueueAdapter]
        JobRepo[JobDefinitionRepository]
        ExecRepo[ExecutionRepository]
        WorkerRepo[WorkerRepository]
    end

    subgraph "Jobs"
        BaseJob[BaseJob]
        EventPhaseJob[EventPhaseJob]
        SmsQueueJob[SmsQueueJob]
        EmployeeSyncJob[EmployeeSyncJob]
        QuickBooksSyncJob[QuickBooksSyncJob]
    end

    Dashboard --> API
    API --> Dispatcher
    CLI --> Dispatcher
    CLI --> Scheduler
    CLI --> WorkerManager

    Scheduler --> JobRegistry
    Scheduler --> Dispatcher
    WorkerManager --> Worker
    Worker --> JobRegistry
    Worker --> RedisQueue

    Dispatcher --> RedisQueue
    Dispatcher --> ExecRepo

    JobRegistry --> JobRepo
    Worker --> ExecRepo
    WorkerManager --> WorkerRepo

    BaseJob --> EventPhaseJob
    BaseJob --> SmsQueueJob
    BaseJob --> EmployeeSyncJob
    BaseJob --> QuickBooksSyncJob
```

### Directory Map

```
userfrosting/src/BuyerKiosk/TaskEngine/
├── Commands/                           # NEW: CLI commands
│   └── TaskCommand.php                 # scheduler:run, worker:start, job:dispatch
│
├── Controllers/                        # NEW: HTTP controllers
│   ├── TaskDashboardController.php     # Admin page rendering
│   └── TaskApiController.php           # REST API endpoints
│
├── Domain/                             # NEW: Core domain logic
│   ├── Job/
│   │   ├── BaseJob.php                 # Abstract job base class
│   │   ├── JobInterface.php            # Job contract
│   │   ├── JobContext.php              # Execution context (store, config)
│   │   └── JobResult.php               # Execution result value object
│   │
│   ├── Scheduler/
│   │   ├── Scheduler.php               # Cron-invoked scheduler cycle
│   │   ├── CronExpression.php          # Cron expression parser
│   │   └── ScheduleEvaluator.php       # Determines which jobs are due
│   │
│   └── Worker/
│       ├── Worker.php                  # Single worker process
│       ├── WorkerManager.php           # Manages worker pool
│       └── WorkerStatus.php            # Worker state enum
│
├── Infrastructure/                     # NEW: External adapters
│   ├── Queue/
│   │   ├── QueueInterface.php          # Queue abstraction
│   │   └── RedisQueueAdapter.php       # Redis implementation
│   │
│   ├── Persistence/
│   │   ├── JobDefinitionRepository.php # Job definitions CRUD
│   │   ├── ExecutionRepository.php     # Execution history CRUD
│   │   └── WorkerRepository.php        # Worker registration
│   │
│   └── Notification/
│       ├── FailureNotifier.php         # Email notifications on failure
│       └── NotificationRateLimiter.php # Prevents notification floods (max 10/hour per job)
│
├── Registry/                           # NEW: Job registration
│   └── JobRegistry.php                 # Maps job names to classes
│
└── Jobs/                               # NEW: Concrete job implementations
    ├── EventPhaseJob.php               # Event phase transitions
    ├── SmsQueueJob.php                 # SMS queue processing
    ├── EmployeeSyncJob.php             # Employee sync from providers
    ├── QuickBooksSyncJob.php           # QuickBooks sync
    ├── FiveStarsSyncJob.php            # FiveStars point sync
    └── AggregateStatsJob.php           # Daily stats aggregation

userfrosting/routes/task-engine/        # NEW: Route definitions
├── pages.php                           # Store-scoped dashboard routes (/admin/:typeNum/tasks/)
├── global-pages.php                    # Global dashboard routes (/admin/tasks/) - super-admin
└── api.php                             # API routes (both store-scoped and global)

userfrosting/templates/themes/default/admin/task-engine/  # NEW: Templates
├── dashboard.html                      # Main dashboard
├── jobs.html                           # Job definitions list
├── executions.html                     # Execution history
├── workers.html                        # Worker status
└── partials/
    ├── stats-cards.html                # Summary metrics
    ├── job-row.html                    # Job list row
    └── execution-detail.html           # Execution detail modal

public_html/css/admin/modules/
└── task-engine.css                     # NEW: Task engine styles

userfrosting/bin/
└── task                                # NEW: CLI entry point
```

### Interface Specifications

#### Data Storage Changes

```yaml
# Central Database: kiosk_buykiosk
Table: task_job_definitions (NEW)
  id: INT AUTO_INCREMENT PRIMARY KEY
  name: VARCHAR(100) UNIQUE NOT NULL        # e.g., "event-phase-processor"
  displayName: VARCHAR(200) NOT NULL        # e.g., "Event Phase Processor"
  className: VARCHAR(255) NOT NULL          # Informational FQCN (validated against JobRegistry)
  schedule: VARCHAR(100) NULL               # Cron expression, NULL = manual only
  queue: VARCHAR(50) DEFAULT 'default'      # Queue name
  scope: ENUM('global','per_store') DEFAULT 'global'
  timeout: INT DEFAULT 300                  # Seconds
  maxRetries: INT DEFAULT 3
  retryBackoff: INT DEFAULT 60              # Base seconds for exponential backoff
  config: JSON NULL                         # Job-specific configuration
  isEnabled: TINYINT(1) DEFAULT 1
  notifyOnFailure: TINYINT(1) DEFAULT 1
  notifyEmails: VARCHAR(500) NULL           # Comma-separated emails
  created_at: DATETIME DEFAULT CURRENT_TIMESTAMP
  updated_at: DATETIME ON UPDATE CURRENT_TIMESTAMP
  INDEX idx_schedule (isEnabled, schedule)
  INDEX idx_queue (queue)

Table: task_executions (NEW)
  id: BIGINT AUTO_INCREMENT PRIMARY KEY
  jobDefinitionId: INT NOT NULL             # FK to task_job_definitions
  typeNum: VARCHAR(10) NULL                 # Store context (NULL for global)
  status: ENUM('pending','running','completed','failed','cancelled') DEFAULT 'pending'
  triggerType: ENUM('scheduled','manual','retry') DEFAULT 'scheduled'
  triggeredBy: INT NULL                     # User ID if manual
  workerId: VARCHAR(50) NULL                # Worker that processed
  attempt: INT DEFAULT 1
  idempotencyKey: VARCHAR(255) NULL         # Stable logical-run key for dedupe/at-least-once safety
  payload: JSON NULL                        # Input data
  result: JSON NULL                         # Output/error data
  progress: TINYINT DEFAULT 0               # 0-100
  errorMessage: TEXT NULL
  errorTrace: TEXT NULL
  scheduled_fire_at: DATETIME NULL          # Scheduled fire time (UTC, minute precision) for scheduled jobs
  queued_at: DATETIME DEFAULT CURRENT_TIMESTAMP
  started_at: DATETIME NULL
  completed_at: DATETIME NULL
  durationMs: INT NULL
  UNIQUE idx_idempotency (jobDefinitionId, idempotencyKey)
  INDEX idx_scheduled_fire (jobDefinitionId, scheduled_fire_at)
  INDEX idx_job_status (jobDefinitionId, status)
  INDEX idx_store (typeNum, status)
  INDEX idx_queued (status, queued_at)
  INDEX idx_worker (workerId, status)
  FOREIGN KEY (jobDefinitionId) REFERENCES task_job_definitions(id)

Table: task_execution_logs (NEW)
  id: BIGINT AUTO_INCREMENT PRIMARY KEY
  executionId: BIGINT NOT NULL              # FK to task_executions
  level: ENUM('debug','info','warning','error') DEFAULT 'info'
  message: TEXT NOT NULL
  context: JSON NULL
  logged_at: DATETIME(3) DEFAULT CURRENT_TIMESTAMP(3)
  INDEX idx_execution (executionId)
  FOREIGN KEY (executionId) REFERENCES task_executions(id) ON DELETE CASCADE

Table: task_workers (NEW)
  id: VARCHAR(50) PRIMARY KEY               # UUID
  hostname: VARCHAR(100) NOT NULL
  pid: INT NOT NULL
  status: ENUM('idle','busy','stopped') DEFAULT 'idle'
  currentExecutionId: BIGINT NULL
  jobsProcessed: INT DEFAULT 0
  jobsFailed: INT DEFAULT 0
  started_at: DATETIME DEFAULT CURRENT_TIMESTAMP
  last_heartbeat: DATETIME DEFAULT CURRENT_TIMESTAMP
  INDEX idx_status (status, last_heartbeat)
```

#### Internal API Changes

```yaml
# Admin Dashboard API
Endpoint: List Job Definitions
  Method: GET
  Path: /api/:typeNum/tasks/jobs
  Permission: uri_task_engine
  Response:
    success: boolean
    jobs: array[JobDefinition]

Endpoint: Get Job Definition
  Method: GET
  Path: /api/:typeNum/tasks/jobs/:jobId
  Permission: uri_task_engine
  Response:
    success: boolean
    job: JobDefinition

Endpoint: Update Job Definition
  Method: PUT
  Path: /api/:typeNum/tasks/jobs/:jobId
  Permission: uri_task_engine
  Request:
    isEnabled: boolean (optional)
    schedule: string (optional)
    config: object (optional)
    notifyEmails: string (optional)
  Response:
    success: boolean
    job: JobDefinition

Endpoint: Dispatch Job Manually
  Method: POST
  Path: /api/:typeNum/tasks/jobs/:jobId/dispatch
  Permission: uri_task_engine
  Request:
    store: string (optional, for per_store jobs)
    payload: object (optional)
    idempotencyKey: string (optional)
  Response:
    success: boolean
    execution: Execution

Endpoint: List Executions
  Method: GET
  Path: /api/:typeNum/tasks/executions
  Permission: uri_task_engine
  Query:
    jobId: int (optional)
    status: string (optional)
    limit: int (default 50)
    offset: int (default 0)
  Response:
    success: boolean
    executions: array[Execution]
    total: int

Endpoint: Get Execution Detail
  Method: GET
  Path: /api/:typeNum/tasks/executions/:executionId
  Permission: uri_task_engine
  Response:
    success: boolean
    execution: Execution
    logs: array[LogEntry]

Endpoint: Retry Execution
  Method: POST
  Path: /api/:typeNum/tasks/executions/:executionId/retry
  Permission: uri_task_engine
  Response:
    success: boolean
    execution: Execution (requeued)

Endpoint: Abort Execution
  Method: POST
  Path: /api/:typeNum/tasks/executions/:executionId/abort
  Permission: uri_task_engine
  Response:
    success: boolean

Endpoint: Get Dashboard Stats
  Method: GET
  Path: /api/:typeNum/tasks/stats
  Permission: uri_task_engine
  Response:
    success: boolean
    stats:
      activeWorkers: int
      pendingJobs: int
      runningJobs: int
      failedJobs24h: int
      completedJobs24h: int
      queueDepths: object[queue -> count]

Endpoint: List Workers
  Method: GET
  Path: /api/:typeNum/tasks/workers
  Permission: uri_task_engine
  Response:
    success: boolean
    workers: array[Worker]

# Global Dashboard API (Super-Admin Only)
Endpoint: Global Dashboard Stats
  Method: GET
  Path: /api/tasks/stats
  Permission: uri_task_engine_global
  Response:
    success: boolean
    stats:
      activeWorkers: int
      pendingJobs: int
      runningJobs: int
      failedJobs24h: int
      completedJobs24h: int
      queueDepths: object[queue -> count]
      jobsByStore: array[{typeNum, pending, running, failed}]

Endpoint: Global Executions List
  Method: GET
  Path: /api/tasks/executions
  Permission: uri_task_engine_global
  Query:
    jobId: int (optional)
    typeNum: string (optional, filter by store)
    status: string (optional)
    limit: int (default 100)
    offset: int (default 0)
  Response:
    success: boolean
    executions: array[Execution]
    total: int

Endpoint: Global Workers List
  Method: GET
  Path: /api/tasks/workers
  Permission: uri_task_engine_global
  Response:
    success: boolean
    workers: array[Worker]
```

#### Application Data Models

```pseudocode
ENTITY: JobDefinition
  FIELDS:
    id: int
    name: string
    displayName: string
    className: string
    schedule: string|null
    queue: string
    scope: 'global'|'per_store'
    timeout: int
    maxRetries: int
    retryBackoff: int
    config: array
    isEnabled: bool
    notifyOnFailure: bool
    notifyEmails: string|null
    created_at: DateTime
    updated_at: DateTime

  BEHAVIORS:
    isDue(lastRun: DateTime): bool
    getNextRunTime(): DateTime|null
    createJob(): JobInterface

ENTITY: Execution
  FIELDS:
    id: int
    jobDefinitionId: int
    typeNum: string|null
    status: ExecutionStatus
    triggerType: 'scheduled'|'manual'|'retry'
    triggeredBy: int|null
    workerId: string|null
    attempt: int
    idempotencyKey: string|null
    payload: array
    result: array|null
    progress: int
    errorMessage: string|null
    errorTrace: string|null
    scheduled_fire_at: DateTime|null
    queued_at: DateTime
    started_at: DateTime|null
    completed_at: DateTime|null
    durationMs: int|null

  BEHAVIORS:
    markStarted(workerId: string): void
    markCompleted(result: array): void
    markFailed(error: Exception): void
    updateProgress(percent: int): void
    canRetry(): bool

ENTITY: Worker
  FIELDS:
    id: string
    hostname: string
    pid: int
    status: WorkerStatus
    currentExecutionId: int|null
    jobsProcessed: int
    jobsFailed: int
    started_at: DateTime
    last_heartbeat: DateTime

  BEHAVIORS:
    heartbeat(): void
    isHealthy(): bool
    markBusy(executionId: int): void
    markIdle(): void
    markStopped(): void
```

#### Integration Points

```yaml
# Internal Components
- from: Scheduler
  to: JobDispatcher
  protocol: PHP Method Call
  data_flow: "Dispatches due jobs to queue"

- from: Worker
  to: JobRegistry
  protocol: PHP Method Call
  data_flow: "Resolves job class from name"

- from: Worker
  to: Store Databases
  protocol: PDO
  data_flow: "Per-store jobs access store data"

# External Services (Job-specific)
- from: SmsQueueJob
  to: Twilio/Vonage
  protocol: HTTPS
  data_flow: "Send SMS messages"

- from: QuickBooksSyncJob
  to: QuickBooks API
  protocol: HTTPS
  data_flow: "Sync financial data"

- from: FiveStarsSyncJob
  to: FiveStars API
  protocol: HTTPS
  data_flow: "Sync loyalty points"

- from: EmployeeSyncJob
  to: WhenIWork/Homebase API
  protocol: HTTPS
  data_flow: "Sync employee data"
```

### Implementation Examples

#### Example: BaseJob Abstract Class

**Why this example**: Shows the contract all jobs must follow and the lifecycle hooks available.

```php
<?php
namespace BuyerKiosk\TaskEngine\Domain\Job;

abstract class BaseJob implements JobInterface
{
    protected JobContext $context;
    protected ExecutionLogger $logger;

    /**
     * Main job execution logic - must be implemented
     */
    abstract public function handle(): JobResult;

    /**
     * Called before handle() - setup logic
     */
    public function beforeHandle(): void
    {
        // Override in subclass if needed
    }

    /**
     * Called after successful handle() - cleanup logic
     */
    public function afterHandle(JobResult $result): void
    {
        // Override in subclass if needed
    }

    /**
     * Called when job fails after all retries
     */
    public function failed(\Exception $exception): void
    {
        // Override in subclass for custom failure handling
    }

    /**
     * Report progress (0-100); optional message is logged
     */
    protected function progress(int $percent, string $message = ''): void
    {
        $this->context->updateProgress($percent, $message);
    }

    /**
     * Log a message
     */
    protected function log(string $level, string $message, array $context = []): void
    {
        $this->logger->log($level, $message, $context);
    }

    /**
     * Get store database connection (for per-store jobs)
     */
    protected function getStoreDb(): ?\PDO
    {
        $store = $this->getStore();
        if (!$store) {
            return null;
        }
        return dbConnectByName($store->getDbName());
    }

    /**
     * Get store object (for per-store jobs)
     */
    protected function getStore(): ?\BuyerKiosk\Core\Store
    {
        if (!$this->context->hasStore()) {
            return null;
        }
        return $this->context->getStore();
    }
}
```

#### Example: Concrete Job Implementation

**Why this example**: Shows how to implement a per-store job with progress reporting.

```php
<?php
namespace BuyerKiosk\TaskEngine\Jobs;

use BuyerKiosk\TaskEngine\Domain\Job\BaseJob;
use BuyerKiosk\TaskEngine\Domain\Job\JobResult;
use BuyerKiosk\EventManagement\Services\EventPhaseProcessor;

class EventPhaseJob extends BaseJob
{
    public static string $name = 'event-phase-processor';
    public static string $displayName = 'Event Phase Processor';
    public static string $queue = 'default';
    public static string $scope = 'per_store';
    public static int $timeout = 120;

    public function handle(): JobResult
    {
        $store = $this->getStore();
        $db = $this->getStoreDb();

        if (!$store || !$db) {
            return JobResult::failure('Store not available');
        }

        $this->log('info', 'Processing event phases', ['store' => $store->getTypeNum()]);
        $this->progress(10, 'Loading events');

        // Create processor with store context
        $integrationService = new \BuyerKiosk\EventManagement\Services\IntegrationService($db);
        $processor = new EventPhaseProcessor($db, $integrationService, null);

        $this->progress(30, 'Evaluating phase transitions');

        try {
            $results = $processor->processStore();

            $this->progress(100, 'Complete');

            return JobResult::success([
                'eventsProcessed' => count($results['processed']),
                'transitioned' => count($results['transitioned']),
                'integrationsActivated' => $results['integrations_activated']
            ]);
        } catch (\Exception $e) {
            $this->log('error', 'Phase processing failed', [
                'error' => $e->getMessage()
            ]);
            throw $e;
        }
    }

    public function failed(\Exception $exception): void
    {
        // Log to store-specific error tracking
        error_log(sprintf(
            'EventPhaseJob failed for %s: %s',
            $this->context->getTypeNum(),
            $exception->getMessage()
        ));
    }
}
```

#### Example: Worker Process Loop

**Why this example**: Shows the worker's main execution loop with signal handling.

```php
<?php
namespace BuyerKiosk\TaskEngine\Domain\Worker;

class Worker
{
    private bool $shouldRun = true;
    private RedisQueueAdapter $queue;
    private JobRegistry $registry;
    private ExecutionRepository $executions;
    private string $workerId;

    public function run(): void
    {
        $this->registerSignalHandlers();
        $this->register();

        while ($this->shouldRun) {
            $this->heartbeat();

            // Reserve work atomically (at-least-once)
            $reservation = $this->queue->reserve(['high', 'default'], 5);

            if ($reservation === null) {
                continue; // No job available, loop again
            }

            $queueName = $reservation['queue'];
            $payload = $reservation['payload'];

            $this->process($queueName, $payload);
        }

        $this->shutdown();
    }

    private function process(string $queueName, array $payload): void
    {
        $execution = $this->executions->find($payload['executionId']);

        if (!$execution || $execution->status !== 'pending') {
            $this->queue->ack($queueName, $payload);
            return; // Already processed or cancelled
        }

        $execution->markStarted($this->workerId);
        $this->markBusy($execution->id);

        try {
            $job = $this->registry->resolve($execution->jobDefinition->name);
            $job->setContext(new JobContext($execution));

            $job->beforeHandle();
            $result = $job->handle();
            $job->afterHandle($result);

            $execution->markCompleted($result->toArray());

        } catch (\Exception $e) {
            $execution->markFailed($e);

            if ($execution->canRetry()) {
                $this->scheduleRetry($execution);
            } else {
                $job->failed($e);
                $this->notifyFailure($execution, $e);
            }
        } finally {
            $this->queue->ack($queueName, $payload);
            $this->markIdle();
        }
    }

    private function registerSignalHandlers(): void
    {
        pcntl_signal(SIGTERM, fn() => $this->shouldRun = false);
        pcntl_signal(SIGINT, fn() => $this->shouldRun = false);
    }
}
```

---

## Runtime View

### Primary Flow: Scheduled Job Execution

1. **Cron triggers scheduler** every minute (`* * * * *`)
2. **Scheduler acquires Redis lock** and exits if another cycle is running
3. **Scheduler promotes due delayed retries** from Redis ZSETs into list queues
4. **Scheduler evaluates** all enabled job definitions against their cron expressions
5. **For due jobs**, scheduler creates an execution keyed by `idempotencyKey` and dispatches only if newly created
6. **Per-store jobs** create one execution per active store
7. **Workers reserve jobs** from Redis pending lists using BRPOPLPUSH into processing lists (at-least-once)
8. **Worker claims job**, updates execution status to "running"
9. **Worker instantiates** job class and calls lifecycle methods
10. **Job executes** business logic, reports progress
11. **Worker updates** execution with result or error
12. **On failure**, retry scheduled with exponential backoff if attempts remaining

**Scheduler locking**: Each cycle acquires a Redis key `task-engine:scheduler-lock` via `SET <key> <uuid> NX EX 60`. If not acquired, the scheduler exits immediately. On shutdown it releases the lock only if the value still matches its UUID.

**Catch-up behavior**: Scheduler compares the last successful execution time to the current time and backfills missed cron fire times within the catch-up window, creating executions for each missed interval.

```mermaid
sequenceDiagram
    participant Cron
    participant Scheduler
    participant MySQL
    participant Redis
    participant Worker
    participant Job
    participant External

    Cron->>Scheduler: * * * * * trigger
    Scheduler->>MySQL: Get enabled job definitions
    MySQL-->>Scheduler: JobDefinitions[]

    loop For each due job
        Scheduler->>MySQL: Create execution (pending)
        Scheduler->>Redis: RPUSH queue payload
    end

    Worker->>Redis: BRPOPLPUSH pending->processing (reserve)
    Redis-->>Worker: ExecutionId payload

    Worker->>MySQL: Update execution (running)
    Worker->>Job: instantiate & handle()

    Job->>MySQL: Log progress
    Job->>External: API calls (if needed)
    External-->>Job: Response

    Job-->>Worker: JobResult
    Worker->>MySQL: Update execution (completed)
    Worker->>Redis: BRPOPLPUSH next job (reserve)
```

### Secondary Flow: Manual Job Dispatch

1. **Admin clicks** "Run Now" in dashboard
2. **API validates** permission and creates execution with triggerType='manual'
3. **API dispatches** to Redis queue immediately
4. **Worker processes** same as scheduled flow

### Error Handling

| Error Type | Handling Strategy |
|------------|-------------------|
| Job throws exception | Catch, log, mark failed, retry if attempts remaining |
| Worker crashes | Worker manager detects via heartbeat, respawns worker |
| Redis unavailable | Workers retry connection with backoff, scheduler logs warning |
| MySQL unavailable | Critical - workers pause, scheduler exits with error |
| Job timeout exceeded | Worker kills job, marks failed, retries |
| Orphan detection | Scheduler checks for "running" jobs older than 2x timeout, marks failed |

### Complex Logic: Retry with Exponential Backoff

```
ALGORITHM: Schedule Retry
INPUT: execution, baseBackoff, maxRetries

1. IF execution.attempt >= maxRetries:
   - Mark execution permanently failed
   - Trigger failure notification
   - RETURN

2. CALCULATE delay:
   - delay = baseBackoff * (2 ^ (execution.attempt - 1))
   - Add jitter: delay += random(0, delay * 0.1)
   - Cap at 1 hour maximum

3. UPDATE existing execution for retry:
   - Set attempt = attempt + 1
   - Set triggerType = 'retry'
   - Set status = 'pending'
   - Clear workerId, started_at, completed_at, errorMessage, errorTrace, progress
   - Keep idempotencyKey unchanged
   - Set queued_at = NOW + delay

4. DISPATCH to Redis delayed queue (sorted set):
   - ZADD delayed:<queue> with score = UNIX_TIMESTAMP(queued_at)
   - Payload includes executionId and target queue
   - Each scheduler cycle promotes due items:
     * ZRANGEBYSCORE delayed:<queue> -inf now
     * For each item: RPUSH <queue> payload, then ZREM from delayed set
```

### Complex Logic: Orphan Job Detection

```
ALGORITHM: Detect Orphan Jobs
RUNS: Every scheduler cycle (once per minute)

1. QUERY all executions WHERE:
   - status = 'running'
   - started_at < NOW - (2 * job.timeout)

2. FOR EACH orphaned execution:
   a. Check if worker is still alive (heartbeat)
   b. IF worker dead OR heartbeat stale:
      - Mark execution as 'failed'
      - Set errorMessage = 'Job orphaned: worker died or timed out'
      - Set errorTrace = 'Detected by scheduler orphan detection'
      - Log tracking event: job_orphaned

   c. IF execution.canRetry():
      - Schedule retry (call Schedule Retry algorithm)
   d. ELSE:
      - Trigger failure notification
      - Log as permanently failed

3. ALERT if orphan_count > threshold (configurable, default 5)
```

---

## Deployment View

### Environment

- **Runtime**: PHP 8.x CLI for scheduler/workers, PHP-FPM for web interface
- **Process Management**: Supervisor or systemd for worker manager and workers (long-running)
- **Cron**: Single entry `* * * * * php /path/to/bin/task scheduler:run` (one cycle, lock-protected)

### Configuration

Required environment variables:
```env
# Redis (existing)
REDIS_URL=redis://localhost:6379

# Task Engine specific
TASK_ENGINE_WORKER_COUNT=4          # Default worker count
TASK_ENGINE_HEARTBEAT_INTERVAL=10   # Seconds between heartbeats
TASK_ENGINE_STALE_THRESHOLD=60      # Seconds before worker considered stale
```

### Dependencies

- **Redis-compatible server**: Valkey preferred; must be running and accessible
- **MySQL**: Central database must be accessible
- **PCNTL Extension**: Required for worker signal handling
- **Predis**: Already installed via Composer

### Performance Targets

- **Scheduler execution**: < 5 seconds per run
- **Job dispatch latency**: < 100ms from queue to worker pickup
- **Dashboard response**: < 500ms for stats API
- **Worker throughput**: ~100-200 jobs/minute per worker (IO-bound)

---

## Cross-Cutting Concepts

### Pattern Documentation

```yaml
# Existing patterns used
- pattern: @docs/patterns/controller-patterns.md
  relevance: HIGH
  why: "BaseController inheritance for page/API controllers"

- pattern: @docs/patterns/namespace-structure.md
  relevance: HIGH
  why: "Feature namespace organization under BuyerKiosk"

# New patterns created
- pattern: @docs/patterns/task-engine-job-pattern.md (NEW)
  relevance: HIGH
  why: "Standard job implementation pattern"

- pattern: @docs/patterns/task-engine-cli-pattern.md (NEW)
  relevance: MEDIUM
  why: "CLI command structure for task engine"
```

### System-Wide Patterns

**Security**:
- Permission `uri_task_engine` required for dashboard access
- Store-scoped: users only see jobs for stores they have access to
- CSRF protection on all mutation endpoints
- Workers only instantiate jobs via JobRegistry whitelist; DB job names must exist in the registry and class names are not executed directly
- No sensitive data in job payloads (reference IDs only)

**Error Handling**:
- Jobs: catch all exceptions, log with context, retry with backoff
- Workers: never crash main loop, respawn on failure
- Scheduler: log errors but continue processing other jobs

**Queue Semantics (at-least-once)**:
- Each queue has a pending list `queue:<name>` and a processing list `processing:<name>`.
- Workers call `BRPOPLPUSH queue:<name> processing:<name>` to reserve work atomically.
- On successful completion, workers ACK by removing the payload from `processing:<name>`.
- Scheduler requeues stale processing items (visibility timeout = `2 * timeout` or configurable) back into `queue:<name>` and marks the execution as orphaned if needed.

**Idempotency & Deduplication**:
- Each execution has an optional `idempotencyKey` representing a logical run; scheduler always sets it for scheduled jobs.
- MySQL enforces uniqueness on `(jobDefinitionId, idempotencyKey)`; duplicate dispatch attempts return/keep the existing execution.
- Default scheduled key format: `<job-name>:<scheduled_fire_at>[:<typeNum>]` in UTC, minute precision.
- Jobs that call external systems must treat side effects as idempotent using this key or stable external IDs.

**Performance**:
- Redis BRPOPLPUSH for efficient reserve/ack polling (no busy-wait)
- Batch processing in scheduler (evaluate all jobs, then dispatch)
- Pagination for dashboard APIs
- Execution log retention: 7 days (configurable)

**Logging/Auditing**:
- Per-execution log table for job-specific logs (7-day retention)
- Execution records retained for 30 days (per PRD F4)
- Worker events to system log
- Dashboard access logged via existing audit system
- Tracking events: job_dispatched, job_completed, job_failed, job_retried, job_aborted, job_progress, job_orphaned, worker_started, worker_stopped

### Component Structure Pattern

```pseudocode
COMPONENT: TaskApiController
  INITIALIZE:
    - Inherit BaseController
    - Initialize repositories

  HANDLE:
    - Check authentication (session)
    - Check store access (checkStoreGroup)
    - Check permission (checkAccess)
    - Validate input
    - Delegate to service/repository

  RESPOND:
    - JSON response with success/error pattern
    - Include CSRF token refresh
```

### Test Pattern

```pseudocode
TEST_SCENARIO: "Job executes successfully and records result"
  SETUP:
    - Create job definition in database
    - Create mock dependencies (Redis, external APIs)

  EXECUTE:
    - Instantiate job with mock context
    - Call handle()

  VERIFY:
    - JobResult indicates success
    - Expected side effects occurred
    - Logs recorded correctly
    - No unexpected exceptions
```

---

## Architecture Decisions

- [x] ADR-1 **Queue Backend**: Redis-compatible (Valkey preferred) with BRPOPLPUSH reserve/ack lists
  - Rationale: Already deployed; native list ops allow atomic reserve + at-least-once delivery without polling
  - Trade-offs: Single point of failure (acceptable, Redis already critical)
  - User confirmed: _Via PRD constraint_

- [x] ADR-2 **Job Storage**: MySQL for definitions/history, Redis for queue only
  - Rationale: MySQL provides durability, query capability; Redis provides speed for queue ops
  - Trade-offs: Dual storage complexity, but clear separation of concerns
  - User confirmed: _Via PRD constraint_

- [x] ADR-3 **Worker Process Model**: One process per worker
  - Rationale: Process isolation, simple crash recovery, clear resource limits
  - Trade-offs: Higher memory than threading (~50-100MB per worker), but PHP threading is problematic
  - User confirmed: _Confirmed_

- [x] ADR-4 **Job Scope Model**: Global + Per-Store with single scheduler
  - Rationale: Flexibility, single point of scheduling logic
  - Trade-offs: Per-store jobs create N executions (N = store count)
  - User confirmed: _Confirmed_

- [x] ADR-5 **Admin Dashboard Scope**: Store-scoped + Global for super-admins
  - Rationale: Store-scoped consistent with other features; global dashboard for operational visibility
  - Routes: `/admin/:typeNum/tasks/` (store-scoped), `/admin/tasks/` (global, super-admin only)
  - Trade-offs: Two dashboard variants to maintain
  - User confirmed: _Confirmed_

---

## Quality Requirements

**Performance**:
- Scheduler completes cycle in < 5 seconds
- Worker picks up job from queue in < 100ms average
- Dashboard API responses < 500ms
- Support 10+ concurrent workers without contention

**Reliability**:
- 99.9% job execution success rate (excluding business errors)
- Scheduler performs bounded catch-up for missed schedules (default 10 minutes) and logs/alerts beyond that window
- Automatic worker recovery within 30 seconds of crash
- Orphan job detection within 2x timeout period

**Usability**:
- Dashboard shows real-time status (refresh every 30 seconds)
- Failed jobs clearly visible with error details
- One-click retry for failed jobs
- Manual dispatch with optional payload override

**Maintainability**:
- New jobs require only: class file + database registration
- Job configuration changes via database (no deploys)
- Execution history searchable by job, store, status

---

## Risks and Technical Debt

### Known Technical Issues

- **PCNTL availability**: Workers require PCNTL extension; must verify on all environments
- **Redis memory**: Queue could grow if workers stop; need monitoring and alerting
- **Long-running jobs**: Jobs exceeding timeout may leave resources in inconsistent state

### Implementation Gotchas

- **Timezone handling**: All timestamps in UTC, cron expressions in UTC
- **Database connections**: Jobs must get fresh connections, not reuse worker's
- **Memory leaks**: Workers should restart after N jobs to prevent memory growth
- **Signal handling**: Must call `pcntl_signal_dispatch()` in job loops

### Technical Debt

- **Legacy job migration**: Existing SMS/FiveStars code will coexist until migrated
- **No distributed scheduler**: Single scheduler is SPOF (acceptable for v1)
- **Limited job priorities**: Only coarse queue-level priority (high vs default); no per-job weights within a queue (Could Have feature)

---

## Test Specifications

### Critical Test Scenarios

**Scenario 1: Scheduled Job Execution**
```gherkin
Given: Job definition with schedule "*/5 * * * *"
And: Current time is 10:05
And: Last execution was at 10:00
When: Scheduler evaluates jobs
Then: New execution created with status "pending"
And: Job payload pushed to Redis queue
And: Execution appears in dashboard
```

**Scenario 2: Worker Processes Job**
```gherkin
Given: Execution in "pending" status in Redis queue
And: Worker is idle
When: Worker polls queue
Then: Worker claims execution
And: Execution status changes to "running"
And: Job handle() method executes
And: Execution status changes to "completed"
And: Result stored in execution record
```

**Scenario 3: Job Failure with Retry**
```gherkin
Given: Job with maxRetries=3
And: Current attempt is 1
When: Job throws exception during handle()
Then: Execution marked "failed"
And: Execution attempt incremented to 2 and requeued
And: Retry scheduled with exponential backoff
And: Error logged with stack trace
```

**Scenario 4: Worker Crash Recovery**
```gherkin
Given: Worker processing a job
When: Worker process crashes unexpectedly
Then: Worker manager detects missing heartbeat
And: New worker process spawned
And: Orphaned execution detected by scheduler
And: Execution marked failed and retried
```

**Scenario 5: Per-Store Job Dispatch**
```gherkin
Given: Job with scope="per_store"
And: 50 active stores in system
When: Scheduler dispatches job
Then: 50 execution records created
And: Each execution has unique typeNum
And: All 50 pushed to queue
```

### Test Coverage Requirements

- **Unit Tests**: BaseJob, Scheduler, Worker, CronExpression, all Repository classes
- **Integration Tests**: End-to-end job dispatch and execution with real Redis
- **Controller Tests**: All API endpoints with auth/permission validation
- **Job Tests**: Each concrete job class with mocked external dependencies

---

## Glossary

### Domain Terms

| Term | Definition | Context |
|------|------------|---------|
| Job | A unit of background work that can be scheduled or manually triggered | Core concept - the "what" being executed |
| Execution | A single run of a job, tracking status and result | Audit trail - "when" and "how" a job ran |
| Queue | A Redis list holding pending job payloads | Infrastructure - "where" jobs wait |
| Worker | A long-running process that pulls and executes jobs | Infrastructure - "who" runs jobs |
| Scheduler | Cron-invoked process that evaluates cron expressions and dispatches due jobs | Infrastructure - "when" to create jobs |

### Technical Terms

| Term | Definition | Context |
|------|------------|---------|
| BRPOPLPUSH | Redis blocking pop-and-push used for reservation | At-least-once delivery without busy-wait |
| Cron Expression | Schedule format like `*/5 * * * *` (every 5 minutes) | Job scheduling configuration |
| Heartbeat | Periodic signal from worker indicating it's alive | Health monitoring |
| Orphan Job | Execution stuck in "running" status due to worker crash | Error recovery detection |
| TypeNum | Store identifier pattern `[a-z]{2}\d+` | Multi-tenant store context |