# TaskEngine Infrastructure Audit — Payroll Readiness

**Spec:** 050-everee-payroll-foundations  
**Task:** T4 (PRD F12, lines 214–220; SDD §"Pattern Documentation", lines 2588–2592)  
**Date:** 2026-05-29  
**Author:** Developer Agent (Sonnet 4.6)  
**Status:** Pending second-engineer sign-off (see end of document)

---

## 1. Queue Inventory

Three queue names are used across the entire codebase. Workers poll them in priority order — `high` first, then `default`, then `low`. This ordering is hard-coded in `Worker.php` (line 22) and confirmed by `Worker::create()` (line 42).

> **Important caveat — DB definitions override static `getQueue()`.**  
> The queue assignments below are drawn from each job class's static `getQueue()` method.  
> At runtime, the effective queue is read from the **database `task_job_definitions.queue`
> column** (populated when the migration was applied), which overrides the static PHP default
> (`JobDispatcher.php:244–255`; `JobDefinition.php:151–153`). Always verify the DB record
> (or migration JSON) for the true runtime queue — the static method is only the fallback.

### `high`

| Job Class | File | Notes |
|-----------|------|-------|
| `AiScheduleGenerationJob` | `src/BuyerKiosk/Scheduling/AiScheduling/Jobs/AiScheduleGenerationJob.php:170` | User-facing AI schedule generation; time-sensitive |
| `SolverScheduleGenerationJob` | `src/BuyerKiosk/Scheduling/AiScheduling/Jobs/SolverScheduleGenerationJob.php:181` | Deterministic solver; also time-sensitive |

### `default`

| Job Class | File |
|-----------|------|
| `EventPhaseJob` | `src/BuyerKiosk/TaskEngine/Jobs/EventPhaseJob.php:34` |
| `EmployeeSyncJob` | `src/BuyerKiosk/TaskEngine/Jobs/EmployeeSyncJob.php:33` |
| `QuickBooksSyncJob` | `src/BuyerKiosk/TaskEngine/Jobs/QuickBooksSyncJob.php:46` |
| `QuickBooksStuckPostingRecoveryJob` | `src/BuyerKiosk/TaskEngine/Jobs/QuickBooksStuckPostingRecoveryJob.php:71` |
| `FiveStarsSyncJob` | `src/BuyerKiosk/TaskEngine/Jobs/FiveStarsSyncJob.php:39` |
| `FiveStarsPointsPostingJob` | `src/BuyerKiosk/TaskEngine/Jobs/FiveStarsPointsPostingJob.php:81` |
| `MockDataGeneratorJob` | `src/BuyerKiosk/TaskEngine/Jobs/MockDataGeneratorJob.php:50` |
| `DemoSimulationResetJob` | `src/BuyerKiosk/TaskEngine/Jobs/DemoSimulationResetJob.php:31` |
| `DemoDeliverySimulationJob` | `src/BuyerKiosk/TaskEngine/Jobs/DemoDeliverySimulationJob.php:61` | **static=`default`, DB=`low`** — migration `20260211_037_004_demo_simulation_job.json:7` inserts with `queue='low'`; DB wins at runtime |
| `PushNotificationQueueJob` | `src/BuyerKiosk/TaskEngine/Jobs/PushNotificationQueueJob.php:59` |
| `FloorPlanTaskGeneratorJob` | `src/BuyerKiosk/TaskEngine/Jobs/FloorPlanTaskGeneratorJob.php:43` |
| `InvoiceGenerationJob` | `src/BuyerKiosk/TaskEngine/Jobs/InvoiceGenerationJob.php:59` |
| `TrialExpirationJob` | `src/BuyerKiosk/TaskEngine/Jobs/TrialExpirationJob.php:41` |
| `SmsStaleCostSweepJob` | `src/BuyerKiosk/TaskEngine/Jobs/SmsStaleCostSweepJob.php:68` | **static=`default`, DB=`low`** — migration `20260211_037_004b_stale_cost_sweep_job.json:7` inserts with `queue='low'`; DB wins at runtime |
| `StatuspageSyncJob` | `src/BuyerKiosk/TaskEngine/Jobs/StatuspageSyncJob.php:38` |
| `RecurringShiftExtensionJob` | `src/BuyerKiosk/TaskEngine/Jobs/RecurringShiftExtensionJob.php:50` |
| `StaffChatRetentionJob` | `src/BuyerKiosk/StaffChat/Jobs/StaffChatRetentionJob.php:76` |
| `ProcessOutboxJob` | `src/BuyerKiosk/StaffChat/Jobs/ProcessOutboxJob.php:69` |
| `CloseReportNotifyJob` | `src/BuyerKiosk/CloseReport/Jobs/CloseReportNotifyJob.php:49` |
| `HourlyMetricsCollectorJob` | `src/BuyerKiosk/Scheduling/AiScheduling/Jobs/HourlyMetricsCollectorJob.php:60` |
| `GoalForecastComputeJob` | `src/BuyerKiosk/Goals/Jobs/GoalForecastComputeJob.php:46` |
| `PosDataCatchupJob` | `src/BuyerKiosk/PosDataCatchup/Jobs/PosDataCatchupJob.php:48` |

### `low`

| Job Class | File | Notes |
|-----------|------|-------|
| `AggregateStatsJob` | `src/BuyerKiosk/TaskEngine/Jobs/AggregateStatsJob.php:30` | |
| `StoreMetricsAggregatorJob` | `src/BuyerKiosk/TaskEngine/Jobs/StoreMetricsAggregatorJob.php:33` | |
| `HomepageStatsJob` | `src/BuyerKiosk/TaskEngine/Jobs/HomepageStatsJob.php:46` | |
| `FcmTokenCleanupJob` | `src/BuyerKiosk/TaskEngine/Jobs/FcmTokenCleanupJob.php:57` | |
| `BackfillMockDataJob` | `src/BuyerKiosk/TaskEngine/Jobs/BackfillMockDataJob.php:69` | |
| `SlideExpirationJob` | `src/BuyerKiosk/TaskEngine/Jobs/SlideExpirationJob.php:50` | |
| `ComebackCashEventAutoEndJob` | `src/BuyerKiosk/TaskEngine/Jobs/ComebackCashEventAutoEndJob.php:45` | |
| `StrandedBuyCleanupJob` | `src/BuyerKiosk/TaskEngine/Jobs/StrandedBuyCleanupJob.php:48` | "runs at end of day" |
| `SmsCostLookupJob` | `src/BuyerKiosk/TaskEngine/Jobs/SmsCostLookupJob.php:61` | |
| `StockoutDetectionJob` | `src/BuyerKiosk/Replenishment/Jobs/StockoutDetectionJob.php:60` | |
| `VelocityAggregationJob` | `src/BuyerKiosk/Analytics/Jobs/VelocityAggregationJob.php:67` | |
| `MomentumAggregationJob` | `src/BuyerKiosk/Analytics/Jobs/MomentumAggregationJob.php:61` | |
| `RetentionAggregationJob` | `src/BuyerKiosk/Analytics/Jobs/RetentionAggregationJob.php:51` | |
| `AiScheduleCleanupJob` | `src/BuyerKiosk/Scheduling/AiScheduling/Jobs/AiScheduleCleanupJob.php:84` | |
| `HourlyMetricsBackfillJob` | `src/BuyerKiosk/Scheduling/AiScheduling/Jobs/HourlyMetricsBackfillJob.php:72` | |
| `MigrateCloseReportsJob` | `src/BuyerKiosk/CloseReport/Jobs/MigrateCloseReportsJob.php:45` | |
| `MigrateLegacyCloseReportsJob` | `src/BuyerKiosk/CloseReport/Jobs/MigrateLegacyCloseReportsJob.php:45` | |

**Total distinct queues: 3** — `high`, `default`, `low`.  
No `critical` or `payroll` queue exists today.

---

## 2. Worker Classes and Dispatch

### 2.1 Class hierarchy

```
JobInterface                      (Domain/Job/JobInterface.php)
└── BaseJob  (abstract)           (Domain/Job/BaseJob.php)
    └── ConcreteJob               e.g. GoalForecastComputeJob.php
```

`JobInterface` mandates **5 static configuration methods** and **3 runtime methods** (`handle()`, `setContext()`, `getContext()`), for 8 methods total (`JobInterface.php:20–78`). Every job is identified by its string name returned from `getName()` — e.g. `'goal-forecast-compute'` (GoalForecastComputeJob.php:35).

Static: `getName()`, `getDisplayName()`, `getQueue()`, `getScope()`, `getTimeout()`.  
Runtime: `handle()`, `setContext()`, `getContext()`.

`BaseJob` provides:

- `setContext(JobContext)` / `getContext()` — lifecycle injection (BaseJob.php:121–136)
- `progress(int, string)` — updates the execution record and optionally logs (BaseJob.php:169–176)
- `heartbeat()` — throttled ping to keep the worker-health row current (BaseJob.php:289–292)
- `checkpoint()` — combined heartbeat + abort-check for loop bodies (BaseJob.php:310–313)
- `shouldAbort()` / `checkAbort()` — cooperative abort via abort signal (BaseJob.php:241–265)
- `log()` / `debug()` / `info()` / `warning()` / `error()` — route to `ExecutionLogger` (BaseJob.php:189–226)
- `getStore()` / `getTypeNum()` / `getStoreDb()` — store-scoped helpers (BaseJob.php:334–361)
- `config(key, default)` / `payload(key, default)` / `getPayload()` — config and payload access (BaseJob.php:374–398)
- `getAttempt()` / `isRetry()` — retry awareness (BaseJob.php:410–423)

### 2.2 Job definitions in the database

Schema is in `migrations/input/20251217_001_task_engine_tables.json` (table `task_job_definitions`). Key columns:

| Column | Type | Default | Meaning |
|--------|------|---------|---------|
| `name` | varchar(100) | — | Unique slug, matches `JobClass::getName()` |
| `queue` | varchar(50) | `'default'` | Queue name; INDEX `idx_queue` |
| `scope` | enum | `'global'` | `global` or `per_store` |
| `timeout` | int unsigned | `300` | Max execution seconds |
| `maxRetries` | int unsigned | `3` | Maximum retry attempts |
| `retryBackoff` | int unsigned | `60` | Base seconds for exponential backoff |
| `config` | json | NULL | Arbitrary per-job config blob |
| `isEnabled` | tinyint | `1` | Can be disabled without deletion |
| `notifyOnFailure` | tinyint | `1` | Trigger failure-notification path |
| `notifySms` | tinyint | `0` | Per-job SMS opt-in (added migration `20260427_001_task_job_definitions_notifySms.json`) |

`JobDefinition.fromRow()` hydrates from this table (JobDefinition.php:84–120). The `JobDefinition` entity is the only source of truth for timeout, retries, and backoff — the job *class* only declares these as fallback statics; the database row wins at runtime.

### 2.3 Dispatch flow (scheduled)

1. `TaskCommandFactory::create()` wires the full object graph (TaskCommandFactory.php:69–167).
2. `Scheduler` calls `ScheduleEvaluator` every minute (invoked by `php userfrosting/bin/task scheduler:run` via cron).
3. For each due job, `JobDispatcher::dispatch()` is called:
   - Checks for an existing execution with the same **idempotency key** to prevent duplicates (JobDispatcher.php:69–77).
   - Creates an `Execution` record in `task_executions` (status=`pending`) and pushes a JSON payload to the Redis queue (JobDispatcher.php:80–112).
4. Idempotency key format for scheduled jobs: `<name>:<YYYY-MM-DDTHH:MM:00Z>[:<typeNum>]` (JobDispatcher.php:204).
5. For `per_store` jobs, `dispatchPerStore()` creates one execution per active store (JobDispatcher.php:158–180).

### 2.4 Dispatch flow (manual)

`JobDispatcher::dispatchManual()` generates a non-repeatable key `<name>:manual:<timestamp>:<random8hex>` (JobDispatcher.php:226–232), allowing multiple concurrent manual runs.

### 2.5 Worker execution loop

Workers poll `['high', 'default', 'low']` in priority order using `RedisQueueAdapter::reserve()`, which performs an atomic `RPOPLPUSH` (or `BRPOPLPUSH` with blocking) from the queue list into a processing list (RedisQueueAdapter.php:66–105). Key Redis structures:

| Key pattern | Type | Purpose |
|-------------|------|---------|
| `taskengine:queue:<name>` | List | Pending jobs |
| `taskengine:processing:<name>` | List | In-flight jobs (at-least-once recovery) |
| `taskengine:processing_meta:<name>` | Hash | `executionId → reserved_at` timestamps |
| `taskengine:delayed:<name>` | Sorted set | Delayed/retry jobs (score = run-at unix ts) |

After `handle()` returns, `ack()` removes the item from `processing:<name>` (RedisQueueAdapter.php:110–131).

### 2.6 Orphan recovery

`ExecutionRecoveryService` scans for executions stuck in `running` for more than `2 × timeout` seconds and either re-queues them for retry or marks them permanently failed (ExecutionRecoveryService.php:25–104). Called from `worker:manager --cleanup` on a 5-minute cron.

---

## 3. Retry Semantics

### 3.1 Configuration

Each job definition row carries `maxRetries` (default 3) and `retryBackoff` (default 60 s). These can be overridden per-job in the database; the code reads them from `JobDefinition.getMaxRetries()` and `JobDefinition.getRetryBackoff()` (JobDefinition.php:166–174).

### 3.2 Algorithm — exponential backoff with jitter

`RetryScheduler.calculateDelay()` implements (RetryScheduler.php:47–81):

```
exponent = max(0, attemptNumber - 2)
delay    = baseBackoff × 2^exponent
jitter   = random(0, delay × 0.10)
final    = min(delay + jitter, maxDelay=3600)
```

Attempt progression (base=60 s, **`maxRetries=4`** to yield 3 actual retries — see §3.3):

| Attempt | Retry # | Formula | Approx delay |
|---------|---------|---------|--------------|
| 2 | retry 1 | 60 × 2^0 = 60 | ~60 s |
| 3 | retry 2 | 60 × 2^1 = 120 | ~120 s |
| 4 | retry 3 | 60 × 2^2 = 240 | ~240 s |

Cap is 3600 s (1 hour). Jitter is applied *before* capping.

**Note:** At the **default `maxRetries=3`**, only 2 automatic retries occur (attempts 2 and 3). Attempt 4 is never reached at defaults — see §3.3.

### 3.3 Retry eligibility check

`RetryScheduler.canRetry()` requires (RetryScheduler.php:90–103):

- `execution.isFailed() === true`
- `jobDef.maxRetries > 0`
- `execution.attempt < jobDef.maxRetries` (strictly less than)

Executions start at `attempt = 1` (`Execution.php:68`). With the default `maxRetries = 3`:

| Attempt at point of failure | `attempt < 3`? | Outcome |
|-----------------------------|----------------|---------|
| 1 | 1 < 3 = **true** | Retry → becomes attempt 2 |
| 2 | 2 < 3 = **true** | Retry → becomes attempt 3 |
| 3 | 3 < 3 = **false** | Permanently failed |

**At `maxRetries = 3` there are exactly 2 automatic retries; attempt 4 never occurs.**
To get 3 automatic retries, set `maxRetries = 4`.

### 3.4 Retry push

`RetryScheduler.scheduleRetry()` calls `queue.pushDelayed(jobDef.queue, payload, runAt)` (RetryScheduler.php:140–144). The delayed sorted set is promoted to the live queue by the scheduler's `promoteDue()` sweep.

### 3.5 Terminal failure and manual re-drive

When `canRetry()` returns false — either retries exhausted or `maxRetries=0` — the execution stays in `status='failed'`. There is **no separate dead-letter queue** and **no automatic re-drive**. The failure remains visible in `task_executions` and triggers `FailureNotifier` (email + optional SMS per `notifySms`). Operators review failures via the admin tasks dashboard.

**However, a manual re-drive path does exist.** Two API endpoints call
`Execution::prepareForRetry()` (which increments the attempt counter, resets status to
`PENDING`, and clears runtime fields) and then re-pushes the execution to its job's queue:

| Endpoint | Handler | Auth |
|----------|---------|------|
| `POST /api/:typeNum/tasks/executions/:executionId/retry` (`routes/task-engine/api.php:99`) | `TaskApiController::retryExecution()` (`TaskApiController.php:618–687`) | Store-level access |
| `POST /api/tasks/executions/:id/retry` (`routes/task-engine/api.php:169`) | `TaskApiController::retryGlobalExecution()` (`TaskApiController.php:1217–1275`) | Super-admin |

The re-drive is immediate (not delayed); the execution is pushed directly to the queue,
bypassing the delayed sorted set. This is a manual operator action — there is no
automation or scheduled DLQ sweep.

---

## 4. Gap Analysis for Payroll

### 4.1 Webhook async processing

**Target workload:** `ProcessEvereeWebhookJob` (T10) — receives an Everee event payload, validates the HMAC signature, idempotency-checks the `webhookEventId`, and performs state mutations. Expected volume: low-to-moderate webhook bursts from Everee, latency-sensitive (Everee may retry within seconds if we do not 200 quickly).

#### What TaskEngine already supports

- **Idempotency at dispatch layer**: `JobDispatcher.dispatch()` checks `findByIdempotencyKey()` before creating a duplicate execution (JobDispatcher.php:70–77). Webhook jobs can pass `webhookEventId` as the idempotency key.
- **Payload delivery**: `Execution.payload` is a JSON column (migration `task_executions`); the entire deserialized webhook body can be carried as the payload array.
- **At-least-once delivery**: `RPOPLPUSH` guarantees the payload survives a worker crash and is recovered by `ExecutionRecoveryService` (RedisQueueAdapter.php:73–76; ExecutionRecoveryService.php).
- **Progress and logging**: `BaseJob` provides structured per-execution logs (BaseJob.php:189–226).

#### What is missing / gaps

| Gap | Severity | Notes |
|-----|----------|-------|
| **No dedicated `payroll` or `high` queue priority for webhooks** | Medium | Webhook jobs would land on `default` unless a new queue is added. Under `default` queue pressure from billing, QB-sync etc., Everee webhook processing could be delayed. Everee's retry window is typically 30–60 s. |
| **No HMAC validation helper in TaskEngine** | Low | HMAC validation belongs in the HTTP layer (Slim route), not TaskEngine. The job receives an already-validated + stored payload. No gap here *per se*, but the storage pattern for the raw payload (pre-validation vs post-validation) needs to be specified in T10. |
| **Payload size limit undocumented** | Low | `task_executions.payload` is a `json` column with no explicit size cap. MariaDB JSON columns default to 64 KB effective limit. Everee webhook bodies are small (< 2 KB); this is not a practical limit but worth noting. |

#### Recommendation

**Fix in Phase 1a** — add a new `payroll` queue priority level (between `high` and `default`) in the following files:

- `src/BuyerKiosk/TaskEngine/Domain/Worker/Worker.php` — add `'payroll'` to the default queue array (line 22 + line 42)
- `src/BuyerKiosk/TaskEngine/Domain/Worker/WorkerProcess.php` — if queue list is also in this file, add it there too
- **`src/BuyerKiosk/TaskEngine/Commands/TaskCommand.php:36`** — add `'payroll'` to `DEFAULT_QUEUES`; this is the list used when starting workers without `--queues`
- **`src/BuyerKiosk/TaskEngine/Domain/Scheduler/Scheduler.php:35`** — add `'payroll'` to `DEFAULT_QUEUES`; this is the list used by `promoteDelayedRetries()` (`Scheduler.php:322–323`)
- `src/BuyerKiosk/TaskEngine/Jobs/ProcessEvereeWebhookJob.php` (new file T10) — return `'payroll'` from `getQueue()`
- Migration JSON to register the job definition on the `payroll` queue

> **Pushed-but-not-polled trap (T10 risk):** A new queue is useless if workers never read it.
> `TaskCommand::DEFAULT_QUEUES` (`:36`) controls which queues a worker polls when started
> without an explicit `--queues` flag. `Scheduler::DEFAULT_QUEUES` (`:35`) controls which
> delayed-retry sorted sets are promoted each minute. Both constants must be updated or the
> `payroll` queue will accumulate items that are never drained and retries that are never
> promoted.

The effort is small (< 2 hours) and prevents a class of latency problems before they appear in production. Workers already poll in configurable order; adding a named level costs nothing at runtime.

---

### 4.2 Daily reconciliation

**Target workload:** `EvereeReconciliationJob` (future Phase 1c) — per-store, scheduled nightly, compares BuyerKiosk punch data against Everee pay-run data, writes discrepancy records, optionally fires alerts.

#### What TaskEngine already supports

- **`per_store` scope**: The scheduler's `dispatchPerStore()` iterates active stores and emits one execution per store with the correct `typeNum` (JobDispatcher.php:158–180). `GoalForecastComputeJob` is a working `per_store` example (GoalForecastComputeJob.php:49).
- **Nightly cron scheduling**: `JobDefinition.isDue()` evaluates cron expressions via `CronExpression::factory()` (JobDefinition.php:220–236). A `"0 2 * * *"` schedule works out of the box.
- **Long timeout support**: `timeout` is configurable per job definition; the `GoalForecastComputeJob` uses 600 s (10 min) (GoalForecastComputeJob.php:57). Reconciliation can set a similarly generous limit.
- **Checkpoint pattern**: `BaseJob.checkpoint()` (BaseJob.php:310–313) enables graceful abort during multi-store iteration loops.
- **Execution audit trail**: Every run produces a `task_executions` row + structured `task_execution_logs`, which constitutes an audit log of reconciliation runs.

#### What is missing / gaps

| Gap | Severity | Notes |
|-----|----------|-------|
| **No `EvereeReconciliationJob` class yet** | Expected — Phase 1c | This is the job class itself; not a TaskEngine infrastructure gap. |
| **No retry strategy for partial reconciliation** | Medium | If a reconciliation run fails mid-store (DB error, Everee API timeout), the retry replays from the beginning. There is no cursor/checkpoint persistence between attempts. For initial Phase 1a scope this is acceptable; large stores may see duplicate discrepancy writes on retry unless the job is idempotent in its writes. |
| **No cross-execution ordering guarantee** | Low | If a store has two reconciliation executions queued simultaneously (manual re-run while scheduled is pending), both will run. Idempotency at the idempotency-key level prevents duplicate *scheduling*, but a manual dispatch would generate its own key. |

#### Recommendation

**Defer to Phase 1c.** The daily reconciliation job is explicitly a Phase 1c deliverable. TaskEngine already provides everything needed: `per_store` dispatch, cron scheduling, timeouts, progress reporting, and execution history. The only Phase 1a prerequisite is ensuring the job class can be registered without conflicts — which requires no infrastructure changes. Design the reconciliation job to be idempotent in its writes (upsert on `(typeNum, payPeriodId, workerId)`) to handle retry replays.

---

### 4.3 PTO accrual

**Target workload:** `PtoAccrualJob` (future Phase 1c) — global or per-store, scheduled weekly or on pay-period boundaries, computes accrued PTO balances and writes them to `ptoBalance` records.

#### What TaskEngine already supports

- **Global scope available**: `getScope()` can return `'global'` for a single-execution accrual run that loops all stores internally, or `'per_store'` for parallelized accrual. Both modes exist today (JobInterface.php:39–44).
- **Config blob**: `task_job_definitions.config` is a JSON column; accrual parameters (accrual rate, policy type, carry-over rules) can be stored there as `JobDefinition.getConfigValue(key, default)` (JobDefinition.php:270–273). This avoids hard-coding accrual constants in the job class.
- **Scheduled trigger**: Same cron-expression mechanism as reconciliation; `"0 0 * * 0"` for weekly works out of the box.
- **Retry semantics**: Three retries at 60/120/240 s cover transient DB errors during accrual writes.

#### What is missing / gaps

| Gap | Severity | Notes |
|-----|----------|-------|
| **No `PtoAccrualJob` class yet** | Expected — Phase 1c | Not a TaskEngine gap. |
| **No per-store config differentiation within a single global job** | Medium | If accrual policies differ per store (different rates, different carry-over caps), a global-scope job would need to look up per-store config from a DB table at runtime rather than reading from `task_job_definitions.config` (which is one row). This is a design constraint, not a blocking gap — either use `per_store` scope (one execution per store, each reading store-specific rules) or have the global job fetch store-specific rows. |
| **No idempotency guarantee across pay-period boundaries** | Medium | The scheduler's built-in idempotency key is time-based (`<job-name>:<fire-at-utc>[:<typeNum>]`, `Scheduler.php:294–305`). A catch-up run or reschedule generates a different fire time and bypasses scheduler-level deduplication. The scheduler exposes **no hook to inject a custom idempotency key** — that parameter is only available on `JobDispatcher::dispatchManual()` (`JobDispatcher.php:128–143`). Period-scoped exactly-once semantics therefore cannot be achieved via a scheduler custom key. |

#### Recommendation

**Defer to Phase 1c.** All TaskEngine primitives needed for PTO accrual exist. Use `per_store` scope so each store's accrual runs independently and can be retried without re-processing all stores.

**Critical design constraint:** Because the scheduler builds its own time-based idempotency key with no custom-key hook (`Scheduler.php:243–261`, `Scheduler.php:294–305`), pay-period-scoped exactly-once semantics **must be implemented inside the job's `handle()` method** as an internal idempotency guard — not via a scheduler custom key. Example pattern:

```php
// Inside PtoAccrualJob::handle()
if ($this->submissionRepo->existsForPeriod($typeNum, $periodStart, $periodEnd)) {
    return JobResult::success('Already submitted for this period — skipping');
}
// ... compute and submit accruals
$this->submissionRepo->record($typeNum, $periodStart, $periodEnd);
```

A `payrollPtoSubmissions` state table (columns: `typeNum`, `periodStart`, `periodEnd`, `submittedAt`, `status`) makes re-drives via `TaskApiController` safe: the job short-circuits without re-submitting to Everee.

> If manual-dispatch wrappers are used outside the scheduler, `dispatchManual()` does
> accept a custom `idempotencyKey` (`JobDispatcher.php:128–143`) — but the scheduler
> path cannot use this.

---

## 5. Summary of Recommendations

| Workload | Recommendation | Key files touched |
|----------|---------------|-------------------|
| Webhook async processing | **Fix in Phase 1a** — add `payroll` queue | `Worker.php:22,42`, `WorkerProcess.php` (if applicable), new `ProcessEvereeWebhookJob.php`, new migration JSON |
| Daily reconciliation | **Defer to Phase 1c** | Job class + migration only; zero infrastructure changes needed |
| PTO accrual | **Defer to Phase 1c** | Job class + migration only; zero infrastructure changes needed |

### Surprising findings

1. **No automatic dead-letter queue exists, but a manual re-drive path does.** Permanently failed executions stay in `task_executions` with `status='failed'`. There is no dedicated dead-letter queue or automated re-drive scheduler. However, `TaskApiController::retryExecution()` (`TaskApiController.php:618–687`) and `TaskApiController::retryGlobalExecution()` (`TaskApiController.php:1217–1275`) provide a manual re-drive path that calls `prepareForRetry()` and re-queues the execution immediately. Routes: `POST /api/:typeNum/tasks/executions/:executionId/retry` (`api.php:99`) and `POST /api/tasks/executions/:id/retry` (`api.php:169`). For high-value payroll operations, alert on `status='failed'` so operators know to trigger this path promptly.

2. **`canRetry()` is strictly less-than, not less-than-or-equal.** `execution.attempt < jobDef.maxRetries` (RetryScheduler.php:103) means a job with `maxRetries=3` gets at most **2 retries** (attempts 1, 2, 3 — but attempt 3 fails the `< 3` check before retry). This is off-by-one relative to the description comment "max 3 retries." Payroll job definitions should set `maxRetries=4` if 3 actual retries are required. Worth filing as a documentation/behavior clarification issue.

3. **`promoteDue()` lacks atomicity per item, but the scheduler lock guards normal operation.** `RedisQueueAdapter.promoteDue()` iterates due items and calls `rpush` + `zrem` separately per item (RedisQueueAdapter.php:154–177) — a crash between those two calls could result in a duplicate promotion. However, the scheduler is protected by a Redis `SET NX EX` distributed lock (`Scheduler.php:85–117`, `132–156`) that ensures only one scheduler instance runs at a time, making this race extremely unlikely in practice. The residual risk (process crash between `rpush` and `zrem` on the same item) is low. Jobs should still be idempotent at the application level to safely tolerate the rare case.

4. **`WorkerProcess.php` not read.** This file was found to exist (`src/BuyerKiosk/TaskEngine/Domain/Worker/WorkerProcess.php`) but was not required reading. If the worker queue list is also hard-coded there, the payroll-queue addition in Section 4.1 must cover that file as well.

---

## 6. File Reference Index

| File | Role |
|------|------|
| `src/BuyerKiosk/TaskEngine/Domain/Job/JobInterface.php` | Contract all jobs must satisfy |
| `src/BuyerKiosk/TaskEngine/Domain/Job/BaseJob.php` | Abstract base with lifecycle, logging, heartbeat, abort |
| `src/BuyerKiosk/TaskEngine/Domain/Job/JobDefinition.php` | Entity for DB row in `task_job_definitions` |
| `src/BuyerKiosk/TaskEngine/Application/JobDispatcher.php` | Dispatch + idempotency-key logic |
| `src/BuyerKiosk/TaskEngine/Commands/TaskCommandFactory.php` | Wiring + job registry |
| `src/BuyerKiosk/TaskEngine/Services/RetryScheduler.php` | Exponential backoff + retry eligibility |
| `src/BuyerKiosk/TaskEngine/Services/ExecutionRecoveryService.php` | Orphan recovery for dead workers |
| `src/BuyerKiosk/TaskEngine/Infrastructure/Queue/QueueInterface.php` | Queue contract |
| `src/BuyerKiosk/TaskEngine/Infrastructure/Queue/RedisQueueAdapter.php` | Redis implementation |
| `src/BuyerKiosk/TaskEngine/Domain/Worker/Worker.php` | Worker entity; default queue order `[high, default, low]` |
| `src/BuyerKiosk/Goals/Jobs/GoalForecastComputeJob.php` | Canonical `per_store` job example |
| `migrations/input/20251217_001_task_engine_tables.json` | Authoritative schema for all TaskEngine tables |
| `migrations/input/20260427_001_task_job_definitions_notifySms.json` | `notifySms` column addition |

---

## 7. Second-Engineer Sign-Off

This audit must be reviewed by one engineer other than the author before Feature 8's
`ProcessEvereeWebhookJob` async-dispatch work (T10) merges, per PRD F12 AC (lines 218–220).

```
Signed-off-by: ___________________________
```