# Task Engine Deployment Guide

## Overview

This guide covers deployment and operations for the Task Engine background job system.

## Prerequisites

- PHP 8.x with PCNTL extension
- Redis server
- MySQL database with migrations applied
- Supervisor or systemd for process management

## Database Migrations

Apply the Task Engine database migrations:

```bash
cd userfrosting
php conductor run
```

This creates:
- `task_job_definitions` - Job schedules and configuration
- `task_executions` - Execution history
- `task_execution_logs` - Detailed logs per execution
- `task_workers` - Active worker registration

## Configuration

### Environment Variables

```bash
# Redis connection
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_DB=0

# Task Engine settings
TASK_ENGINE_ENABLED=true
TASK_ENGINE_WORKER_COUNT=3
TASK_ENGINE_RESERVE_TIMEOUT=5
TASK_ENGINE_RECYCLE_AFTER=1000

# Notification settings
TASK_ENGINE_NOTIFY_EMAIL=admin@example.com
TASK_ENGINE_NOTIFY_FROM=noreply@buyerkiosk.com
```

### Supervisor Configuration

Create `/etc/supervisor/conf.d/task-engine.conf`:

```ini
[program:task-scheduler]
command=/usr/bin/php /var/www/userfrosting/bin/task scheduler:run
directory=/var/www
user=www-data
autostart=true
autorestart=true
stderr_logfile=/var/log/task-engine/scheduler.err.log
stdout_logfile=/var/log/task-engine/scheduler.out.log
startsecs=10
stopwaitsecs=30

[program:task-worker]
command=/usr/bin/php /var/www/userfrosting/bin/task worker:run --queues=high,default,low
directory=/var/www
user=www-data
autostart=true
autorestart=true
stderr_logfile=/var/log/task-engine/worker.err.log
stdout_logfile=/var/log/task-engine/worker.out.log
numprocs=3
process_name=%(program_name)s_%(process_num)02d
startsecs=10
stopwaitsecs=60
```

### systemd Configuration (Alternative)

Create `/etc/systemd/system/task-scheduler.service`:

```ini
[Unit]
Description=Task Engine Scheduler
After=network.target redis.service mysql.service

[Service]
Type=simple
User=www-data
WorkingDirectory=/var/www
ExecStart=/usr/bin/php /var/www/userfrosting/bin/task scheduler:run
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

Create `/etc/systemd/system/task-worker@.service`:

```ini
[Unit]
Description=Task Engine Worker %i
After=network.target redis.service mysql.service

[Service]
Type=simple
User=www-data
WorkingDirectory=/var/www
ExecStart=/usr/bin/php /var/www/userfrosting/bin/task worker:run --queues=high,default,low
Restart=always
RestartSec=10
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target
```

Enable workers:

```bash
systemctl enable task-scheduler
systemctl enable task-worker@{1..3}
systemctl start task-scheduler
systemctl start task-worker@{1..3}
```

## CLI Commands

### Scheduler

```bash
# Run scheduler daemon (continuously evaluates schedules)
php bin/task scheduler:run

# Run single scheduler cycle (for testing)
php bin/task scheduler:tick
```

### Workers

```bash
# Run worker daemon
php bin/task worker:run --queues=high,default,low

# Run single job (for testing)
php bin/task worker:once
```

### Job Management

```bash
# Dispatch a job manually
php bin/task task:dispatch <job_name> --store=<typeNum>

# Abort a running job
php bin/task task:abort <execution_id>

# List active workers
php bin/task worker:list

# View job status
php bin/task task:status <execution_id>
```

## Registering Jobs

Jobs must be registered in the JobRegistry. Add to `userfrosting/src/BuyerKiosk/TaskEngine/Registry/JobRegistry.php`:

```php
$this->jobs = [
    'sync-customers' => SyncCustomersJob::class,
    'generate-reports' => GenerateReportsJob::class,
    'cleanup-sessions' => CleanupSessionsJob::class,
    // Add your jobs here
];
```

## Creating Job Definitions

Insert job definitions via SQL or admin dashboard:

```sql
INSERT INTO task_job_definitions (
    name,
    display_name,
    schedule,
    queue,
    job_class,
    for_each_store,
    enabled,
    timeout,
    max_retries,
    config
) VALUES (
    'sync-customers',
    'Customer Sync',
    '0 * * * *',  -- Every hour
    'default',
    'BuyerKiosk\\TaskEngine\\Jobs\\SyncCustomersJob',
    1,  -- Per-store job
    1,  -- Enabled
    300,  -- 5 minute timeout
    3,   -- 3 retries
    '{}'
);
```

## Admin Dashboard

Access the dashboard at `/admin/:typeNum/tasks`

Features:
- View active workers and their status
- Monitor queue depths
- See recent executions with status
- View scheduled jobs and next run times
- Manual job dispatch ("Run Now" button)
- Retry failed jobs
- Abort running jobs
- View detailed logs per execution

## Monitoring

### Health Checks

1. **Scheduler Health**: Check that scheduler process is running
2. **Worker Health**: Check workers have recent heartbeat (< 60s)
3. **Queue Depths**: Monitor queue depths don't grow unbounded
4. **Failed Jobs**: Alert on high failure rate (> 10%)

### Metrics to Track

- Jobs processed per minute
- Average job duration
- Queue depth per queue type
- Worker utilization (busy vs idle time)
- Retry rate
- Failure rate

### Log Files

- `/var/log/task-engine/scheduler.log` - Scheduler activity
- `/var/log/task-engine/worker.log` - Worker activity
- Database `task_execution_logs` table - Per-job logs

## Troubleshooting

### Jobs Not Running

1. Check scheduler is running: `ps aux | grep scheduler`
2. Check Redis connection: `redis-cli ping`
3. Check job is enabled in database
4. Check cron expression is valid
5. Check logs for errors

### Jobs Stuck in Running

1. Check worker processes are alive
2. Look for orphaned jobs (running > 2x timeout)
3. Scheduler will auto-detect and requeue orphans

### High Queue Depth

1. Add more workers: increase `numprocs` in supervisor
2. Check for slow jobs causing backup
3. Consider priority queues for critical jobs

### Worker Crashes

1. Check error logs for exceptions
2. Verify memory limits aren't being exceeded
3. Check for infinite loops in job code
4. Workers auto-restart via supervisor/systemd

## Scaling

### Horizontal Scaling

- Run workers on multiple servers
- All workers share Redis and MySQL
- No sticky sessions required

### Queue Priority

Jobs can be assigned to priority queues:
- `high` - Critical, time-sensitive jobs
- `default` - Normal priority
- `low` - Background, non-urgent jobs

Workers process queues in priority order.

### Performance Tuning

- Increase `numprocs` for more workers
- Adjust `recycleAfter` to control memory
- Use `timeout` appropriately per job type
- Monitor and optimize slow jobs

## Security

- Dashboard requires `uri_task_dashboard` permission
- API endpoints require authentication + CSRF token
- Store isolation enforced via `checkStoreGroup()`
- Job classes must be registered (whitelist)

## Backup & Recovery

- Job definitions in MySQL (backup with database)
- Execution history in MySQL (30-day retention default)
- Redis queues are transient (jobs re-enqueue on scheduler tick)
- Workers are stateless (can restart anytime)