# Spec 051 — Rollback Playbook (T8.9) Per-flag rollback effects, the kill-switch procedure, invite-queue drain/pause semantics, and what is NEVER rolled back. Companion to [`scheduling-free-launch.md`](./scheduling-free-launch.md) (the forward launch sequence) — this file covers undoing any part of that sequence, at any point, including mid-incident. **Core guarantee this whole document depends on:** every Spec 051 flag is an env var read fresh per request/process (memoized only within a single request's lifetime — see `SchedulingEntitlement::resetForTesting()`'s docblock: *"production code never needs to call this since each HTTP request is a fresh PHP process/memo"*). Flipping a flag OFF takes effect for every NEW request/job immediately after web processes + TaskEngine workers are restarted (same restart requirement as the launch runbook — a flag change is invisible to a process that already loaded the old value). --- ## 1. Kill-switch procedure (M1-06 — fastest path back to pre-051) **When to use:** an entitlement-related incident (e.g., the M1-01 inventory missed a premium check and a "free" store hit an upgrade wall, or the opposite — a premium check leaked through such that a paying customer's experience changed unexpectedly). ```bash # 1. Unset BOTH scheduling-free flags on every web process + worker: unset SCHEDULING_FREE unset SCHEDULING_FREE_PILOT_STORES # (or remove/comment the lines in .env and restart) # 2. Restart web processes and every TaskEngine worker: launchctl unload ~/Library/LaunchAgents/com.buyerkiosk.taskengine-worker*.plist launchctl load ~/Library/LaunchAgents/com.buyerkiosk.taskengine-worker*.plist # (production: restart PHP-FPM pool / whatever supervises the web process) ``` **Effect:** `SchedulingEntitlement::isEntitled($typeNum)` reduces EXACTLY to `PremiumService::isPremiumActive($typeNum)` — this is not an approximation, it is the literal code path (`isSchedulingFree()` returns false for every store when both flags are unset/false, so `isEntitled()` short-circuits to the untouched commercial predicate). Every downstream consumer (`isSchedulingEntitled()`/`isChatEntitled()` Twig functions, mobile `features.scheduling`/`features.chat`/`features.aiScheduling`, the trial banner, `SchedulingAccessPolicy`) reverts to **byte-identical pre-051 behavior** — this is the dark-deploy safety property proven by `AllFlagsMatrixIntegrationTest`'s "flags-all-off byte-identical" assertion and re-verified at T7.5. **No code deploy required** — this is exactly what M1-06 promises. **Time to effect:** the time to restart web + worker processes (seconds to low minutes) — there is no cache to bust, no migration to run. **Verification after flip:** hit `/admin/:typeNum/schedule` on a formerly-free, non-premium store and confirm the marketing/upgrade page (or whatever pre-051 behavior was) renders again. --- ## 2. Per-flag rollback effects (Release Manifest table, expanded) | Flag OFF | Immediate effect | Data effect | Re-enable safe? | |---|---|---|---| | `SCHEDULING_FREE` / `SCHEDULING_FREE_PILOT_STORES` | Entitlement reverts to commercial-premium-only (see §1) | **None** — no row is written or deleted by this flag in either direction | Yes — flipping back on is a pure read-path change | | `ONBOARDING_HUB_ENABLED` | Hub route stops rendering the M3–M12 checklist; falls back to whatever `SchedulingAccessPolicy`'s route matrix says for hub-disabled mode (legacy marketing page for not-entitled stores, or a redirect for entitled ones) | **Progress data retained** — `onboardingProgress`, `onboardingObservations`, `onboardingUserProgress` rows are NEVER deleted or altered by this flag; a later re-enable shows the SAME progress a store had before | Yes | | `ONBOARDING_INVITES_ENABLED` | New delivery, sweep, and expiry jobs check this flag and skip; a delivery already past its guard may finish | **Queued rows persist; wall-clock token expiry continues** — see §3 | Conditional — inspect/reissue elapsed invites before resume; use worker shutdown for a hard barrier | | `ONBOARDING_WALKTHROUGH_ENABLED` | Calendar page stops rendering the walkthrough trigger/engine; `WalkthroughEngine.js`'s `bootstrap()` never installs (no publish-fetch bridge either) | **Per-user walkthrough progress retained** (a separate table from step completion, per M3-04) — a user resuming after re-enable sees their prior progress | Yes | | Provider values (`stores.schedulingProvider`) | N/A — this is not a flag, it is data. **No rollback path exists or should exist** for this column via any Spec 051 mechanism — see §4 | — | — | | System Alert (M2 announcement) | Deactivate via `PUT /api/system-alerts/admin/alerts/:alertId/toggle`, or let `expiresAt` lapse | Acknowledgment history retained (Spec 040 behavior, untouched) | Re-publishing is a NEW alert (a new `alertId`), not a re-activation of the old one, if a new campaign is intended | | `ONBOARDING_CAMPAIGN_ID` | New event writes stop carrying a `campaignId` (revert to `NULL` — "ambient/organic-looking") | **Existing `onboardingEventLog` rows keep their stamped `campaignId` forever** — clearing the var does not retroactively touch any row | Setting a NEW campaign slug later starts a fresh campaign window; **never reuse a slug** already used for a closed campaign (its cohort/KPI rows would collide) | **Ordering for a full rollback (all flags):** reverse of the launch order — walkthrough off → invites off → hub off → (pilot list shrink or) fleet-wide `SCHEDULING_FREE` off last, so a store never ends up "entitled + hub off" for longer than necessary (that combination shows nothing user-visible different from fully-off, but there's no reason to linger there). If the INCIDENT is entitlement-specific, use the §1 kill switch directly — it is faster and doesn't require touching the other three flags at all. --- ## 3. Invite-queue drain/pause semantics (detail) This is the one flag whose "off" state has real in-flight state to reason about — `inviteMessages` rows already `queued`/`sending` when the flag flips off. **What happens to a `queued` row when `ONBOARDING_INVITES_ENABLED` flips off:** - A newly-started `InviteDeliveryJob::handle()` checks the flag before claim and returns early (`skipping send, message left queued`) — the row's `state` column is untouched, `attemptCount`/`nextAttemptAt` are untouched. - `InviteSweepJob::handle()` (the periodic re-queue/dead-letter sweep) ALSO checks the flag first and returns early (`skipping sweep, all rows left untouched`) — a message that would have been re-queued or dead-lettered simply sits exactly as it was. - `InviteExpiryJob::handle()` uses the same guard, so it does not transition pending invites to expired while the queue is paused. - **The row's own token/expiry timers keep running in real time** regardless of the flag — `userInvites.tokenExpiresAt` is a fixed 24h-from-issue timestamp (`InviteDeliveryService::TOKEN_EXPIRES_HOURS`), computed at ENQUEUE time, not at send time. A credential invite enqueued 20 hours before the flag flips off, then left flag-off for 6 hours, will have its token EXPIRE while queued — the member sees "ask your manager to resend" (M9-06) once the message eventually is allowed to send, or the row eventually dead-letters on attempt-count exhaustion once the flag flips back on and the job actually tries. - **This is a best-effort PAUSE, not a DRAIN.** No queued row claimed after the guard is disabled is sent. A delivery already past its one-time guard can still complete; the flag is not an instantaneous barrier for in-flight work. No queued/sending row is auto-cancelled or auto-failed by flipping the flag off. Jobs that start after the change no-op at their guard. For an incident requiring a hard barrier, stop workers and wait for active jobs to finish before treating delivery as fully paused. **Re-enabling after a pause:** turning `ONBOARDING_INVITES_ENABLED` back on resumes normal processing on the next job/sweep cycle. Inspect pending invites whose `tokenExpiresAt` elapsed during the pause before resume: the expiry job may expire them before delivery, and delivery then dead-letters their messages. Reissue those invites rather than claiming every queued row resumes unchanged. **What flipping this flag off does NOT do:** - Does NOT revoke any token already issued (a member who already has a working, unexpired activation link can still use it — the flag only gates NEW sends/sweeps, not already-completed enqueue-time work). - Does NOT stop `ActivationController`'s `GET /activate/:token` redemption endpoint — that endpoint has no dependency on `ONBOARDING_INVITES_ENABLED` at all (a member who already received their link, by whatever means, redeeming it is a matter of `ActivationController`, not this flag). - Does NOT touch billing — any SMS already sent and billed under `schedulingInvite` category before the flag flipped stays billed exactly as it was (D-6: distinct category, store's existing SMS billing policy). --- ## 4. What is NEVER rolled back These are explicit, load-bearing invariants — none of the mechanisms in this document (kill switch, per-flag rollback, invite pause) touches any of the following, by design: 1. **`stores.schedulingProvider` values.** Per M4-04: *"No store's provider is changed by migration, announcement, or any path other than [explicit activation] and [M5's support-assisted path]."* Rolling back ANY Spec 051 flag never flips a store's provider back to `none`/unset. A store that activated BK-native scheduling stays BK-native-provider even if every 051 flag is turned off afterward — turning the hub flag off just stops SHOWING the checklist; it does not un-activate the store. This is deliberate: undoing a real business decision (a store chose to activate) is out of scope for a feature-flag rollback and would require the SAME explicit-action discipline M4 itself enforces. 2. **Billing data.** Per Decision D-1/D-2: premium infrastructure and its billing records are retained dormant, never destructively cleaned up. `PREMIUM_MODULE` line items, trial/premium history, SMS usage rows under `schedulingInvite` — none of this is touched by any rollback path in this document. Commercial wind-down (Q1) is Spec 036 territory, a SEPARATE, deliberate, product/billing-owned action — never an automatic side effect of a Spec 051 flag flip. 3. **Onboarding progress/observation history.** As noted in §2's table, `onboardingProgress` (latched, per M3-05 — completion never regresses), `onboardingObservations`, and `onboardingUserProgress` rows persist across every flag flip in either direction. A completed step stays completed. 4. **Invite/activation audit trail.** `userInvites` and `inviteMessages` rows are never deleted by any rollback mechanism (only the existing retention/privacy-deletion job, `051-privacy-retention.md`, touches these — and only for user-initiated data-deletion requests, an unrelated concern). 5. **`onboardingEventLog`/`onboardingCampaignCohort` rows.** Once written (with whatever `campaignId` was live at write time), these rows are immutable history — the KPI denominator source for a campaign stays exactly as snapshotted (§3 of the launch runbook) regardless of any later flag change. 6. **Mobile app code/config.** Nothing in this rollback playbook reaches into either mobile app repo — the `clientId` login-field work (T4.1) is backend-only and additive; rolling back Spec 051 flags does not affect what either app sends or how it behaves. **If an incident seems to require touching any of the six items above, STOP — that is not a Spec 051 flag-rollback scenario, it needs its own explicit, reviewed action** (a targeted migration per the project's migration system, a Spec 036 billing decision, or a privacy-deletion request) — never a flag flip. --- ## 5. Rollback drill record (T8.7 — EXECUTED 2026-07-22) **Environment:** local dev (`dev2.buyerkiosk.com`, one agent session, no concurrent flag mutation). Baseline `.env` at drill start: exactly `SCHEDULING_FREE_PILOT_STORES=ob01` + `ONBOARDING_HUB_ENABLED=true` (MD5 `f96feb7644da0ce11d37e3478a7345c7`, verified against the two-line contract before the drill began — no `ob03`/other-051-flag remnants found). **Restart requirement, confirmed empirically:** every flag flip below took effect on the **very next web request with no process restart** — this matches `SchedulingEntitlement`'s own docblock ("each HTTP request is a fresh PHP process/memo" in this dev environment's serving model). The "restart web + worker processes" language elsewhere in this doc remains correct for **TaskEngine workers** (long-running daemons that only load `.env` at worker start) and for any production deployment using persistent PHP-FPM workers — neither was exercised by this drill, which only touched web-facing routes/APIs. Treat the timings below as "flag-edit to browser/curl-verified," not as a worker-restart benchmark. **Method:** `.env` edited directly (Edit tool), timestamped with sub-second Python wall-clock reads immediately before/after each edit; verification via a real logged-in admin session (Chrome, `rvanvuren`) using in-page `fetch()` (same-origin, credentialed) for API checks and navigation/screenshots for page-level checks. Stores used: `ob01` (pilot, `schedulingProvider='none'`), `ob02` (fleet-only test subject, not pilot-listed, `schedulingProvider='none'`, non-premium), `pc00` (real active premium BK-native store — used for calendar-page checks that require a store already past the `activate` step, so the calendar renders directly instead of redirecting to the hub). ### Timing table | Step | Flags after edit | Start (UTC) | Verified (UTC) | Elapsed | |---|---|---|---|---| | 1. Full-on | `SCHEDULING_FREE=true`, `ONBOARDING_INVITES_ENABLED=true`, `ONBOARDING_WALKTHROUGH_ENABLED=true` (+ baseline hub=true, pilot=ob01) | 20:01:28.628 | 20:02:59.123 | **90.5s** | | 2. Kill switch (M1-06) | `SCHEDULING_FREE=false` (others unchanged) | 20:03:04.910 | 20:03:23.329 | **18.4s** | | 3a. Hub off | `ONBOARDING_HUB_ENABLED=false` (invites/walkthrough on) | 20:03:34.479 | 20:03:58.074 | **23.6s** | | 3a-correction. Hub off, corrected URL re-check | same as 3a | 20:05:17.764 | 20:05:29.542 | 17.8s | | 3b. Invites off | `ONBOARDING_INVITES_ENABLED=false` (hub back on, walkthrough on) | 20:04:03.714 | 20:04:54.438 | **50.7s** | | 3c. Walkthrough off | `ONBOARDING_WALKTHROUGH_ENABLED=false` (hub on, invites back on) | 20:04:58.771 | 20:05:13.447 | **14.7s** | | 4. Full restore to baseline | both temp flags + `SCHEDULING_FREE` removed entirely | 20:05:35.219 | 20:05:40.308 | **5.1s** | **Total drill wall-clock (first flip → final restore verified): ~4m 12s.** Every individual flag-flip-to-verified window is comfortably sub-minute; the slowest single step (full-on, 90.5s) included five independent assertions (two page navigations, two `fetch()` probes, one billing-config probe), not flag-propagation latency — the flag itself was live on the very next request in all cases, confirmed by immediately-following curl/fetch calls returning the new behavior with no wait/retry needed. **Self-correction found and fixed during the drill:** the first pass at steps 3a/3b used an incorrect URL for the hub state endpoint (`/ob01/schedule/onboarding/state`, missing the `/api` prefix — the route is actually mounted at `/api/:typeNum/schedule/onboarding/state`, per `routes/api.php:123` → `routes/groups/onboarding-api.php`). Both initial attempts returned Slim's generic HTML 404 (route not found), which happened to coincidentally look like "the hub-gate correctly 404s" for step 3a but was **not actually exercising `OnboardingApiController`'s own gate logic**. Caught by re-running step 3b's check with the corrected URL and seeing a *200* (JSON state payload) where the wrong URL had returned 404 regardless of flag state — proof the URL, not the flag, drove the earlier result. A dedicated re-check ("3a-correction" row above) re-ran the corrected URL against `ONBOARDING_HUB_ENABLED=false` and got the real, application-level 404 envelope (`{"error":"Not Found","code":"not_found","requestId":"req_..."}`) — confirming the hub-gate 404 behavior is correct once the right endpoint is hit. No code changes were needed; this was purely a drill-script URL error, corrected within the same session. Every other endpoint check in this drill (`invite-status`, billing `config`) used routes cross-referenced directly against `routes/team-members.php`/`routes/billing/api.php` and were correct on the first attempt. ### Degradation matrix (assertions, with evidence) | Step | Assertion | Result | |---|---|---| | 1. Full-on | `ob02` (not pilot-listed, non-premium) navigating to `/admin/ob02/schedule` | **Redirects to `/admin/ob02/schedule/get-started`** (hub PITCH view) — see note below, not a literal calendar render | | 1. Full-on | `ob02` `/schedule/get-started` | Renders "Get Started with Scheduling" hub pitch (200) | | 1. Full-on | `GET /ob02/api/team-members/invite-status` | `200 {"data":[],"page":1,...}` (non-404 shape) | | 1. Full-on | `pc00` calendar page source | `window.ONBOARDING_WALKTHROUGH_ENABLED === true` | | 1. Full-on | `GET /api/billing/ob02/config` | `200`, `scheduling_invite` category present in `smsCategories` | | 2. Kill switch | `ob02` `/admin/ob02/schedule` | **Reverts to the Team+ Module marketing/upsell page** (pre-051 byte-identical rendering) | | 2. Kill switch | `ob01` (pilot) `/admin/ob01/schedule` | Still redirects to the hub (pilot entitlement independent of fleet flag) — confirms kill switch is fleet-scoped, doesn't touch the pilot allowlist | | 3a. Hub off | `ob01` `/schedule/get-started` | 302s to `/admin/ob01/schedule` — legacy calendar renders directly (byte-identical dark-deploy fallback) | | 3a. Hub off | `GET /api/ob01/schedule/onboarding/state` | `404 {"error":"Not Found","code":"not_found",...}` | | 3a. Hub off | `GET /ob01/api/team-members/invite-status` | Still `200` — invites endpoints unaffected by hub flag | | 3b. Invites off | `GET /ob01/api/team-members/invite-status` | `404 {"error":"Not Found","code":"not_found",...}` | | 3b. Invites off | `GET /api/billing/ob01/config` | `200`, `scheduling_invite` **absent** from `smsCategories` (category hidden) | | 3b. Invites off | `ob01` `/schedule/get-started` | Still renders hub pitch — hub unaffected by invites flag | | 3b. Invites off | `GET /api/ob01/schedule/onboarding/state` | Still `200` — hub API unaffected by invites flag | | 3c. Walkthrough off | `pc00` calendar page source | `window.ONBOARDING_WALKTHROUGH_ENABLED === false`; "Take the tour" trigger absent from the page (present in the step-1 screenshot, gone here) | | 3c. Walkthrough off | `GET /ob01/api/team-members/invite-status` | Still `200` — invites unaffected by walkthrough flag | | 3c. Walkthrough off | `GET /api/ob01/schedule/onboarding/state` | Still `200` — hub unaffected by walkthrough flag | | 4. Restore | `.env` byte-diff vs. pre-drill baseline | **MD5 identical** (`f96feb7644da0ce11d37e3478a7345c7`) | | 4. Restore | `ob02` `/admin/ob02/schedule` | Marketing page (not entitled) | | 4. Restore | `ob01` `/admin/ob01/schedule` | Redirects to hub pitch (still pilot-entitled) | **Note on the step-1 "ob02 renders the calendar" assertion:** the drill instructions anticipated `ob02` rendering the calendar directly once fleet-entitled. In this codebase's actual `SchedulingAccessPolicy::decidePage()` logic, an entitled store with `ONBOARDING_HUB_ENABLED=true` (which is part of this environment's *permanent* baseline, not something step 1 turns on) and `schedulingProvider='none'` redirects the calendar route to the get-started hub (`ACTION_REDIRECT_GET_STARTED` → `ACTION_HUB_PITCH`), not `ACTION_RENDER`. This is by-design (see `SchedulingAccessPolicy` docblock) and is arguably the *stronger* proof that fleet entitlement flipped — pre-flip, the same navigation showed the Team+ Module marketing/upsell wall; post-flip, it shows the internal onboarding hub instead. A store that had already completed `activate` (like `pc00`) renders the calendar directly regardless, which is what the hub-off checks (3a/4) exercise instead. No code defect — just a wording mismatch between the drill's assertion text and the current, correct route-matrix behavior; verified against `SchedulingAccessPolicy.php` directly. **Conclusion:** every M1-06 kill-switch and per-flag rollback claim in §1-§3 of this document is now empirically verified against the real running application, not just unit-tested. No gaps found in the flag-gating logic itself; the one issue found (wrong drill-script URL for the state endpoint) was a testing artifact, not a product defect, and was caught and corrected within the same drill. --- ## References - [`scheduling-free-launch.md`](./scheduling-free-launch.md) — forward launch sequence, decision-gate checklist - `docs/specs/051-scheduling-onboarding/product-requirements.md` — Release Manifest (rollback column), Decisions Log - `docs/specs/051-scheduling-onboarding/implementation-plan.md` — T8.7 (kill-switch drill), Risk Register (R1, R9) - `userfrosting/src/BuyerKiosk/Scheduling/Onboarding/SchedulingEntitlement.php` - `userfrosting/src/BuyerKiosk/TaskEngine/Jobs/InviteDeliveryJob.php`, `InviteSweepJob.php` - `userfrosting/docs/runbooks/051-privacy-retention.md`