LGS Forge

Appearance

Empty Queue — Design ​

Date: 2026-07-22 Status: Approved, pending implementation

Problem ​

A merchant who queues many sync jobs (e.g. selects a large batch of sets, or accumulates a backlog behind a variant-daily-limit throttle) has no way to back out. Today the only per-job control is DELETE /api/sync/:jobId/terminate (server/routes/api.js:2859), one job at a time, and it marks jobs failed — which misrepresents jobs that never started. There is no "clear everything that hasn't run yet" operation.

Decision ​

Add a one-click "Empty Queue" action that cancels every not-yet-started sync job for the shop, leaves in-flight and finished jobs untouched, and marks the cleared records cancelled (not failed).

Scope of "empty" ​

SyncJob records for req.shop with status ∈ {queued, pending}.

  • queued is the status every enqueue site writes (server/routes/api.js:2329, :2609, :3469; server/queues/processors/syncProcessor.js:361).
  • pending is in the status enum (server/models/SyncJob.js:38) but no code path ever writes it as a top-level status (the 'pending' literals elsewhere are pipeline-phase statuses and other models' fields). Included for robustness; tests should not assume it occurs in the wild.
  • throttled is explicitly excluded. It is a terminal state, not the backlog: when the variant daily cap hits, the processor marks the original job throttled with completedAt set and its partial success counts preserved, and creates a fresh SyncJob with status: 'queued' plus a delayed BullMQ job keyed by the new record's id (server/queues/processors/syncProcessor.js:356-388). The backlog the merchant wants gone is that queued resume record — already covered by the {queued, pending} scope, with BullMQ state delayed → removable. Flipping the throttled parent to cancelled would rewrite a completed, partially-successful history record and remove nothing from any queue.
  • In-flight states (running, validating) and terminal states (completed, failed, cancelled, throttled) are untouched. This honors "leave the running job alone."
  • sealed_add records are written already-completed (server/models/SyncJob.js:11-14) and are naturally excluded by the status filter.

Known cosmetic trade-off: after clearing, a throttled parent record still carries resumeScheduledFor — a resume promise the merchant just cancelled. We deliberately leave that record untouched; the adjacent cancelled resume row in sync history tells the story. Revisit only if merchants report confusion.

Why cancelled, not failed ​

cancelled is already load-bearing in the processor, which gives the clear operation a free safety net if BullMQ removal fails or races:

  • A job found already-cancelled at pickup is skipped without doing work (server/queues/processors/syncProcessor.js:51).
  • The failure path preserves cancelled rather than overwriting with failed (server/queues/processors/syncProcessor.js:490).

The existing terminate endpoint's use of failed conflates "kill a running job" with cancellation; for jobs that never started, cancelled is the honest status.

Server ​

syncService.clearQueuedJobs(shop) ​

  1. Find SyncJobs for shop with status ∈ {queued, pending}.
  2. Per job, resolve its BullMQ job — jobs are keyed by the SyncJob's Mongo _id (getJob(syncJobId), same resolution the terminate endpoint uses at server/routes/api.js:2869-2871) — in the queue matching its jobType: syncQueue for product_sync, priceUpdateQueue for price_update.
  3. Order of operations (race safety): check BullMQ state first.
    • active → skip entirely: no removal, no Mongo write. An unconditional cancelled write would trip checkIfCancelled mid-run (server/queues/processors/syncProcessor.js:27-32) and abort a running job, violating the design's own constraint.
    • waiting / delayed / paused → remove(), wrapped in try/catch (best-effort, matching the terminate endpoint's pattern).
  4. Mark the record with a status-guarded atomic update so a job the worker picked up between the find and the write is never flipped. The filter also carries shop — redundant with the shop-scoped find, but every write to a per-store collection stays tenant-guarded on its own:
    js
    SyncJob.updateOne(
  { _id: job._id, shop, status: { $in: ['queued', 'pending'] } },
  { $set: { status: 'cancelled', completedAt: new Date(),
            lastError: 'Cleared by merchant (queue emptied)' } }
)
    The residual window (BullMQ active check passed, worker grabs the job before the update lands, Mongo status still queued) is benign: the processor's already-cancelled check at pickup usually discards it before any work happens. In the narrowest double-race — the guarded update lands after that pickup check but before the processor saves running — the processor's own status save wins and the job runs to completion; the only artifact is a cleared count off by one and a stale lastError on a record that ends completed. No handling warranted.
  5. Return { cleared, skippedRunning } (counts).

No per-job notifications. The terminate endpoint sends a failure notification per job (server/routes/api.js:2902); a bulk clear must not — one merchant click would otherwise spam a Slack message per cleared job. This deviation is deliberate.

Repeatable price-update schedule is safe by construction ​

Queued price_update SyncJobs only come from the manual run-now endpoint (server/routes/api.js:3466-3477), which pre-creates the record and keys the BullMQ job by its id. Scheduled repeatable price jobs have no SyncJob until the processor creates one at run time, so per-record resolution can never touch — and never accidentally deregister — the repeatable schedule. Do not "simplify" this into draining priceUpdateQueue wholesale; that would break repeatable scheduling.

POST /api/sync/empty-queue ​

  • Calls syncService.clearQueuedJobs(req.shop), returns { ok: true, cleared, skippedRunning }.
  • No request body — shop comes from auth. No Zod body schema, consistent with DELETE /api/products (bodyless, shop-scoped). Route still sits behind the standard auth middleware chain.

Client ​

EmptyQueueButton.jsx (new) ​

  • retroui Button + confirm Dialog: "Clear all queued syncs that haven't started yet? Running jobs finish normally."
  • On confirm → api.post('/api/sync/empty-queue') via utils/api.js.
  • Shows result: "Cleared N queued jobs" / "Queue is already empty" (cleared === 0).
  • Fires onCleared callback on success.

Queue.jsx ​

  • Holds a refreshKey; renders <EmptyQueueButton onCleared={bump} /> in the header and passes key={refreshKey} to SyncHistory so the list refreshes after clearing.

The confirm dialog is UX only — the server is the sole enforcement point (CLAUDE.md rule 7); there is no client-side eligibility filtering to drift out of sync.

Game parity (§5.1) ​

Game-agnostic: keys on shop + status, touches no plugin, per-game model, or importer. mtg / pokemon / riftbound are covered identically; nothing to mirror. game is never read, defaulted, or branched on (§5.5 clean).

Edge cases ​

  • Empty result: no matching jobs → { ok: true, cleared: 0, skippedRunning: 0 }; client shows "Queue is already empty."
  • BullMQ job missing (already consumed, or Redis flushed): Mongo record is still marked cancelled — same "not found in queue, still mark in DB" behavior as terminate (server/routes/api.js:2889-2891).
  • Job goes active mid-clear: skipped, counted in skippedRunning.
  • remove() throws: swallowed per-job; the status-guarded update still runs, and the processor's cancelled-at-pickup check discards the orphaned BullMQ job.
  • Throttled backlog: the queued resume record is cleared like any other queued job; its throttled parent is untouched (see Scope).

Testing ​

  • Service (syncService.test.js, via _setDeps() injection):
    • queued job with waiting BullMQ state → removed and marked cancelled, counted in cleared.
    • queued resume job with delayed BullMQ state → removed and cancelled (the throttle-backlog case).
    • job whose BullMQ state is active → not removed, no status write, counted in skippedRunning.
    • running / completed / throttled / failed records → never selected, never written.
    • price_update queued record → resolved against priceUpdateQueue, not syncQueue.
    • remove() throwing → record still marked cancelled.
    • Jobs belonging to a different shop → untouched.
  • Route: no dedicated route test — server/routes/api.js has no test harness in this repo (only the small standalone routers webhooks.js and publicBuylist.js are route-tested), and the handler is a three-line wrapper over the fully-tested service, mounted after dualModeAuth like every other authenticated route. Deliberate deviation, consistent with every other endpoint in api.js.
  • Client (EmptyQueueButton.test.jsx): dialog opens on click; confirm posts to the endpoint; onCleared fires on success; zero-cleared response renders "Queue is already empty."