Appearance
Streaming phase 1 of the price update — design
Date: 2026-09-03 Status: Proposal. Not implemented. Architectural → propose-then-wait (CLAUDE.md §3). Baseline: main @ 463b1f0. Trigger: alchemists-refuge scheduled price updates have OOM-looped since 2026-09-03T02:00Z. Diagnosis in §1. Scope decision: fix only. Peak-memory telemetry and fail-fast guards were considered and deliberately deferred to follow-up PRs (§7).
0. TL;DR
- Phase 1 (
fetchAllProductsWithPagination) materialises every variant in the store into aMapand the processor holds it through phases 2–4. At 764,806 variants that crosses the worker's--max-old-space-size=2560ceiling and V8 aborts. - Phase 2's batching (
VARIANT_BATCH_SIZE= 5000) already bounds the catalog and price side. It never bounded phase 1. This is the second half of the same lesson CLAUDE.md §5.6 records for Mongo reads: bounding the query is necessary but not sufficient — the read must also stream. - Fix: make phase 1 an async generator and have phase 2 consume it lazily, buffering per game so today's homogeneous 5000-sized catalog
$inlookups are preserved exactly. - Peak resident variants drop from ~765,000 to ~20,000 (4 games × 5000).
- No schema change, no client change. One merchant-visible consequence: phase 2 stops being a visible 10–15 minute stage (§4).
1. Measurements — what is actually happening
1.1 The failure
Four independent sources agree.
Worker syslog (GCE lgs-ledger-worker):
Mark-Compact (reduce) 2551.8 (2571.2) -> 2544.7 (2557.7) MB, ...
FATAL ERROR: Ineffective mark-compacts near heap limit
Allocation failed - JavaScript heap out of memory
systemd[1]: lgs-worker.service: Main process exited, code=killed, status=6/ABRTThe cap it died against — scripts/create-systemd-service.sh:97:
ExecStart=/usr/bin/node --max-old-space-size=2560 server/worker.js2547 MB against a 2560 MB cap. The worker died at its configured ceiling, not a V8 default.
sync_jobs — every failed run froze at the same point:
json
"phase2": { "status": "running",
"progress": { "variantsChecked": 590051, "totalVariants": 764806,
"stage": "mtg_preload_mongo_fetch_start" } }Sentry — MongoWaitQueueTimeoutError (LGS-LEDGER-6), first on 2026-09-03 at 04:04Z, after the 02:00Z job started. Downstream, not causal.
1.2 The growth curve
| Run (02:00Z) | Variants | Duration | Outcome |
|---|---|---|---|
| 2026-08-31 | 755,861 | 46 min | completed |
| 2026-09-01 | 755,862 | 150 min | completed |
| 2026-09-02 | 762,210 | 127 min | completed ← last success |
| 2026-09-03 | 764,806 | — | OOM loop |
The store grows ~2,500 variants/day. It did not jump; it drifted across a line.
1.3 Why the error message misleads
The chain is: OOM GC thrash stalls the event loop → the Mongo pool's waitQueueTimeoutMS fires on everything → the WorkerHeartbeat write is starved → staleJobReaper correctly reaps the job as stalled. So every failed row reads:
Interrupted: the worker running this job (lgs-ledger-worker-1537743) stopped responding.
That is accurate about the last link and useless about the cause. The reaper is working as designed — healthy long runs (150 min, 127 min) were verified never reaped (wasInterrupted: false). Do not chase it.
1.4 This is a repeat
The comment above the heap cap records the same error string hitting before, on the old e2-small, in this same code path. The fix then was to grow the box. GCE_WORKER_DEPLOYMENT.md:329 already names the real prerequisite:
Do not downsize to e2-small without first bounding peak memory in the price-update path.
Peak memory in that path was never bounded. Resizing again is the treadmill.
1.5 Blast radius
Store-specific in origin, global in effect: it is a process crash, so sync and billing on that worker die with it every ~2 hours. Other stores' price updates complete normally (hrh2kw-1q, ufkes-dev both fine throughout).
2. Where the memory goes
Held simultaneously at peak, in calculatePriceUpdates:
| Structure | Size | Bounded today? |
|---|---|---|
variantMap (priceUpdateService.js:220) | 764,806 entries × ~14 fields | no |
variantsByGame (line 419) | references to the same objects, ~590k after filtering | no |
updates (line 407) | ~36k entries (~4.7% of variants) | no, but small |
productsToActivate | draft products only | no, but small |
| per-batch catalog docs + price snapshots | VARIANT_BATCH_SIZE = 5000 | yes (the §5.6 fix) |
variantMap dominates and is referenced by the processor through phases 3–4 (priceUpdateProcessor.js:265), so it is never collectable.
variantsByGame holds references, not copies — it is an additional index structure, not a second full payload. Worth stating because it looks like a doubling in the diff and is not.
3. The change
3.1 Shape
Shopify page (50 products / ~1,250 variants)
→ yield each variant
→ append to a PER-GAME buffer
→ when a game's buffer reaches VARIANT_BATCH_SIZE (5000):
price that batch → append to updates[] → clear buffer
→ on pagination end: flush every remaining per-game buffer3.2 Per-game buffers are load-bearing
A Shopify page contains mixed games. Buffering into one shared 5000-slot buffer and splitting at flush time would produce smaller, more numerous, per-game $in lookups — changing Mongo round-trip counts and price-cache hit rates as a side effect of a memory fix.
Buffering per game and flushing a game's buffer when it reaches VARIANT_BATCH_SIZE preserves today's batch composition exactly (priceUpdateService.js:535). With the four currently registered plugins (mtg, pokemon, riftbound, lorcana — verified in server/plugins/index.js) the bound is 4 × 5000 = 20,000 buffered variants.
3.2.1 Why games × batchSize is not the bound
Games are registered at runtime via registerPlugin, and the pluggability programme's stated target is ~100 games. 100 × 5000 = 500,000 buffered variants would reinstate exactly the problem this spec removes — quietly, years later, with no code change to blame.
So the bound is an aggregate cap, not a per-game one:
MAX_BUFFERED_VARIANTS (default VARIANT_BATCH_SIZE × 4 = 20,000)
on append:
if total buffered across all games >= MAX_BUFFERED_VARIANTS:
flush the LARGEST buffer // most batch-efficient victimPer-game buffers still flush at VARIANT_BATCH_SIZE as in §3.2; the aggregate cap only bites when many games are present at once. In that case some batches are smaller than 5000 — a deliberate trade of batch efficiency for a hard memory bound, and only for stores carrying more games than today's entire catalogue.
The default preserves current behaviour exactly for any store carrying four games or fewer, which is every store today.
3.3 Interface
fetchAllProductsWithPagination becomes an async generator, renamed streamManagedVariants — the old name promises a materialised result and would be a lie after this change.
calculatePriceUpdates changes its first parameter from Map to any async-iterable of variant data. Because for await...of also consumes sync iterables, existing test fixtures that pass an array or map.values() keep working unchanged — this keeps the diff off the test suite.
It returns variantsSeen, so the processor stops reaching for variantMap.size (lines 227, 236, 265, 283 of the processor).
3.4 Resulting peak
| Before | After | |
|---|---|---|
| Buffered variants | 764,806 | ≤ MAX_BUFFERED_VARIANTS (20,000) |
updates[] | ~36,000 | ~36,000 (unchanged) |
| Per-batch catalog docs | bounded | bounded |
Peak becomes a function of MAX_BUFFERED_VARIANTS, not store size and not game count (§3.2.1). That is the property being bought; the absolute numbers will drift.
3.5 Explicitly not in this change
updates[] and productsToActivate still accumulate across the whole run for phases 3–4. At ~4.7% of variants they are not what broke, and streaming them would mean writing to Shopify before the full calculation completes — a real semantic change (partial repricing on crash) that deserves its own decision. Left alone.
4. What the merchant sees
syncStatusService picks the first running phase in an if/else chain, testing phase 1 before phase 2. Phase 1's progress is asymptotic on pageCount and needs no denominator (syncStatusService.js:103), so while pagination runs the UI is unchanged. Phase 2's counters accumulate underneath and surface once pagination completes and the final buffers flush — at which point totalVariants is finally correct.
Net effect: phase 2 goes from a visible 10–15 minute stage to a near-instant one. The bar jumps ~28% → 50%.
The per-phase detail row will read Checked 200,000 / 200,000 variants mid-run (client/src/components/PriceSyncStatus.jsx:189). That is truthful — everything fetched so far has been priced — but reads differently than today.
No schema change and no client change. If the concurrent-phase display proves confusing in practice, collapsing the two phases in the UI is a follow-up, not part of this PR.
5. Correctness details
5.1 Collection-missing retry
priceUpdateProcessor.js:213 self-heals a deleted managed collection by clearing the cache, recreating it, and restarting pagination. Today that is safe because phase 1 has produced nothing yet. Under streaming, a restart after any pricing has occurred would double-count every counter and duplicate entries in updates[].
The throw happens on the first page in practice (priceUpdateService.js:273), so this is defensive — but silent stat corruption is exactly the failure mode this repo keeps paying for. Requirement: accumulators (counters, buffers, updates, productsToActivate) are created per attempt, so a retry starts from zero.
5.2 Cancellation
Currently checked between phases. Streaming makes phases 1–2 one long stretch, so checkIfCancelled moves to once per flush — bounded work between checks, and no busier than today's per-batch cadence.
5.3 §5.11 — the aliased array
Buffers are allocated and owned internally and cleared in place. No caller-supplied array or Map is mutated. The current gameVariants.slice(offset, offset + batchSize) copy disappears along with the array it copied from.
5.4 §5.1 — game parity
This path is plugin-driven and game-agnostic; there is no per-game branch to mirror. Parity here means the buffering works for every registered game (mtg, pokemon, riftbound, lorcana) and for variants whose game is unmanaged or unknown (filtered before buffering, as today at line 419). Covered by a mixed-game stream test (§6).
5.5 §5.5 — no identity defaults
variantData.game continues to come from _detectGame(boosterGame, managedBy). No fallback is introduced at the buffering layer; an unrecognised game routes to the existing unmanaged counter rather than to a default game's buffer.
6. Testing
Co-located in server/services/priceUpdateService.test.js, Vitest, _setDeps for injection (never vi.mock of server source — CLAUDE.md §3).
- Bounded buffering. Feed a 50,000-variant stream with
variantBatchSizeinjected small. Assert the maximum simultaneously-buffered variant count never exceedsMAX_BUFFERED_VARIANTS. This is the test that would fail if someone re-materialises. 1b. Aggregate cap holds with many games (§3.2.1). Same stream spread across more games thanMAX_BUFFERED_VARIANTS / batchSizeallows; assert the cap is still respected and that the largest buffer is the one flushed. - Laziness. Assert pricing begins before the source is exhausted — a source that records how many items it has yielded when the first catalog lookup fires. Guards against a well-meaning
const all = [...stream]creeping back in. - Batch composition unchanged. A mixed mtg/pokemon/riftbound/lorcana stream produces per-game catalog lookups of the same sizes as the current group-then-batch implementation. Pins §3.2.
- Counter equivalence. Capture the current implementation's full result for a fixed mixed-game fixture before touching it, commit that as a golden object, and assert the streaming implementation reproduces it field for field —
variantsChecked,pricesMatched,noPriceData,unmanaged,priceLocked,draftProductsWithPricing, plus the contents ofupdatesandproductsToActivate. Capturing the golden first is the point; a golden written after the rewrite only pins the rewrite's own behaviour. - Retry resets accumulators (§5.1): a source that throws
MANAGED_COLLECTION_MISSINGafter N variants, then succeeds, yields counters for the successful attempt only. - Cancellation mid-stream (§5.2) stops within one flush.
Coverage: the two modified files must each report ≥70% on all four metrics. Read the per-file table — the Vitest 4 threshold gate is silently inoperative (CLAUDE.md §7).
6.1 Verification before re-enabling the merchant
Unit tests cannot prove the heap ceiling is cleared. Before the paused schedule is turned back on for alchemists-refuge, run one manual price update against the real store with the worker's RSS watched, and confirm it completes. That store is the only known instance at this scale, so it is the only meaningful test.
7. Deferred, deliberately
- Peak-memory telemetry — recording peak heap and max buffered variants per run would turn "invisible until OOM" into a number to watch. Wanted; separate PR.
- Fail-fast guard — an explicit ceiling that fails the job with a named limit rather than a V8 abort. Wanted; separate PR. A too-tight bound fails runs that would have succeeded, so it needs its own thought.
- Streaming phases 3–4 (§3.5).
- Stalled-job alerting — a job that stalls and retries reports
recentFailed: 0and never trips the queue-depth alert, which is why this ran for 13 hours unnoticed. Arguably the most valuable follow-up.
8. Adjacent lever — moving Redis off the worker VM
Raised during design; independent of this spec and not a substitute for it.
Redis is currently self-hosted on the same 4 GB e2-medium as the worker. That is what forces the coupling documented at scripts/create-systemd-service.sh:97:
Raise both together or neither — a heap larger than the box just trades the V8 abort for a kernel oom-kill.
Moving Redis to a managed service (Upstash) frees its resident memory and would let --max-old-space-size rise safely — worthwhile on its own, and it removes a single-box failure mode where an OOM-kill could take the queue's backing store with it.
It does not remove the need for this change. Unbounded is unbounded: at ~2,500 variants/day the store re-crosses any fixed ceiling, and the next store to reach this size starts the cycle again. Headroom buys time; bounding the path is the fix.
If it proceeds, three hazards apply and each has burned this repo before:
- §5.7 (Split Brain) —
REDIS_URLlives in both the live Cloud Run env and a GitHub Actions secret consumed bydeploy-api.yml. Update one and the next deploy reverts it: API enqueues to one Redis, worker listens on another, jobs strand invisibly. Both must change in the same sitting, worker unit included. - §5.13 (Shared Redis Pool) — the
mainpool carries BullMQQueue.add(),QueueEvents(blockingXREAD), andpriceCacheService's bulkmget/pipeline over thousands of keys. A managed provider's per-connection limits, command timeouts and TLS handshake behaviour land on all three profiles at once, andQueueEventsthrows at startup ifmaxRetriesPerRequestis bounded. - Bulk command cost —
priceCacheServiceissuesmget/pipelines over ~7.5k keys per run. On a hosted, per-command-priced, network-latency-bound Redis that is a different cost and latency profile than a unix-local instance. Measure before committing.
Recommendation: sequence it after this fix, so the two changes can be told apart if either regresses. It deserves its own spec.
9. Definition of done
- [ ]
npm testexits 0;npm run lintexits 0 with no new warnings. - [ ] Per-file coverage ≥70% on both modified files, read from the table.
- [ ] Bounded-buffering and laziness tests present and failing against the pre-change implementation.
- [ ] Counter equivalence verified against the current implementation.
- [ ] No client change;
priceUpdateStatsshape unchanged. - [ ] One manual run against
alchemists-refugecompletes with RSS observed below the cap (§6.1), before the schedule is re-enabled. - [ ] Commit message states the merchant-visible outcome.