The escalation arrived as a refund threat. A customer had run a campaign, and thousands of people scanned its QR codes that day expecting to reach the sales team. One salesperson on the code had already hit his daily quota — the system showed him offline for assignment — and every new scan still connected to him. The rotation to the next salesperson never fired. The campaign closed far below target, the customer demanded a refund, then churned: a six-figure annual contract. The same mis-assignment had been producing quieter support tickets for months. The logic was correct; the member list it computed from was not.
A live QR code is printed once and keeps working while the people behind it change. The customer scans in WeChat, WeCom routes to whoever the code currently lists, and we maintain that list. At 30,000 tenants with 10 to 20 staff on a QR code, most of them stayed idle until a campaign drove traffic to them for days or weeks.
I owned this system — took it over as it stood. The tickets fell into two classes with one cause: staff shown offline still receiving scans, staff over quota still receiving scans.
The routing was never ours
WeCom owns the part that connects people. Tencent's enterprise messenger matches each scan to a contact-way configuration on its side and opens the chat with whoever that config lists. Our platform never sees the scan that opens the chat. So assignment, for us, meant one thing: maintaining that config. Deciding which staff IDs belong in it, and pushing every change through WeCom's API.
Two stores had to agree. Our staff state — online, on duty, under quota, per tenant — lived in MongoDB. WeCom's config decided where customers actually went. When the two disagreed, nothing raised an error; production simply kept routing by the stale side, and scans landed on someone we had marked offline an hour earlier. Repairing it was not free either: the fix itself was a rate-limited API call with its own failure modes.
The QR codes came in two families. Native codes handed member selection to WeCom's random assignment, and our only job was who appeared in the config. Sequenced and quota codes ran on our own landing page, so we decided the order and the per-person volume as well. The offline and quota checks mattered most on these codes: a rotation that never happens is a list that never got updated.
Two years of features, six writers
Before I took over, feature work had put this state together piece by piece:
- State split across many collections. Online/offline flags (an admin
endpoint, work-schedule timers), the per-staff-per-code ceiling records,
the "already removed today" marker, and the counts that decided who was
full — all in MongoDB, incremented by
$incon each scan callback. Four switches per code — offline check, add limit, daily limit, round limit — plus a company-wide daily total decided how those numbers were applied. - Many paths wrote it. The scan callback, the admin offline endpoint, the quota counters, work-hours jobs, resignation handlers, org-change events. Each pushed its own update toward WeCom when it decided the list should change: inline, with its own retry, at its own moment.
- One slice already had a queue. Sequenced and quota codes used a Redis ZSET with a two-level cursor, built by a teammate in 2021. Staff state-change events went through Kafka to a consumer that adjusted the set accordingly — full recompute on edits and shift changes, add a returning member, delete one who left or hit quota. Scans read the set through Lua. But every one of those paths went through the same consumer, asynchronously, and a full rebuild meant deleting and recreating the whole key. Between "staff hit quota" and "set rebuilt without him", scans kept drawing the old membership.
Each path was sensible on its own; the bug was in none of them. The same state had many concurrent writers, and each writer pushed a list computed from a partial view.
Quiet days hid it: with little concurrent traffic, the writers rarely interleaved badly enough to lose a change. Campaign day removed that safety margin — scan volume and status changes peaked in the same minutes, exactly when the gap between "state says stop" and "WeCom config says go" was widest.
Both failure modes had been reaching support for months, as rare tickets. Each was reproducible only under load, so each got chased as a possible logic bug and closed when no code defect turned up.
Three fixes that should have worked
Once the defect was named — state and routing config drifting apart — three fixes suggested themselves. Each one fails on a structural fact.
Check at scan time. Intercept the scan, read staff state, route around anyone offline or over quota. For native codes this is impossible in principle — the scan never touches our platform. Sequenced and quota codes do pass through our landing page, so the check was possible there — but it failed twice over. It put a database read on every scan of the hottest path we had. Worse, it read the same state the concurrent writers were updating, so the check itself could see a stale view and pass the person it was meant to skip.
Sweep and repair. A periodic job that walks every QR code, compares our state against WeCom's config, and pushes a repair where they disagree. As a safety net this is the right shape, and it stayed on the list for later. As the primary fix it fails on arithmetic. WeCom documents a ceiling of 10,000 calls a minute and 150,000 an hour per enterprise for a single API, and the 6M codes sit concentrated with heavy users: working through one heavy tenant's codes at that ceiling takes tens of hours, in a window shared with the live pushes that keep arriving. This failure appears in minutes and has to be corrected in seconds; a sweep of tens of hours can only fix the slow kind of drift.
Better retries on every writer. Make every write path idempotent, add exponential backoff, let each writer retry until it lands. This is the status quo with more machinery. A retry makes one write more likely to succeed; it says nothing about two writers pushing lists computed from different snapshots — last write wins, and the winner's list is already stale. The defect was order, and retries do not order anything.
All three patched where the symptom surfaced; the alternative was to remove the multiple writers.
One writer, recomputing
I put one Kafka queue between "state changed" and "WeCom config changed". Every event that can alter a QR code's membership goes to one topic, keyed by the QR code's own ID: an addition, a status flip, a schedule transition, a resignation, an org change, a config edit. A consumer group works through them. For each affected code it recomputes the eligible members from current state, using the same four switches each old writer had checked on its own. It pushes that update to WeCom, and only then writes the result back to our store. WeCom is where routing happens, so it gets the correct list first. And the store is what the dedupe check reads — the record of the last list successfully pushed — so it must only ever reflect completed pushes. Write the store first and the two writes can be split by a failed call: the record then claims a list WeCom never received, the redelivered event recomputes the same list, the dedupe check matches it against the lying record, skips the call, and the divergence is permanent. The store is written only after the push succeeds, so the dedupe record is never ahead of what WeCom actually holds.
The state has to move before the event does. The counter that decides
who is full is a MongoDB document per staff member and code, incremented
by $inc. The consumer handling WeCom's customer-added
callback increments it while it handles the addition, before it
publishes the refresh. By the time the refresh runs, the count is
current, and the event carries no counter value — the consumer reads the
count itself. Every other trigger follows the same rule: write the flag
or the counter where the business event happens, then emit the refresh.
Concurrency did not disappear — each handler still owns its own fact,
and different facts never conflict. What the queue removed is the thing
the old writers raced on: concurrent composition of the same derived
list. The queue carries a reason to recompute, not the state to
apply.
Sequenced and quota codes still rotate on Redis, through the same ZSET and two-level cursor, but the two families now share one writer. For either family, the refresh consumer recomputes from the counts and flags as they stand and writes both routing targets — the config WeCom reads and the set our landing page draws from — out of one computation, in one serial order. Before, each target had its own writer at its own moment — two writers for one code's membership, which was the race.
Serialization comes from the key. Events are keyed by the QR code's ID, and the consumer routes each key to a fixed goroutine, so one code's events apply one at a time while codes proceed in parallel. What needs ordering is a single code's member list, and nothing else shares it. The shared eventbus client we extended supplies that per-key mode, along with the retry path below.
The quota loop closes on the same queue, and it is where the load comes from. A scan callback increments a staff member's counter past the ceiling. The handler marks the ceiling document with an idempotent "already removed today" flag, notifies the admin, looks up every code that staff member sits on, and emits one membership event per QR code. Dozens of QR codes, sometimes more than a hundred. One removal becomes that many WeCom pushes, and campaign day concentrates both the removals and the QR codes.
What the constraints forced
Rate pressure and business freshness are two problems. The design separates their cadence, not their budget — both lanes still draw on one rate window.
The lane a code runs in is a business decision: the tenant's tier, how fresh that code's membership has to be, an operator's switch. I kept that decision a pure function evaluated per event — the code's creation and update age, a tenant whitelist — with no lane state stored anywhere. Every event starts on the fast topic; when the function says slow, the consumer republishes it to the slow topic without a call. Producers never judge lanes; the decision lives in one place, the serial consumer. A stateless call survives rebalances by construction, and a rule change takes effect on the next event.
The cost of statelessness is a transition window. When policy flips a code from slow to fast, one event may still be queued on the slow topic while the next runs fast — briefly two in-flight workers for one code. I accepted that: both lanes run the same single-writer computation, the consumer takes a per-code lock in Redis around each refresh, and whichever push lands last was computed from the latest state. The next business event on that code recomputes from current state on the correct lane, so the divergence is transient by design — eventual consistency, bounded by how often the code's own traffic produces events.
The vendor's rate limit is handled optimistically, on the call path. Before calling WeCom, the handler makes two checks. The first is dedupe: compare the list just recomputed with the last list pushed for that code, and if they match there is nothing to send. That does not shrink the fan-out; ten QR codes with ten different lists are ten real pushes. It removes the no-ops — a retry, an edit that resolves to what is already there.
The second check is the tenant's pre-limiter: a token bucket in Redis, spent through a Lua script so every instance counts against one window. With a token, push. Without one, the handler does not wait and does not hold the message open while the window refills. It acks as not-done and requeues the event onto a retry topic with a delay — the pause is explicit, not a busy wait — after which it re-enters under the same key, behind whatever that code's stream processed, through the same two checks. No second writer, and no consumer goroutine parked mid-message.
Slow is a topic of its own, not a branch inside a handler: the slow lane enforces its interval by sleeping between a code's pushes — a deliberate down-throttle, not a message held open — and its workers are separate from the fast lane's, so a sleeping slow worker parks only that code's slow stream and never delays the same code's fast events. The interval, seconds by policy, is the business decision, not the limiter. Both lanes draw on the same tenant's rate window — ordering is per code, the vendor's quota is not — so a code on slow competes with its tenant's fast work for the vendor's limit. It draws fewer attempts from that shared window.
A stateless lane call also means a code can linger on slow after a burst — extra delay, not divergence, on codes whose membership is not changing. I took that staleness over maintaining per-code lane state.
Failures do not hold the code's stream. A failed call is reported as not-done and redelivered by the eventbus under the same key, instead of being left unacknowledged while the consumer backs off. Holding the offset would be the obvious choice and the wrong one even at code granularity. A failing call in a retry storm pins its partition, and the codes sharing that partition — a whole tenant's worth, under a busy campaign — wait behind it. Ack and redeliver keeps the blast radius at one code. WeCom's errors are specific enough to act on, too: a QR code deleted on their side returns a distinct error, which the consumer treats as a cleanup trigger instead of a retry-forever case.
Requeueing gives up publish order; dropping gives up the message after a day. Neither matters: handlers recompute at processing time, so a late trigger reaches the same current truth, and a dropped event loses nothing that the next event would not recompute anyway. The design needs one writer per code. Arrival order was the part we could give up. The residual is a QR code no later event touches — staff went offline during a multi-hour backlog, nothing followed, the stale list stays. Campaign QR codes are never idle that long, which is why the rule held in practice: an assumption about traffic, not a guarantee about state.
What it cost, and what it bought
Migration ran as double-write through the second half of 2022 and cut over in early 2023. The new event chain went live while the old synchronous paths kept running; for weeks a diff job compared the member lists the two paths pushed and alerted on any mismatch — it raised none. A config switch then retired the old paths. Tenant traffic did not notice.
Freshness became bounded. Seconds on the fast lane, the slow lane's cadence, or the next redelivery after a throttled push, never the instant of an inline write. After the new chain went live: p95 from event to WeCom's config under five seconds, measured from event publish to push completion in the metrics pipeline, and six consecutive months without one ticket for either failure mode, against at least three a month before.
WeCom's API became the explicit throughput ceiling. Under the multi-writer design nobody owned the rate limit; throttling showed up as scattered errors after the fact. One consumer made the ceiling visible and manageable, with per-tenant rate limiting, redelivery on throttle, and robot alerts on throttle responses, and turned a recurring incident source into a scheduling parameter. The system held it: around 30,000 tenants, around 6M live QR codes among heavy users, several hundred scans per second in short campaign spikes, peak-day volume under a million.
One dependency more than before. Kafka availability now gates membership updates. The failure mode errs toward safety: if the queue stalls, updates are delayed rather than misapplied, and scans run against the last consistent list. The real cost is operational. A second system to monitor, plus poison messages and lag, a class of bug MongoDB-only code never had.
The churned account came back. After running on the refactored system for a while, they re-subscribed. A customer who has lost technical trust does not return for a discount.
These are production numbers, not a load test, and the complaint record is the proof. I leave the incident account's campaign loss unquantified on purpose: it is the customer's data, not ours to publish. The churn and the re-subscription carry that part of the argument.
What I would keep, and what I left open
I would keep the central decision: one serialized queue, keyed by the QR code, applying to WeCom first and our store second.
The production surprise was that serialization was the easy part; living with WeCom's rate limits was the long tail. If I designed it again, I would budget the dependency's shape too: the rate window and the drain rate should come from the fan-out arithmetic — how many live QR codes a member sits on — not the event rate, which understates calls by that factor. Coalescing is half-built: a Redis marker drops a new refresh event while one for the same code is still queued; the missing half is merging events at push time, since the push replaces the whole list.
On Pulsar I would try collapsing the lanes into subscription semantics: one topic with a Key_Shared subscription keyed by the QR code ID. The broker routes each key to a single consumer, so the fast/slow split, the separate worker sets, the per-code lock, and the hand-rolled retry topic all disappear — a demotion or a throttle becomes a negative ack with a redelivery delay, the same ack-as-not-done pattern we had built by hand on Kafka. A redelivered message can come back out of order, which is the arrival order the design had already given up.
Reconciliation is the part I never closed. The rejected sweep had the need right; its shape was the problem. The queue guarantees a fresh read, not a correct outcome — a bug in the recompute gets applied just as consistently. But 6M QR codes at the documented rate limits is tens of hours of work to protect a handful of suspects. The backstop should be a dirty set — codes whose events were dropped, whose pushes failed permanently, or whose recomputes errored — swept at low frequency. We had no such layer; the evidence was customer complaints, weaker than a system this size should run on.
One boundary is permanent: assignment lives in WeCom's config, and nothing inside our platform makes that system synchronous with us — the design only keeps the delay across it bounded and visible.
Previous in this series: Running a Million-DAU Casual Game on Go: Rooms as the Concurrency Boundary. Next in this series: Debugging a Sporadic Go OOM in a Recursive JSON Encode Path.