Four Go engineers had to build the data platform alongside featuredelivery. We could not operate a separate OLAP database, sync pipeline,and dedicated data engineering function, so we built TiCDC →Kafka → Apache Flink → analytical tables inTiDB, served by QueryHub, a self-developed SQL-template service.

Context and requirements

The backend foundation came from ScalingFour Engineers to Eighteen Go Services with Generated Contracts.From June through November 2025, I initiated and ledthe architecture and rollout for the CDC path, warehouse layering, andquery serving in a separate data-platform workstream. ConfigHub was partof that workstream.

Product managers needed transaction, settlement, and promotionmetrics. Operations needed to detect abuse while a campaign was stillactive. Finance needed reconciliation. The production schemas wereoptimized for writes, not analytical reads.

The daily cost appeared in three ways: new metrics required ad-hocproduction queries or a new ETL pipeline; metrics varied by dimension,granularity, and refresh requirement; and T+1 reporting was too slow foran active promotion-abuse investigation.

The product had a fast feature cadence, correctness-sensitivetransaction flows, and live-ops reports that changed every sprint. Westandardized on TiDB and required:

• Hourly aggregation of transaction, settlement, promotion,referral/commission, and wallet-transfer events.
• Multi-stream joins and dimension-table lookups for detail andwide-table analysis.
• Logical ODS → DWD → DWS layering with explicit ownership.
• Analytics tables updated within seconds rather than waiting for T+1reports.
• Daily and monthly report boundaries in the viewer's timezone.
• Stable pipeline definitions deployed through the existing K8s andCI/CD path.
• Recoverability from TiCDC replay, Flink restarts, and batchcorrections without duplicating aggregate rows.

Options Considered

A custom streaming pipeline could keep business logic close to theservices and avoid adopting another framework. We rejected it because itwould have made the team own checkpoint/recovery behavior, statemigration, event-time ordering, duplicate suppression, scaling,backpressure, and latency tuning. Flink did not remove those concerns,but it provided tested primitives for them. Sink correctness remainedour responsibility.

Architecture / Design

The platform had two table groups—selected production tables andderived analytical tables—and both were queryable through TiDB. Selectedproduction tables were stored in TiKV, with TiFlashmaintaining asynchronous columnar Raft Learner replicas. For derivedmetrics, TiCDC sent selected changes to Kafka; Flink aggregated them andwrote ads_* tables to TiDB over MySQL/JDBC; TiKV persistedthe rows and TiFlash maintained the serving replicas.

For the direct replicas, TiKV → TiFlash was a storage-engineoperation rather than an external ETL job. One command enabledthe replica:

1

ALTER TABLE <source_table> SET TIFLASH REPLICA 1;

TiFlash then bootstrapped existing rows through a Raft Learnersnapshot and applied subsequent Raft updates. We did not write aseparate full-load/backfill job, store high-watermark offsets, or run anincremental sync process for those replicas. This did not eliminateoperational work: replica progress, storage headroom, IOPS, scans, andcompaction became part of the TiDB capacity model. The TiCDC → Kafka →Flink path remained conventional ETL because it transformed selectedevents into new ads_* aggregates.

Logically, the warehouse followed ODS → DWD → DWS, but ODS was not acopied store. The selected business tables were the ODS layer; TiCDCread them in place. Flink joined event streams and dimensions intodetail/wide tables for DWD, while SQL jobs materialized hourlyaggregates into ads_* DWS tables. Because all layers livedin TiDB, the layering set naming, ownership, and refresh rules; it didnot require a separate database.

The captured sources covered transaction settlements, order amounts,referral/commission ledgers, promotion records, and wallettransfers.

Key Technical Decisions

SQL-first processing

SQL jobs produced hourly aggregate rows keyed by bucket time andbusiness dimensions. A separate DataStream JAR handled workloads thatneeded custom state or event cleaning: it owned user-session statedriven by device and transaction events, and cleansedstandard-operating-procedure (SOP) lifecycle-management data. This keptmost metrics reviewable while isolating timers, keyed state, andchange-only output.

Idempotent hourly aggregates

Each hourly aggregate table used the bucket start timestamp plusbusiness dimensions as its primary key. Flink JDBC sinks upserted thosekeys, so early fires, checkpoint restarts, or batch reruns replaced thesame row instead of creating duplicates.

1
2
3
4
5
6
7
8
9
10
11

-- Simplified Flink sink shape; several hourly tables followed this pattern.
CREATE TABLE ads_user_transfer_source_symbol_hourly (
    created_at BIGINT NOT NULL,      -- UTC millisecond bucket start
    source     STRING NOT NULL,
    symbol     STRING NOT NULL,
    amount     DECIMAL(38, 18),
    amount_usd DECIMAL(38, 18),
    PRIMARY KEY (created_at, source, symbol) NOT ENFORCED
) WITH (
    'connector' = 'jdbc'
);

Airflow scheduled the H+1/T+1 offline jobs used for reconciliation.The batch lane handled historical backfill and selected incrementalrecovery after a pipeline fault; jobs reread a selected time range andrecalculated the same keys. Streaming and batch paths therefore sharedone serving table; that avoided a separate reconciliation target butincreased contention on the TiDB cluster and required closer attentionto TiFlash sync capacity.

QueryHub template service

I designed and built QueryHub, the SQL-template service. It storednamed SQL templates as versioned configuration. A template carried itsdata source, SQL, parameters, cache policy, status, and version. Queryexecution became a configuration change rather than a new Go handler,route, DTO, and deployment. Reviewed SQL seeds were still required, anda new visualization could still require frontend wiring.

Redis keys included the template name, sorted-parameter hash, andcache-group version. Hot queries reused results while related templatescould be invalidated together. Reviewed templates exposed metricdefinitions, parameters, cache invalidation, and access asconfiguration.

Timezone-aware reporting

Streaming jobs wrote UTC millisecond buckets. QueryHub templatesaccepted a :time_zone parameter, converted reportboundaries to UTC, and grouped stored buckets into viewer-local dates ormonths. Templates generated date/month series, left joined aggregatesonto them, and filled inactive periods with zero. The same physical rowssupported different local reporting boundaries and consistent MoM/YoYcomparisons.

Implementation

The platform spanned two Kubernetes clusters. TiDB, PD, TiKV, andTiFlash ran in a dedicated TiDB K8s cluster, while TiCDC captured theirchanges into Kafka. Flink and QueryHub ran in the business K8s cluster,in different namespaces. That boundary separated database workloads frombusiness-service deployments, but it did not isolate the analytical pathfrom production TiDB capacity: TiCDC read source tables, Flink andreconciliation wrote derived tables, and TiFlash maintained replicas andserved scans on the same TiDB cluster.

Each metric followed the same rollout path: Flink job,ads_* table, optional TiFlash replica, QueryHub template,and frontend chart mapping.

For dashboard queries, the admin frontend called QueryHub'snamed-query API through the gateway. QueryHub returned generic tabularrows, which ECharts rendered in the existing React admin.

When TiFlash Freshness FellBehind

The first production problem was TiFlash storage sizing. Its datavolumes initially used cloud-volume defaults, which left too littleheadroom for the replica's concurrent workloads. During peak periods,TiDB monitoring raised alerts for replica synchronization lag andrelated freshness metrics; analytical tables also looked stale. Writes,replica maintenance, and analytical reads competed for the smallper-volume IOPS budget.

TiKV had its own floors: SSD storage (preferably NVMe/PCIe), sinceTiDB writes became Raft-replicated row storage that TiCDC read, andIOPS/throughput headroom for TiKV's RocksDB LSM (Log-StructuredMerge-tree) path — queueing there raised write and Raft latency evenwithout TiFlash contention. The incident surfaced on TiFlash firstbecause columnar replicas added a second storage workload with scans,merge/compaction, and synchronization traffic.

Each TiFlash data volume moved from the cloud-volume default to40,000 provisioned IOPS. With two TiFlash nodes, the layer had 80,000IOPS in aggregate, but each volume still had an independent budgetrather than a shared cluster-wide pool. After the change, thepeak-period alerts stopped, TiFlash replicas kept up with the streamingpath, and second-level freshness became dependable in normaloperation.

This choice also changed cost. The TiFlash layer added dedicatednodes and higher per-volume provisioned IOPS, so each analytical replicaincreased baseline storage and IOPS cost as well as query load. We paidfor that choice with more storage headroom and provisioned IOPS ratherthan operating a separate OLAP service.

Hardware floors alone were insufficient for sizing TiFlash. I wouldsize it separately from TiDB HA node floors, using changed-row volume,replica sync traffic, scan shape, compaction pressure, and freshnesstargets.

TiFlash also added an operational checklist. Creating anads_* table did not automatically enable its TiFlashreplica; we ran the replica DDL separately and verified that the tablewas queryable before calling the dashboard production-ready.

Results

The platform moved dashboards from ad-hoc production queries to arepeatable pipeline. TiCDC change volume peaked above 1,000 changed rowsper second; that figure was the capacity reference for Kafkapartitioning, Flink parallelism, and TiFlash sizing. Online Grafanadashboards also recorded roughly 10–20 million transaction-related gRPCcalls per day, with peaks of at least 500 calls per second, but thosenumbers measured request pressure rather than TiCDC rows or inserts. TheFlink jobs covered transaction/settlement, promotion, commission-ledger,wallet-transfer, campaign, and online/session data; the Java DataStreamJAR added SOP lifecycle-management cleansing.

Trade-offs

Analytics shared the production TiDB cluster's capacity andoperational path, coupling analytical pressure to productionperformance. QueryHub reduced service-code work, but metric review andfrontend wiring remained necessary.

I would keep TiCDC → Kafka → Flink and the idempotent TiDB sinks, butsize the shared cluster for analytical work earlier. Managing productionand analytical tables together saved staffing effort; the cost was thatan analytical freshness incident became a shared TiDB capacityproblem.

Before changing storage, I would keep dashboards on analytical tablesonly, set QueryHub query limits and timeouts, and isolate batchreconciliation windows. The useful alerts group by path: pipeline lagcovers TiCDC lag, Kafka partition skew, and Flink checkpoint duration;the write path covers JDBC upsert rate and TiKV write latency; theserving engine covers TiFlash replica lag, disk IOPS and latency, andcompaction activity. Those groups would localize pressure to capture,processing, writes, or serving rather than only announcing that theshared cluster was busy.

If analytical scans or production-latency risk kept growing, I wouldmove analytical detail and ads_* serving out of theproduction TiDB cluster while preserving the pipeline. ClickHouse couldsuit sustained detail scans and long retention, but it would changecorrection semantics: the current correction path relies on keyed TiDBupserts, while ClickHouse would require explicit ordering,merge/deduplication, and correction behavior. I would consider thatmigration when sustained scan cost, retention, and query concurrencyjustified it.

Previous in this series: ScalingFour Engineers to Eighteen Go Services with GeneratedContracts.

Next in this series: Buildinga Predicate-Driven Audience Pipeline for Lifecycle Email.

The service ran across twelve pods, supporting SOP workflows for morethan 30,000 enterprise customers. It consumed close to ten millionevents per day and supported daily outreach to hundreds of thousands ofend users. The alerts kept coming for over a week: on most days a burst— one pod lost, then another within minutes as if the failure movedbetween them — and three or four on the worst days, until each burstexhausted itself and went quiet. Nothing in the standard instrumentscaptured why.

Events moved through this service in both directions: it consumedupstream events through the SDK's envelope decoder, and published itsown events back through the same SDK's json package. Eachevent type had its own code on the wire and its own handler.

This was the second memory incident on that path in a short window,and the two looked nothing alike. The first was a slow climb: memoryrose with business peaks and eventually reached the limit. I had alreadyfixed that one: on March 7 I moved the JSON package behind threeinternal SDK paths — the event SDK's shared json, a cache,and an executor SDK — from encoding/json to jsoniter. At our consumerate, stdlib's unmarshal allocated heavily per event; jsoniter reusedits read buffers, memory fell back and stayed down. Because it was onepackage-level swap, decode and publish both went through jsoniter fromthen on.

Days later, a feature iteration landed in this service. It consumedWeCom'sowner-change event — an employee departs, WeCom hands their customersover — and this service's assignment engine distributed those customersamong the remaining staff by rule, publishing an assignment event foreach split. Departures are rare — a handful per tenant per month — sothese events were a small fraction of the millions of daily events. Thefeature had shipped once before with a flat event type, verified end toend by the automated suite on a test tenant.

The second iteration added chain support: a departing employee'scustomers could exceed one receiver's capacity, so the assignment enginere-dispatched the remainder to the next receiver, and that dispatch wasitself an assignment event. Each re-dispatch carried the assignment itcontinued, so the engine could audit how one departure was split anddeduplicate overlapping splits. Every step of that chain was the sameevent type, so the field carrying the previous assignment pointed at thetype itself — an anonymous embedded pointer, the event layer's standingconvention. Every event in the chain was published through the SDK'sjson package, the one I had just switched to jsoniter. Thealerts that followed were this article's subject: a growth curve pointsat accumulation, a spike at a single allocation path gone wrong, and theinstruments that watched the first had nothing to say about thesecond.

The migration left jsoniter behind every JSON touchpoint in theservice: envelope decode, cache, an executor SDK, and the publish path.The first three were dependency pins, swappable one at a time. Thepublish path was event-layer business code and shipped only inside theevent SDK's builds. That distinction decided how the suspicion could betested, and later how the failure was isolated.

What the failinginstrumentation could not see

Four standard instruments were considered:

• Logs: the triggering events were ordinary, and theprocess died before the consumer could record its state.
• Scheduled pprof sampled too infrequently to catch asub-second spike.
• Threshold-triggered pprof threw false positivesfrom normal allocation and could still miss a jump between polls.
• Core dumps: the memory-limit kill sentSIGKILL before a core could be written.

All four needed the process to survive long enough to leave evidence;this one did not. After a week it had no time signature — what it mostplausibly indexed was the set of recent changes on the JSON path, whichonly the build could step through one at a time.

The codec suspicion

The JSON change I knew about was my own March 7 migration, sojsoniter was the first suspect. On March 11 I swapped it for sonic on the cache path —a dependency swap. The publish path, business code inside the SDKbuilds, was beyond that kind of swap and stayed on jsoniter. The OOMrecurred within a day; I rolled sonic back the next morning.

That experiment ruled out the cache path, not jsoniter as a whole.Jsoniter still ran on the envelope, the executor SDK, and the publishpath. No dependency swap could reach those remaining paths; stepping theSDK builds could. I stopped swapping libraries and started bisectingbuilds, treating the recent JSON changes as a single bundled delta.

The bisection

On March 12 I rolled back to the last known stable build and bisectedthe incident delta — one staged deploy and observation window per step,eight days across eight rollbacks. The rollback gave a baseline, not ananswer — pods still died daily, so a quiet window proved nothing. Thebuilds stepped the internal event SDK between versions that carried thenew concrete event type and versions that did not; jsoniter itself nevermoved. The SDK version was the deployable handle for the bundled change:subscription-layer code and its dependency pins shipped together, sostepping it moved the new event type in and out of the process.

The bisection isolated the build delta that introduced the concreteevent type. Tracing it through the service: the engine built each chainevent and published it through the SDK's json package,where jsoniter lazily compiled an encoder for the type. The reducedTransferEvent case reproduced the resulting unboundedtraversal locally.

The mechanism, reconstructed

The production type graph is reconstructed as containing theanonymous embedded-pointer cycle; the business shape below is likewise areconstruction of the iteration's semantics, not a transcript of itsfields.

jsoniter compilesa codec on first use — encoder and decoder share the same type walk— before the first byte is read or written. Named recursive fields arebounded by jsoniter's placeholdercodec. An anonymous embedded pointer takes a different path: describeStructrecursively expands the embedded struct without a visited-typeguard. The decodedepth limit is downstream and cannot protect a process that dieswhile compiling the codec.

1
2
3
4
5

// jsoniter v1.1.12, reflect_extension.go:348-373, abridged
if field.Anonymous() && (tag == "" || tagParts[0] == "") {
    // ...
    structDescriptor := describeStruct(ctx, ptrType.Elem())
}

1
2
3
4
5
6
7
8
9
10

type TransferEvent struct {
    *TransferEvent // the assignment this event continues
    CompanyID string `json:"companyId"`
    From      string `json:"from"`
    To        string `json:"to"`
    Reason    string `json:"reason"`
}

// jsoniter.Marshal(TransferEvent{...})
// → fatal error: stack overflow, before the first byte is written

Why production became an OOM

The incident ran on Go 1.18, and the reduced case reproduces therewith jsoniter v1.1.12 — the version the service had pinned.The same case reproduced in every release from v1.1.5through v1.1.12, while the standard library and a namedrecursive pointer field remained safe.

Production ran multiple goroutines inside an 8 GB Pod, and jsonitercompiled codecs lazily — concurrent first-use misses could place severalgoroutines in the same recursive path, so process memory grew inparallel and the Pod limit could arrive before any single goroutine hitits own stack limit. The harness below demonstrates that arithmetic.Most Pods never received a rare assignment event at all. Because thespike ended too quickly for a useful profile, the exact RSS contributionfrom goroutine stacks, heap objects, and runtime allocations remainsunknown.

The publish call never returned: the assignment event the engine wasemitting never reached the broker, and the owner-change message theservice had consumed to trigger it was never acknowledged either. Whenthe Pod was OOM-killed, its consumer connection closed with that messageunacknowledged, and the shared subscription redelivered it to anotherPod, where the same chain repeated the failure. This wasconsumer-disconnect redelivery, not an application-issued nack — the OOMnever advanced any retry counter. The burst ended only when the messageexhausted its redelivery budget and landed in the dead-letter topic:each departing employee's event burned through a handful of pods, thenwent quiet. We only found that out during the retrospective — nobody hadlooked at the dead-letter topic during the week of debugging, though itheld the failing messages, their retry trails, and the event code thatwould have pointed at the trigger from day one.

The harness below reproduces that loop locally as a single copy-pasteprogram, scaled down 16× (64 MB stack walls instead of Go's ~1 GBdefault) and with the pod limit simulated by an RSS threshold. Eachchild process is one "pod": eight goroutines hit the same first-usecompile, and the parent replays the unacknowledged message every timethe child dies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

// Command oom-harness reproduces the incident's cross-pod cycle in miniature.
// Each child process is one "pod": N goroutines hit the same first-use
// jsoniter compile of a self-embedding type. The parent is the "cluster":
// on every death it replays the unacknowledged message, as the shared
// subscription did after each OOM-killed pod disconnected.
//
//go run .          # runs three redelivery rounds, prints per-pod peak RSS
//
// Safety: the child caps each goroutine's stack at 64 MB (16x below the
// runtime default), the parent fixes three rounds, and each pod dies in
// ~100ms — the whole run stays under a second and under 800 MB.
package main

import (
"fmt"
"os"
"os/exec"
"runtime/debug"
"sync"
"syscall"
"time"

jsoniter "github.com/json-iterator/go"
)

// Event is the incident's type graph, reduced: an anonymous embedded
// self-pointer. Compiling a codec for it recurses without a visited set.
type Event struct {
*Event
ID string `json:"id"`
}

const (
workers    = 8   // concurrent first-use compiles per pod
stackMB    = 64  // per-goroutine stack ceiling (production: ~1 GB)
podLimitMB = 300 // simulated pod memory limit (production: 8 GB)
)

// child is one "pod": workers goroutines, one poison publish each.
func child() {
debug.SetMaxStack(stackMB << 20)
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
_, _ = jsoniter.Marshal(Event{ID: "assign-1"}) // the publish that never returns
}()
}
wg.Wait() // unreachable: the process dies inside the first-use compile
}

// peakRSS reads the finished child's peak resident set from Rusage.
// Linux reports ru_maxrss in KB, macOS in bytes.
func peakRSS(cmd *exec.Cmd) int {
if cmd.ProcessState == nil {
return 0
}
ru, ok := cmd.ProcessState.SysUsage().(*syscall.Rusage)
if !ok {
return 0
}
kb := ru.Maxrss
if kb > 1<<20 { // bytes -> KB
kb /= 1024
}
return int(kb) / 1024
}

func main() {
if len(os.Args) > 1 { // child mode: invoked by the parent below
child()
return
}
for redelivery := 1; redelivery <= 3; redelivery++ {
cmd := exec.Command(os.Args[0], "pod")
start := time.Now()
_ = cmd.Run()
peak := peakRSS(cmd)
fmt.Printf("redelivery %d: pod died in %v, peak RSS %d MB", redelivery,
time.Since(start).Round(time.Millisecond), peak)
if peak >= podLimitMB {
fmt.Printf(" — over the %d MB pod limit: an OOMKill before any single %d MB stack wall\n",
podLimitMB, stackMB)
} else {
fmt.Println()
}
}
}

1
2
3

redelivery 1: pod died in 100ms, peak RSS 773 MB — over the 300 MB pod limit: an OOMKill before any single 64 MB stack wall
redelivery 2: pod died in 85ms, peak RSS 760 MB — over the 300 MB pod limit: an OOMKill before any single 64 MB stack wall
redelivery 3: pod died in 86ms, peak RSS 773 MB — over the 300 MB pod limit: an OOMKill before any single 64 MB stack wall

The peak scales with concurrency — eight goroutines × one stack each— which is the arithmetic behind the production OOM: process memory grewin parallel while the stack limit only ever guarded one goroutine at atime. On this laptop the child still dies to the stack wall — no cgroupexists to kill it first. Inside a real container limit, the memory wallarrives first, the process dies without a word, and the parent'sredelivery loop replays the cross-pod cycle the incident showed.

Two controls in the harness's style isolate the trigger — same call,same shape, one property changed each time:

1
2
3
4
5
6
7
8
9
10

// Named recursive field: jsoniter's placeholder codec terminates compilation.
type NamedEvent struct {
Previous *NamedEvent `json:"previous,omitempty"`
ID       string      `json:"id"`
}

_, _ = jsoniter.Marshal(NamedEvent{ID: "assign-1"}) // completes

// Same anonymous shape with the standard library: field resolution terminates.
_, _ = json.Marshal(Event{ID: "assign-1"}) // completes

The standard library uses a visitedmap when resolving fields — the very function jsoniter's branchlacks. The same domain shape had been serialized underencoding/json since the feature first shipped; onlyjsoniter's compilation entered the failing traversal.

Sonic avoids the failing traversal at the field-resolution step. Inv1.14.0, its resolver delegatesto encoding/json.typeFields; in v1.14.1and v1.15.4, it performs that traversal locally witha visited-type map. Both approaches stop revisiting the anonymouslyembedded type. Separately, sonic's JIT and optimized codec compilers tracktypes already being compiled and handle recursive fields withoutrepeatedly expanding their work. For this embedded-cycle case, thefield-resolution guard is the relevant protection. These areretrospective checks (Go 1.24.9, macOS, JIT and optimizedpaths both ways); they verify the reduced case, not the production sonicversion or runtime.

Why the automated suitestayed green

The suite that had verified the first release stayed green for amechanical reason. It exercised the real chain on a test tenant —simulate a departure, let the engine publish, consume — but by the timethis iteration ran, that tenant's engine had been switched off for aparallel feature's testing, and nothing flagged the gap — a green runlooked exactly like an empty one. With the engine off, no assignmentevent was ever built; nothing published; and jsoniter compiles a codeconly on first use — zero publishes meant zero compilations. The greenwas real but empty: no test ever reached this change. Unit tests couldnot either — they call the engine with struct literals and neverserialize an event.

The fix

With the cause isolated, the fix followed directly: the publishertook its own sonic reference instead of the SDK's sharedjson, so the envelope and every other consumer of thatvariable kept jsoniter. I validated the move first against the reducedreproduction, which sonic encodes without entering the failingtraversal. The jsoniter migration kept the win it had already delivered:the envelope and cache paths stayed on it, and the slow-climb incidentdid not come back. Fixing the type instead (adding a field name or tagto break the anonymous embedding) would also have closed this failure,as the named-field control demonstrates; the migration was chosenbecause it left the shared event contract untouched while covering everyevent type at once.

The production OOM stopped after the migration and never recurredthrough the months that followed, while the service kept running until Ileft the company. These results make the publish-path encoder thestrongest operational explanation for the OOM. They do not identify theexact production RSS split between goroutine stacks, heap, and runtimeallocations; the spike ended before a useful profile could be captured.The type graph itself remained unchanged; the source comparison in theprevious section explains why sonic handles the embedded cycle.

Lessons from theinvestigation

• The shape of the failure decided the tool. A sub-second spike withno time signature ruled out accumulation, pointed at a single allocationpath, and made every runtime instrument useless — the process diedbefore leaving evidence. I lost days to codec swaps and quiet windowsbefore accepting that; build bisection was the expensive but onlydecisive step — eight rollbacks to one delta. For a fast-fatal failure,read the shape first and step builds early.
• I swapped one package-level json variable to fix slowmemory growth and silently moved the publish path to jsoniter with it.Now, before any dependency swap, I list every caller of what I'mtouching.
• The dead-letter topic held the failing messages and their retrytrails for the whole week we chased the OOM; we found it in theretrospective. Now a consumer incident starts with DLQ depth and retrycounters, before any profiling.

Previous in this series: Runninga Million-DAU Casual Game on Go: Rooms as the ConcurrencyBoundary.

Next in this series: Buildinga Production Go Scheduler for 200K+ Delayed Jobs perSecond.

I led the architecture selection and ECS implementation. Profilingshowed how much CPU time component lookups consumed.

Designing for 10,000players per instance

Tencent's publishing team planned to roll out access in stages. Theservice needed to absorb each traffic increase by adding instances. Loadtests used a fleet of slightly lower-spec instances running bots throughthe real client protocol: entering rooms, moving, and collecting. Thetarget was 10,000 real players per server instance: about 667 rooms withfifteen players each, updated at 20 Hz.

Late room-state updates led us to investigate the room's serialupdate cycle.

Outside rooms, another service handled idle collection throughrequest-response calls. Inside rooms, players shared a grid of up to200×200 cells for movement, exploration, combat, and collection.Clicking a destination triggered server-side pathfinding. Playermovement step size was defined separately from the simulation tick. Aroom held about fifteen real players, 30+ combat NPCs, and a varyingnumber of bot players, with a room capacity of 30–100, depending on roomclass. The room service updated shared state on each tick, with threeconstraints:

• Responsive movement. Delayed server updates were visible toplayers.
• A bounded update cycle per room. Each room updated every 50 ms (20Hz), and all work for that tick had to finish within the interval.
• Bounded broadcast fanout. Room population and visibility limits hadto cap the audience for each update.

Choosing theroom boundary and its execution model

I made two choices: which state each execution unit would own, andhow it would run the simulation. Actor-styleownership determined who held mutable state and how work reached it.ECSdetermined how components were stored and systems iterated overthem.

Keeping shared statewithin one room

Giving each player its own actor would separate ownership of playerstate, but interactions involving multiple players and shared room statewould require coordination between actors. For example, if two playersclaimed the same collection cell within one tick, an actorimplementation could resolve the conflict by giving a room actorauthority over the cell and routing both claims to it. The cost isforwarding requests, returning results, and defining when the playeractors apply those results. Visibility queries, movement collisions, andPvP interactions raise the same ownership question. I kept theinteracting state under one room owner, so these operations couldcomplete within one simulation pass.

An actor-based design can also batch inputs and resolve them in astable order; Proto.Actorprovides a Go implementation of actor messaging. When player state issplit across actors, the design must also define tick boundaries andcoordinate the state used in each tick.

A single global simulation loop would couple unrelated rooms: a slowroom would delay the others, and the serial loop could not use multipleprocessors.

Choosing ECS for the roominterior

The closest alternative was a room-level actor: one goroutine andmailbox per room, with entities stored as objects and events handled bymethods. It provided the same ownership boundary as the shipped design.In the candidate design, handlers processed messages in arrival orderand dispatched each event separately. My tick-based design batched theframe's inputs and let systems query components directly for operationssuch as finding the players within an area of interest (AOI).

I chose ECS to support frequent gameplay changes inside existingrooms. Components held buff effects and expiration times; systemsapplied the effects in tick order. Items could reuse the same componentsand systems. Adding mechanics this way reduced branching in playerobjects and message handlers.

I could order buff application, movement, damage calculation, andexpiration explicitly, with no concurrent writes inside the room. Fixedinputs and simulation time made interactions between mechanics easier toreproduce in tests.

The speed-item example below uses an array-based ECS.EntityID indexes the component arrays, and masks recordwhich components are present on each entity. Room.Tick runsitem application, buff expiration, and movement serially. Adding orremoving a component changes membership while retaining itspre-allocated slot.

Each tick simulates the 50 ms interval beginning atnowMS. Buffs are active on[appliedAt, expiresAt), so a buff expiring at 100 ms doesnot affect movement over [100, 150). Durations align withticks, and each player can have one speed buff. Positions usethousandths of a cell; distance depends on speed and elapsed time.

Save as main.go and run go run main.go:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109

package main

import "fmt"

type EntityID int

type ComponentMask uint8

const (
HasPosition ComponentMask = 1 << iota
HasMoveSpeed
HasSpeedBuff
)

// Components contain data only.
type Position struct{ MilliCells int }
type MoveSpeed struct{ MilliCellsPerSecond int }
type SpeedBuff struct{ Multiplier, ExpiresAtMS int }

// Room owns component arrays; masks record component membership.
// Entity IDs are fixed array indices in this reduced example.
type Room struct {
Masks      []ComponentMask
Positions  []Position
MoveSpeeds []MoveSpeed
SpeedBuffs []SpeedBuff
}

func (r *Room) Has(id EntityID, required ComponentMask) bool {
return r.Masks[id]&required == required
}

// Query visits only entities with all required components.
func (r *Room) Query(required ComponentMask, visit func(EntityID)) {
for id := range r.Masks {
if r.Has(EntityID(id), required) {
visit(EntityID(id))
}
}
}

type UseSpeedItem struct {
Target                 EntityID
Multiplier, DurationMS int
}

func SpeedItemSystem(r *Room, nowMS int, commands []UseSpeedItem) {
for _, cmd := range commands {
if !r.Has(cmd.Target, HasMoveSpeed) {
continue
}
r.SpeedBuffs[cmd.Target] = SpeedBuff{cmd.Multiplier, nowMS + cmd.DurationMS}
r.Masks[cmd.Target] |= HasSpeedBuff // add component
}
}

func BuffExpirySystem(r *Room, nowMS int) {
r.Query(HasSpeedBuff, func(id EntityID) {
if nowMS >= r.SpeedBuffs[id].ExpiresAtMS {
r.Masks[id] &^= HasSpeedBuff // remove component
r.SpeedBuffs[id] = SpeedBuff{}
}
})
}

func MovementSystem(r *Room, dtMS int) {
r.Query(HasPosition|HasMoveSpeed, func(id EntityID) {
speed := r.MoveSpeeds[id].MilliCellsPerSecond
if r.Has(id, HasSpeedBuff) {
speed *= r.SpeedBuffs[id].Multiplier
}
r.Positions[id].MilliCells += speed * dtMS / 1000
})
}

func (r *Room) Tick(nowMS, dtMS int, commands []UseSpeedItem) {
SpeedItemSystem(r, nowMS, commands)
BuffExpirySystem(r, nowMS)
MovementSystem(r, dtMS)
}

func main() {
const player, control, stationary EntityID = 0, 1, 2
r := Room{
Masks:      []ComponentMask{HasPosition | HasMoveSpeed, HasPosition | HasMoveSpeed, HasPosition},
Positions:  make([]Position, 3),
MoveSpeeds: []MoveSpeed{{1000}, {1000}, {}},
SpeedBuffs: make([]SpeedBuff, 3),
}
want := []int{100, 200, 250, 300}
for step, nowMS := range []int{0, 50, 100, 150} {
var commands []UseSpeedItem
if step == 0 {
commands = []UseSpeedItem{{Target: player, Multiplier: 2, DurationMS: 100}}
}
r.Tick(nowMS, 50, commands)
if r.Positions[player].MilliCells != want[step] ||
r.Positions[control].MilliCells != (step+1)*50 ||
r.Positions[stationary].MilliCells != 0 ||
r.Has(player, HasSpeedBuff) != (nowMS < 100) {
panic(fmt.Sprintf("unexpected state at %d ms: %+v", nowMS, r))
}
fmt.Printf("[%d,%d) ms: positions=%d,%d,%d buff=%t\n",
nowMS, nowMS+50, r.Positions[player].MilliCells,
r.Positions[control].MilliCells, r.Positions[stationary].MilliCells,
r.Has(player, HasSpeedBuff))
}
fmt.Println("PASS: component add/remove, expiry boundary, and movement query")
}

Output (positions in thousandths of a cell):

1
2
3
4
5

[0,50) ms: positions=100,50,0 buff=true
[50,100) ms: positions=200,100,0 buff=true
[100,150) ms: positions=250,150,0 buff=false
[150,200) ms: positions=300,200,0 buff=false
PASS: component add/remove, expiry boundary, and movement query

The position-only entity stays still, and the control player keepsits base speed. The buffed player returns to base speed at theexpiration boundary.

A room-level actor could implement the same gameplay and tests,including ECS internally. In the candidate's event-handler design,interactions between effects would require ordering rules acrosshandlers. I chose component composition and an explicit system sequenceto keep those rules together as gameplay grew.

In the shipped service, one goroutine owned each room's mutablestate, and Go'sruntime scheduled room goroutines across the available processors.Systems ran serially within that goroutine, accessing components for therelevant entities on every tick.

Cross-room state lived on a separate tier with its own consistencyrules, so features spanning rooms required additional coordination.

A slow entity could delay its room's tick while unrelated roomscontinued, provided the process and scheduler stayed healthy. Extracores could run other rooms, but could not shorten that room's serialtick. A room that exceeded its deadline needed less work per tick,further optimization, or a lower population cap.

Roomscheduling, serial ticks, and state management

Each instance pooled rooms, pre-allocating them to keep allocationoff the hot path and recycling them as demand fell. Players joined roomclasses matching their level and intent, keeping new players amongpeers. A central scheduler tracked instance load and placed each newroom on the least-loaded instance.

I initially stored components in entity-keyed maps to supportchanging entity counts without reserving full-capacity arrays for everyroom. Rooms with fewer entities could allocate storage as needed, thoughmaps carried their own bucket and key overhead. Each system looked upcomponents while iterating its entities. Player input arrived throughthe connection layer; each tick consumed the inputs accumulated sincethe previous update.

State changes were sent only to players whose area of interestincluded the affected location. PvP interaction used the same visibilityboundary. AOI filtering and room population caps bounded broadcastfanout.

Persistence ran outside the tick. Room snapshots were takenasynchronously and flushed periodically to MongoDB. A restart restoredeach room from its last persisted snapshot, potentially losingsubsequent updates. That was acceptable for this casual game. Snapshotflush lag and the recovery loss window were not measured; I would addmetrics for both.

Map lookups exhausted thetick budget

pprofshowed map operations consuming 60–85% of the profiled CPU time. Thesefigures reflect the load-test results I confirmed at the time; they arenot quoted from retained profiles. The hot frame wasaeshash, Go's AES-NI-based hash implementation. The Go 1.13runtime routed maplookups through its hashingfunctions to AESENC-basedcode.

Across all systems and rooms, component lookups required roughly 13million hash computations per second per instance, before buckettraversal and key comparison. Map insertions and growth addedallocations. GC ran about once a second during the test. Entity IDscould directly index component arrays; the maps added a hash lookup toeach access.

I kept the room boundary and serial system order, and replacedcomponent maps with pre-allocated fixed-capacity arrays indexed byentity ID. Storage was reused across room lifecycles. Sequentialcomponent traversal also favored cache locality, though cache misseswere not measured separately.

Pre-allocated arrays reserved space up to each room's entity cap evenwhen occupancy was low. Room reuse saved allocation work but retainedthat storage. Growing arrays with occupancy or grouping entities bycomponent set could reduce unused slots, but would addstorage-management work. I accepted the reserved space to keep lookupsdirect and reuse storage.

After the change, p99 tick elapsed time, measured withtime.Since, fell below 50 ms, and instance CPU utilizationsettled near 70%. Profiles showed less hot-path allocation churn, but Ido not have before-and-after measurements of GC pause frequency.

Broadcast fanout was not the bottleneck in this test. The tickoverruns were resolved before production.

Production results andcapacity

At 10,000 players per instance and fifteen players per room, eachinstance ran about 667 rooms at 20 Hz. A typical room contained roughly60–80 entities: players, combat NPCs, bots, and collection objects. Atthat density, the 8-vCPU instance averaged 70% CPU utilization duringthe same load-test window. Dividing its CPU consumption by the number ofroom ticks gives about 0.42 ms per room tick:

1
2
3

CPU consumption = 8 vCPU × 70% = 5.6 CPU-seconds per second
room tick rate  = 667 rooms × 20 ticks/second = 13,340 room ticks/second
CPU per room tick = 5.6 / 13,340 seconds ≈ 0.42 ms

This average includes connection handling, serialization, GC, andother instance work.

Room goroutines waited between updates, allowing hundreds of rooms toshare the 8 vCPUs. We added instances as room density approached theper-instance limit.

The memory model was:

1

memory = process_baseline + rooms × room_memory + buffers + headroom

Estimated per-room memory was a few hundred KiB at fifteen players.At ~667 rooms per instance, that amounted to a few hundred MiB. The 16GiB instances also provided capacity for the Go process, playerconnections, queues, snapshots, caches, runtime metadata, and headroom.The fleet used the same instance configuration throughout: 8 vCPU / 16GiB on TencentCloud.

Concurrency outside activity peaks was in the tens of thousands. Asingle instance carried over 10,000 concurrent players. At activitypeaks, the fleet served roughly 200,000 concurrent players across 20+instances, with over one million DAU.

The talk Overwatch GameplayArchitecture and Netcode gave me many of the ideas behind thisdesign, especially separating component state from system behavior andorganizing data for sequential access.

**Next in this series: OneQueue for Every State Change: Fixing Silent Over-Assignment in a Live QRCode System..