The legacy batch center and xxl-job had reached their dispatch ceiling, and every new instance multiplied lock contention on the same task table. We replaced them with a scheduling center on TiDB, etcd, and Apache Pulsar. Its on-the-hour workload peaked above 200K execution commands per second; production incidents later led us to revise its isolation model.
Debugging a Sporadic Go OOM in a Recursive JSON Encode Path
The alert came from Grafana. A Go service on the SOP event chain reached its 8 GB Pod memory limit and restarted; process memory went from flat to the limit in under a second, with no correlation to traffic or time of day.
The service ran across twelve pods, supporting SOP workflows for more than 30,000 enterprise customers. It consumed close to ten million events per day and supported daily outreach to hundreds of thousands of end users. The alerts kept coming for over a week: on most days a burst — one pod lost, then another within minutes as if the failure moved between them — and three or four on the worst days, until each burst exhausted itself and went quiet. Nothing in the standard instruments captured why.
One Queue for Every State Change: Fixing Silent Over-Assignment in a Live QR Code System
The escalation arrived as a refund threat. A customer had run a campaign, and thousands of people scanned its QR codes that day expecting to reach the sales team. One salesperson on the code had already hit his daily quota — the system showed him offline for assignment — and every new scan still connected to him. The rotation to the next salesperson never fired. The campaign closed far below target, the customer demanded a refund, then churned: a six-figure annual contract. The same mis-assignment had been producing quieter support tickets for months. The logic was correct; the member list it computed from was not.