Four Go engineers built eighteen domain services for a consumer transaction platform with weekly live-ops releases — time-boxed in-game events and promotions whose cadence set the delivery rhythm. To keep integration work manageable, we used a single API DSL to generate service conventions, Protocol Buffers contracts, and Swagger/OpenAPI documentation.
Context and requirements
The product team was assembled from scratch in early 2025 with seventeen people: four Go engineers building the backend, six frontend developers, two QA, two UI designers, and three product managers driving the live-ops roadmap. That roadmap required weekly feature releases rather than a long platform-build cycle. I initiated and led the platform program that ran from February to May: an architecture template, a contract toolchain, a shared gateway, base libraries, and CI, all built in parallel with the first business services. The platform then became the default path across identity and access, financial transactions and ledger integrations, consumer engagement, growth and live-ops, and platform infrastructure. The repository statistics cover the eighteen services; the shared API gateway and one tooling repository sit outside that count.
The same four engineers had to ship against money-movement flows and around-the-clock availability while the platform was being built. We also made one choice early: standardize on go-zero rather than build another framework. We used the platform to encode conventions while relying on go-zero for runtime infrastructure.
The release-unit boundary followed a real difference in change frequency. Live-ops code changed weekly by design: time-boxed events, promotions, and campaign pages moved with the product roadmap. Ledger and identity code changed less often and required more conservative review. Keeping them in separate deployable units meant a weekly activity release did not require redeploying the money path or the authentication path. That principle set the coarse release boundaries; the remaining domain split was an ownership and evolution choice, and the platform's contract, generation, and default-runtime mechanisms kept the resulting governance cost manageable.
Each service needed auth, error codes, structured logging, RPC wiring, gateway registration, and tracing. Maintaining these by hand across the service fleet would consume integration time and let implementations diverge. The platform needed:
- One contract entry per service, from which handlers, RPC, and docs all derive.
- A scaffold that generated the required architecture.
- Platform-wide error codes, auth, and context propagation as libraries.
- Gateway-managed response caching for consumer-facing interfaces, configured at the interface level rather than rebuilt inside each service.
- CI and tracing by default on every service.
- Generated service shells with production defaults from their first commit.
- An onboarding cost we could measure.
Options Considered
The main choice was the contract direction. go-zero already provided
service discovery and goctl, its API-first code generator; the team knew
it well. The alternative to goctl's .api DSL was
proto-first: define gRPC contracts and
derive HTTP from them.
Proto-first would have made proto the canonical internal contract,
but the client-facing HTTP route, request shape, auth annotation, cache
option, or Swagger document still needed a second definition. That was
the double-contract drift we wanted to remove. It also
would have pushed us toward maintaining more of the generation chain
ourselves. DSL-first gave up some proto-level control: services that
needed advanced gRPC features fell back to hand-written proto. We
accepted that rare exception because one .api file
could remain the entry for handlers, generated proto, and Swagger.
Architecture / Design
The C4 container view below shows the platform as it runs today. Clients speak HTTPS to a single API Gateway, which routes to domain services over gRPC using Protocol Buffer descriptor sets generated from the shared contract repository; Kubernetes Endpoints resolves live addresses. Services keep per-service schemas in TiDB, cache in Redis, and export spans to Zipkin. For user-facing pushes, services publish to Kafka and ws-hub fans notifications out over websockets. An i18n hook on the gateway response chain localizes dynamic response content, so services stay locale-blind.
Key Technical Decisions
DSL as the single contract entry
A single authentication .api file carried the routes,
types, and three access-control tiers:
1 | syntax = "v1" |
Handlers and types came from goctl; our plugins generated proto and Swagger from the same source. There was no second route definition to drift from.
The contract repository held fifty-five .api contract
files and sixty-eight generated proto schemas. Between March 4 and
October 9, 2025, it absorbed 3,568 commits — more than any single
service — with four hundred to seven hundred landing every month. Every
interface change in the platform flowed through that one repository.
Fill the generation gaps with plugins
Vanilla goctl generated HTTP handlers from .api but not
proto or Swagger. We added goctl-proto and goctl-swagger plugins so one
.api file produced all three in the same commit.
The scaffold as specification
The onion/DDD structure lived in the README and custom goctl templates. A generated service started with the layering, outbox, and repository seams already in place, before business code was added.
Transactional domain events through the outbox
Services persisted domain events in an outbox table within the same
TiDB transaction as the business-state change. TiCDC captured committed
outbox rows and delivered them to Kafka as domain_event_*
topics. This kept event persistence consistent with the business
transaction and removed the gap between committing data and publishing
an event. Delivery remained asynchronous, and consumers still needed
idempotent handling on replay.
Low-latency ws-hub notifications were published directly to Kafka. Transactional domain events followed the outbox path.
Domain boundaries and data ownership
All eighteen services shared one TiDB cluster, so the boundary was drawn at the table level. The user service owned auth, KYC, and customer-profile tables; the ledger service owned currency orders, account balances, and token metadata. Cross-service access ran over gRPC, and a repository-layer scan confirmed zero cross-domain table access. Disjoint table ownership preserved service boundaries without operating eighteen databases.
These were ownership and change boundaries rather than eighteen independent failure domains: the services shared one TiDB cluster and the platform's gateway, Kafka, and tracing infrastructure, so a cluster-level failure or a problematic migration could still affect multiple domains.
Platform-wide error codes
bizerr encodes every error as a six-digit code: two digits of service ID, two of function, two of error. Clients match on the prefix; anyone can decode an error to its origin without grep. Errors carry captured stacks and wrap their causes, so production incidents could be traced through the chain without re-running anything. The registry doubles as the coordination point — a new service claims its service ID once, and an unregistered code causes a runtime panic.
That panic treated an unregistered code as a programming error rather than an operational error. It was intended to fail in the first test run instead of allowing an error with no message, locale, or service origin to reach a user. The cost was that an untested path could still become a production failure, so centralized registration and error-path tests mattered; a build-time registry check would have been the better enforcement point.
1 | // each service claims an id; codes are registered with metadata |
Gateway as the composition edge
The gateway wrapped a maintained go-zero fork and owned everything cross-cutting: JWT auth, header processing, response wrapping, CORS, response caching, and an i18n hook on the response chain that localizes response content per user locale. Consumer-facing interfaces enabled caching through gateway configuration, so a service did not have to implement its own response-cache path. Because localization also lived in the gateway, services stayed locale-blind; adding a language required one gateway change.
Keeping these authentication and edge policies in one gateway implementation also reduced security-policy drift across services. Each service still enforced its own domain authorization, while route-level authentication and cache policy came from the same contract-driven configuration path. That made the boundary easier to review and reduced the chance that independently implemented services would apply inconsistent rules.
The cache setting stayed in the same interface contract. A consumer-facing banner route looked like this:
1 | @server( |
The proto plugin emitted this route as a per-method cache option with
a 300-second TTL and CACHESCOPE_GLOBAL. At startup, the
gateway read the descriptor and installed Redis cache middleware only
for routes with a positive TTL. The key covered the HTTP method, path,
query, JSON body, Accept-Language, and
User-Agent. On protected routes, JWT ran before cache, so
the authenticated provider and user IDs also entered the key. An
authenticated request was therefore isolated even when the annotation
requested a global cache.
The i18n hook localized the CMS's dynamic content — product descriptions, promotions, and operational copy — per user locale, managed by operations through the language service. Error messages took a different path by design: services return language-neutral error codes, and the frontend maps each code to a localized string through its own static i18n library, avoiding a backend round-trip for error localization.
Shared generation and CI templates
Every convention that could be enforced by generation was enforced there: custom goctl templates generated the onion structure, makefile targets pinned the base-library set, and shared CI templates standardized pipelines for Go, React, and Next.js services.
Runtime and deployment choices
We skipped Istio because go-zero already carried service discovery and the microservice mechanics we needed. We used the Kubernetes API instead of a custom deployment system, kept the maintained go-zero fork instead of a new RPC framework, and left early configuration in ConfigMap until business operations needed config-hub's runtime changes.
Implementation
A generated service started with an .api skeleton, onion
directory tree, Dockerfile, and deployment manifests. Its build
referenced the six base libraries, the go-zero fork, and service SDKs;
shared GitLab includes supplied Go, React, and Next.js pipelines. Zipkin
tracing was a config line. The gateway loaded method descriptors at
startup and resolved backends through Kubernetes Endpoints, so adding
pods did not require gateway redeployment.
Pod rescheduling, ws-hub reconnection, and gateway instance failure paths were designed but never systematically tested under failure conditions. A chaos suite that restarts gateways, interrupts service discovery, and exercises ws-hub reconnection under load would validate those assumptions.
Adoption
The scaffold had to stay ahead of the services while the four-person team shipped features. The template evolved with the first services; every later service generated from it. The only non-generated convention was claiming a service ID in the bizerr registry.
Once the scaffold and DSL stabilized, team members started writing business logic with AI coding agents while feature delivery continued. The contract defined interfaces, the generated tree supplied the layering, and the error registry defined failure shapes, constraining the code agents produced.
Where the Defaults Leaked
A contract change reached QA
The .api files had no change monitoring. When one
interface definition changed, no consumer saw the generated proto or
Swagger diff before merge. QA's automated consumer-call tests caught the
incompatible change before release. A CI step that diffs generated proto
and Swagger against the previous version and posts the delta would
expose these changes before merge.
An admin editor crossed four contracts
One admin editor exposed the missing composition boundary. A single page called content/activity, catalog, payment-metadata, and localization services directly. It loaded catalog providers, playable items, token metadata, and localized components. On save, the frontend first upserted localized copy, read back generated references, merged them into the business payload, and then called the business create or update API.
The browser owned the cross-service write sequence. If localization succeeded but the business write failed, localized components remained while the business record did not. Several admin pages also independently fetched catalog providers, payment metadata, or localized labels, so a small visible change could cross multiple domain contracts.
For this editor, I would add a BFF endpoint that owns the page shape and localization-plus-business write sequence, including compensation and idempotency. Domain services would retain content, catalog, payment, and localization ownership. Other admin pages could use the same boundary instead of assembling four separate contracts in the frontend.
Results
All eighteen Go domain services came from the scaffold. Rare advanced gRPC cases used hand-written proto, but no service was hand-rolled outside the template.
The estimated development-cost result was a 60% reduction in onboarding work, measured in boilerplate lines and person-days to a first deployed endpoint. It was an internal estimate, not a controlled experiment.
The platform took four months of focused work alongside the first business services. It replaced days of recurring boilerplate per service with a generation command and shared outbox and error-registry conventions for services handling money. The same scaffold supported the June and July live-ops expansion.
The engineering result was a governable form of domain decomposition for a four-engineer backend team. Contract-first design, code generation, and platform defaults kept the cost of repeated service setup, contract coordination, and shared conventions under control. The evidence measures governance and delivery cost; it does not claim that one service topology is universally better than another.
Trade-offs
The platform reduced repeated setup, but it added its own costs. The maintained go-zero fork was a permanent maintenance duty; we took fixes on our schedule, not upstream's. Onion layering added indirection for thin CRUD services even when generated, and the central error-code registry required coordination so two services could not silently claim the same ID.
This investment was harder to justify on a short product runway or with only a few services. The platform's role was to make domain decomposition affordable: release boundaries followed ownership and change frequency, while generation kept repeated service setup and shared conventions manageable.
The remaining gaps were at the cross-service seams: contract changes had no generated diff visible to consumers, and one admin page composed four domain contracts in the browser. Generation made per-service defaults cheap; it did not replace consumer-visible contract checks or a proper composition boundary.
Previous in this series: Debugging a Sporadic Go OOM in a Recursive JSON Encode Path.
Next in this series: Building a Real-Time Data Platform with TiDB CDC and Flink.