Menu

Research · Methodology + tooling · 2026

Batch2go: isolating batching effects by placement

Yu-Heng Hong · Guan-Ju Peng — National Chung Hsing University · Open-source implementation of the second study

Primary artifact · GitHub

matthewhoung/batch2go

The proxy, the three controlled conditions, the run harness and the matplotlib analysis pipeline. C1, C2 and C3 are byte-frozen at a release tag; every result cell carries a completion marker so a run can be resumed idempotently, and each session writes a machine-readable manifest.

Go 1.25+MIT3,960 runs10 OTel metrics
Read the code

Serving systems batch requests at more than one point in the stack: an upstream component may aggregate before transmission, and the inference server may batch again at execution. Conventional end-to-end evaluation sees only the sum, which conflates transport-layer aggregation with compute-layer parallelism. This work treats batching placement as an isolable architectural variable, with three controlled conditions — a no-batching baseline (C1), a compute-layer control (C2) and a transport-layer treatment (C3).

The conditions are instantiated by Batch2go, a single-flight Go proxy that decomposes batch cycle time into a compute service time Sk and a transport overhead Dk. Single-flight execution is what makes the decomposition possible at all: it defines a regime in which Sk and Dk are separately observable rather than entangled in one aggregate measurement. Those two quantities then predict the upstream concurrency for a composed multi-flight deployment as M* = ⌈Dk/Sk⌉ + 1 — a number derived from the decomposition before any multi-flight configuration had been run.

Scale
3,960 runs · 4 CV models · an 8× payload range, 588 KiB to 4,800 KiB
Transport share
Dk/(Dk + Sk) rises from 32% to 69% across that payload range
Crossover
The decomposition ratio R changes sign inside a 2,028–4,800 KiB band — a hardware-calibrated regime boundary, not a tuning knob
R at c=100
0.66 ResNet-50 · 0.06 MobileNetV2 · 0.19 YOLOv5s · −1.65 YOLOv8n
Tail
About a 7× reduction in P99 tail-latency variability at the main control point (exploratory)
Stack
Go 1.25+, binary TCP framing with no HTTP overhead, a bounded queue with backpressure, OTel + Prometheus + Grafana

研究摘要

這個研究把「batching 發生在哪一層」當成一個可以單獨控制的架構變數。傳統的 end-to-end 量測只看得到總和，分不出傳輸層的聚合跟計算層的平行化各自貢獻了多少。

Batch2go 是一個 single-flight 的 Go proxy，把一個批次週期拆成計算時間 Sk 跟傳輸開銷 Dk。拆開之後就能事先算出上游應該開多少並行度，再拿去驗證——而預測的位置跟實際量到的吞吐量峰值是吻合的。那個 proxy 本身就是量測儀器，不只是受測物。

會做這個題目，是因為前一個研究量到 Triton 的網路 I/O 是單體式的 59 倍：瓶頸不在模型算得多快，而在資料怎麼搬。這是同一條研究線上的下一步，不是另一個獨立的題目。

Evidence · Figure 2

The prediction was made before the measurement

Two side-by-side throughput plots against maximum in-flight requests M. In the left panel, ResNet-50 with a predicted optimum of M*=2: throughput rises from about 448 requests per second at M=1 to a peak near 581 at M=2, then settles around 555 — all far above the 251 RPS unbatched baseline. In the right panel, YOLOv8n with a predicted optimum of M*=4: throughput rises from about 90 RPS at M=1 through 114 at M=2 to a peak near 144 exactly at M=4, then falls to 140 at M=8, crossing the 123.7 RPS unbatched baseline between M=2 and M=4. In both panels a black dashed vertical line marks the optimum predicted in advance from the formula, and the measured peak lands on that line. The finding is that a prediction made before measurement was correct for two models eight times apart in payload size.

Scroll the figure sideways →

The dashed vertical line in each panel is not fitted to the data — it is M* = ⌈Dk/Sk⌉ + 1 computed from the single-flight decomposition, before the multi-flight deployment was ever run. The measured throughput peaks land on it in both panels: M*=2 for ResNet-50 and M*=4 for YOLOv8n, two models an eightfold apart in payload size. A formula derived from decomposed measurements predicted the operating point in advance, and was right twice.fig. 2 — c4 multi-flight throughput vs m · two panels · 1659×695

Primary artifact · GitHub

matthewhoung/batch2go

The proxy, the three controlled conditions, the run harness and the matplotlib analysis pipeline. C1, C2 and C3 are byte-frozen at a release tag; every result cell carries a completion marker so a run can be resumed idempotently, and each session writes a machine-readable manifest.

Go 1.25+MIT3,960 runs10 OTel metrics
Read the code
Back to top