Adaptive Routing in a Tiny Inference Cluster

The desktop lab is a seeded browser simulation. It does not run inference on real hardware, and its 50 ms SLO is illustrative, not a production figure.

01Problem formulation

A stream of inference requests arrives at a router sitting in front of $N = 3$ replicas. Each request must be assigned to exactly one eligible replica the moment it arrives, or be shed if none is healthy with queue space. For an accepted request:

The replicas are not interchangeable. They differ in mean service time, in the depth of work already queued, in health, and in what the router has observed about them so far. A good router cares about more than average latency: it is trading off queueing delay, tail latency, throughput, and SLO misses simultaneously. Conceptually (and only conceptually, nothing in the simulator optimizes this objective directly) each decision is trying to minimize expected latency plus a penalty for growing queues and violated SLOs.

02Tail latency

Under moderate load most requests finish quickly, so the mean and p50 look healthy. The failure hides in the tail: a small fraction of requests lands behind an overloaded or slow replica and waits far longer than the work itself requires. Variable request sizes also produce a service-time tail at low load; a p50–p99 gap alone does not establish queueing as the cause. Queueing can amplify that existing tail as load approaches capacity. Under sustained overload, waiting queues fill to the 50-request cap per replica. Full replicas are skipped, and arrivals are shed when no eligible replica has room. Read latency alongside shed and unfinished counts. See the queueing background [7].

03Routing policies

Round Robin cycles a cursor over an ordered list of eligible replica IDs. Let $E_t$ contain healthy replicas with fewer than 50 waiting requests, using zero-based list positions and one-based replica IDs:

The cursor starts at zero and advances only on an accepted assignment. If $|E_t| = 0$, the request is shed and the cursor stays put. Round Robin needs this cursor but no service-time estimates. Assignments are evenly divided while the eligible set is fixed; changes in membership can change the shares. A slow replica keeps receiving traffic until its waiting queue reaches capacity.

Least Queue picks the eligible replica with the smallest outstanding load: waiting requests plus whatever is currently in flight,

not the length of its waiting queue alone (the simulator caps that separately, at 50 per replica, before shedding new arrivals). Least Queue routes to whichever eligible replica minimizes it, ties resolving deterministically to the lowest-numbered replica:

Outstanding load is useful but incomplete: two requests waiting behind a slow replica can cost more than three behind a fast one.

Latency Aware maintains an exponentially weighted moving average of each replica's observed service time,

and minimizes a proxy for completion cost:

The $\alpha$ parameter is the responsiveness dial: small $\alpha$ gives stable but slow adaptation, large $\alpha$ reacts fast but chases noise. See the EWMA reference [6]. The cost proxy treats each outstanding request as a full average service time. It does not estimate the active request's remaining service or individual queued token counts.

04Adaptive routing

The adaptive policy is a deliberately small UCB-inspired routing heuristic. It is not a trained model and has no claimed UCB confidence or regret guarantee. The exploration idea comes from UCB [4]. Each completion produces a reward that maps the observed service time onto [0, 1] (service time, not end-to-end latency, on purpose): queueing delay the router caused with its own past decisions is priced in separately at decision time, and folding it into the reward would also fold congestion into the estimate intended to describe replica quality:

The router keeps an EWMA estimate of reward $\bar{r}_i$ (updated exactly like the latency-aware estimate above, with the same $\alpha$) and a completed-observation count $n_i$ per replica. Assignments still waiting or in flight do not increment that count:

then scores each eligible replica as:

with $c = 0.35$ weighting exploration and $T$ the total completed-observation count across replicas. The congestion term is a queue-wait proxy: outstanding load $o_i$ times the service-time estimate $\hat{s}_i$, divided by the same ceiling $L$ that normalizes the reward. That puts all three terms on a dimensionless scale. That alone does not establish an exact latency objective: this proxy does not account for residual service, and clipping rewards at zero breaks the linear relationship for service times above $L$.

Before clipping occurs, the matched EWMA updates and initial values give $\bar{r}_i = 1 - \hat{s}_i/L$. Without exploration, the score is then $1 - (o_i + 1)\hat{s}_i/L$, a normalized version of the latency-aware proxy. This is not an exact conditional expected latency, and does not directly optimize p99 or SLO compliance.

$\bar{r}_i$ is deliberately an EWMA and not a lifetime running average. The cluster is non-stationary (a scenario can turn a replica into a straggler or bring it back mid-session), and a plain running mean over hundreds of pulls would move by roughly $1/n$ per new observation, making the policy steadily more sluggish the longer a session runs. The EWMA has fixed responsiveness per completed observation, not per unit of simulated time. A replica receiving little traffic produces few updates, so recovery can still be slow. The completed-observation count $n_i$ is not decayed the same way: it only shrinks the exploration bonus for frequently sampled replicas. A long prior history can therefore change rediscovery time after a slowdown; there is no claimed history-independent recovery bound. Combining EWMA rewards and lifetime counts is a heuristic choice, distinct from the confidence constructions used by discounted and sliding-window UCB [5].

The score itself has three terms: exploitation, exploration, and congestion avoidance. Prefer replicas that have recently served quickly, occasionally sample under-observed ones to refresh estimates, and back off any replica whose queue is growing. The accent color in the console marks this policy. Latency Aware also adapts its service estimates online.

05Failure modes

Round Robin feeds a straggler. In the Straggler scenario one replica's service time stretches. While all replicas remain eligible, it receives roughly $1/N$ of assignments. Its waiting queue grows toward the cap, after which it is skipped until space opens. Other replicas may still have spare capacity. Latency-aware and adaptive routing can shift load away; how quickly depends on completions, traffic, and prior estimates.

Greedy routing under-explores. A purely greedy version of the adaptive policy can stop sampling a replica that once looked slow, and miss its recovery while other routes remain cheaper. Congestion on alternatives can still cause a greedy policy to revisit it. The exploration term varies with $\sqrt{\ln T / n}$, so a starved replica's bonus grows if its own observation count stays fixed while total observations rise. This encourages revisiting it, without providing a fixed rediscovery deadline.

Stale estimates. Both EWMAs (latency-aware's service-time estimate and adaptive's reward estimate) only update when a replica completes work. With $\alpha = 0.15$, roughly the last 7 observations carry about 68% of the weight, and the last 19 carry at least 95%. A sudden slowdown still takes several completions to register. During that window the router keeps routing on yesterday's numbers. This is also why the reward estimate has to be an EWMA rather than a running average over the whole session: a running average would only get more resistant to change as pulls accumulate, so a straggler introduced late in a long session would be recognized far slower than one introduced early. The EWMA avoids that loss of responsiveness per observation; the time needed to collect those observations still varies.

Queue amplification. As the arrival rate approaches service capacity, waiting time becomes hypersensitive to small service-time differences (the classic queueing-theory hockey stick). Push the traffic slider up under Advanced controls and the same heterogeneity that was harmless at 100 req/s dominates the tail at 250 req/s.

06Simulation model

This is a browser simulation. It does not run inference on real GPUs, and no number it produces is a benchmark of any real system. Requests arrive as a Poisson process: exponential inter-arrival times from a seeded PRNG. That is a useful synthetic baseline with realistic short-term burstiness, not a claim that production traffic is literally Poisson. Each request carries a synthetic shape: prompt and output token counts drawn from bounded log-normal distributions, because real inference requests vary a lot and a prompt-heavy request is not the same work as a decode-heavy one. Its base service demand is

with synthetic per-token coefficients (not measured from any real GPU or service). Each replica is a FIFO waiting queue plus a single in-flight slot (a deliberately single-server abstraction, with no continuous batching). Modeled service time is three factors multiplied together: the request's demand (above) times the replica's current speed relative to a reference request, times a per-request uniform jitter of 0.9–1.1 (read at service start, so a mid-run straggler affects requests started after the onset). The 8 / 11 / 16 ms figures in the table below are that middle factor made concrete: at the default heterogeneity setting, they are exactly the mean service time each replica would produce for a reference request whose own demand equals 11 ms, not a fixed cost added to every request, and not a measurement of any real accelerator. Queues cap at 50; arrivals beyond that are shed and reported, never silently dropped. When a replica becomes unavailable, its in-flight request is allowed to finish, its waiting queue is re-routed through the router (those requests keep their original arrival time, so their latency includes the detour), and it receives no new traffic until recovered. Reported latency is always end-to-end (exact arrival to completion, queue wait included). The SLO-miss rate is conditional on completed requests: shed and unfinished requests are excluded and reported separately. In the live console, percentiles cover the last 250 completions and throughput a trailing 1000 ms window, both bounded so the readouts stay responsive to current conditions.

07Experimental methodology

Compare Policies is run as a small controlled experiment, not a screenshot of the live view. The entire workload (arrival times, token counts, runtime jitter) is drawn from the seeded PRNG at arrival time, with a fixed number of draws per arrival, before any routing decision. Every policy therefore replays byte-identical workloads (common random numbers); the only variable is the policy. Each comparison runs 10 deterministic seeds per policy, discards a 3 s warm-up to reduce initialization effects, then observes arrivals over the next 12 s. Warm-up does not guarantee equilibrium, particularly under overload. Requests arriving after warm-up through the final cutoff form a fixed measurement cohort. Its completed requests feed the nearest-rank percentiles; requests shed or still unfinished at cutoff are counted separately. Scenario events (a burst window, a straggler slowing and recovering, a failure) fire at fixed simulated times shared by all policies and seeds, so the benchmark tests reaction to change, not a cluster that was already broken at t = 0. The comparison reports per-policy medians and the observed min–max p99 range across seeds. This range is descriptive, not a confidence interval; small differences do not establish a reliable winner. Policies see no oracle information: only current queue state, health, and statistics from completed requests, never a request's sampled service time or anything about the future. And nothing in the harness favors any policy; when Round Robin or Least Queue wins a workload, the comparison says so.

For each seed, cohort arrivals equal completed plus shed plus unfinished requests, including losses during failure rerouting. SLO misses divide deadline misses by completed cohort requests; shed rate divides cohort losses by cohort arrivals. No post-cutoff draining is performed, so latency statistics are conditional on finishing by the cutoff and can underrepresent the slowest requests. Compare them alongside loss and unfinished counts. Throughput counts all completions during the measurement window, including warm-up carryover. Per-seed results expose exact counts; independently calculated median counts need not add up. A run without completed cohort requests has no latency or completed-request SLO-miss estimate, shown as an em dash and excluded from those medians and ranges.

Optimize useful completions, not just speed

The comparison also reports on-time request goodput and SLO attainment. Let $N_{\text{on time}}$ count measurement-cohort requests completed by the cutoff with end-to-end latency at most 50 ms. For a measurement duration $\Delta t$ in seconds:

All cohort arrivals stay in the attainment denominator, including shed and unfinished requests. This is success observed by cutoff, not eventual success: a pending request may still meet its deadline later. With arrivals but no successes, attainment is zero; with no arrivals, it is undefined. Goodput is zero when no on-time completions occur. A policy cannot improve attainment merely by excluding its rejected requests from the denominator.

Recent serving research makes the objective distinction explicit. JITServe [1] studies SLO-aware goodput under uncertain request information; SOLA [2] uses state-aware scheduling to balance first-token and per-token SLOs. Bari, Hegde, and de Veciana [3] distinguish throughput-optimal scheduling under stated assumptions from SLO-aware scheduling. These are modern context, not algorithms implemented here. This lab has neither token-level scheduling nor real GPU measurements; its request-level goodput is not the same metric as token goodput.

What the optimization check established

An offline study compared exploration weights 0, 0.05, 0.1, 0.2, 0.35, and 0.5 across 24 synthetic conditions: three traffic rates, two heterogeneity settings, and steady, burst, straggler/recovery, and failure scenarios. Selection used eight development seeds, maximizing mean per-run attainment with each condition weighted equally. The existing $c = 0.35$ was selected, so the coefficient is unchanged. Validation then used 24 separate seeds: 576 runs per policy. The full configuration and raw results are recorded in the repository's routing study.

On those validation runs, mean attainment was 72.42% for Adaptive and 72.39% for Latency Aware—a small descriptive difference, not evidence of universal superiority. Independent seeds do not establish generalization to new workload distributions. No confidence interval or global-optimality claim is made. The study does not justify stronger exploration, deadline-based rejection, or a claim that this simulator matches production serving systems.

Reproduce from the repository: node scripts/evaluate-routing.mjs routing-study.json. See docs/serving-validation.md for the protocol and limitations.

08Implementation

The engine is pure TypeScript with no dependencies, stepped at a fixed 0.5 ms simulated timestep with a remainder accumulator, so results do not depend on frame rate, tab throttling, or how callers chunk time; rendering reads snapshots and never writes back, and the benchmark runs the same engine headlessly, faster than real time, with no animation. The constants below are imported from the engine source. The table cannot drift from the code.

At each timestep boundary, the engine finishes service earned during the elapsed interval before starting queued work. A newly started request earns no service retroactively, so its end-to-end latency is at least its modeled service time. Starts and completions are rounded to the timestep grid: an idle request can incur less than two quanta of scheduling overhead, and a busy queue accumulates rounding across preceding requests. This is a discrete-time approximation, not an exact continuous-time solver.

A traffic change rescales the remaining inter-arrival time rather than leaving a stale arrival scheduled at the old rate. The remaining exponential hazard is preserved without drawing another random number. For old and new rates$\lambda$ in the same units:

In the browser, simulated time is dilated (1 real second ≈ 120 simulated ms) so individual requests are visible; the simulation pauses when the console scrolls out of view, and continuous animation is dropped entirely when the reader prefers reduced motion.

09What this model leaves out

Real inference serving is enormously more complicated: continuous batching, KV-cache and prefix locality, token-level scheduling, TTFT/ITL as distinct metrics, memory pressure, heterogeneous accelerators, autoscaling, admission control, network effects, request priorities, and model parallelism all shape production tail latency. None of that is modeled here. Prefill and decode exist only as cost terms in a request's service demand. There is no separate first-token timing, so the simulator reports end-to-end latency rather than inventing a TTFT it doesn't model. The routing policies also deliberately ignore the token counts they could observe: queues are compared by request count, not queued work, which keeps Least Queue an honest, simple baseline. This deliberately small model isolates one thing: the interaction between routing decisions and queueing behavior, so that the difference between the four policies is legible in a browser tab.