L4 inference · measured development study

Clocks, workload,
and the energy between requests.

Paired measurements are available; latency-qualified adaptive savings have not been established.

Exploratory paired result

Observed held-out comparison: Adaptive candidate had 0.77% lower GPU energy across 8 valid matched pairs relative to the paired fixed setting; the largest TTFT and mean-ITL ratios were 1.029 and 1.009. This aggregate is descriptive; latency and selection qualification are reported separately.

Observed held-out comparison: GPU-managed clocks had 11.50% higher GPU energy across 8 valid matched pairs relative to the paired fixed setting; the largest TTFT and mean-ITL ratios were 1.077 and 1.037. This aggregate is descriptive; latency and selection qualification are reported separately.

On one 64-request code rate_2x development block, requested 660 MHz had 26.37% lower GPU energy than 210 MHz, with p95 time to first token changing from 111.45 to 26.53 seconds. Mean board power rose from 33.40 to 40.53 W; completion time fell from 238.05 to 144.45 seconds. The 660 MHz request delivered a 350.26 MHz time-weighted mean. This single-block contrast is descriptive and does not establish adaptive savings or a common winner across sources.

On one 64-request conversation rate_4x development block, requested 660 MHz had 4.04% higher GPU energy than 210 MHz, with p95 time to first token changing from 7.63 to 3.36 seconds. Mean board power rose from 39.24 to 42.28 W; completion time fell from 168.50 to 162.66 seconds. The 660 MHz request delivered a 250.19 MHz time-weighted mean. This single-block contrast is descriptive and does not establish adaptive savings or a common winner across sources.

Prefill-heavy probes: at requested 660 versus 210 MHz, whole-workload GPU energy was 31.60% lower at concurrency 1 (2 matched repetitions); 29.95% lower at concurrency 4 (2 matched repetitions). These request-shape comparisons do not isolate exact GPU phase energy.

Decode-heavy probes: at requested 660 versus 210 MHz, whole-workload GPU energy was 0.30% lower at concurrency 1 (2 matched repetitions); 0.47% lower at concurrency 4 (2 matched repetitions). These request-shape comparisons do not isolate exact GPU phase energy. Singleton decode reported 54.8–55.4 W mean board power despite the configured 40 W limit; the discrepancy is unresolved and lacks physical-meter validation.

Long-context generation probes: at requested 660 versus 210 MHz, whole-workload GPU energy was 3.33% lower at concurrency 1 (2 matched repetitions); 10.41% lower at concurrency 4 (2 matched repetitions). These request-shape comparisons do not isolate exact GPU phase energy.

The retained inventory contains 80 independently valid completed trials, 3 invalid outcomes, and 13 unstarted outcomes.

The adaptive-candidates-v3-20260906T162440Z acquisition failed start-temperature admission before any request was dispatched. Its final window was 65 °C against the unchanged 62 °C ceiling after 180.03 seconds. No serving energy, latency, or adaptive-policy result was measured; the failed and unstarted outcomes remain in the inventory.

The invalid dev_conversation_fixed_2040 record reports a watchdog interruption with 55/64 requests completed and a sampled peak temperature of 77 °C. It contributes no energy or latency comparison point; subsequent unstarted trials remain missing.

The dev_conversation_demand_phase trial in adaptive-candidates-fresh-v3-20260906T164810Z failed conditioning before any request for that trial was dispatched. Its independently verified no-serving outcome supplies no energy, latency, or policy-performance measurement. Earlier completed trials in that acquisition remain separate evidence.

40 WConfigured power-limit setting
62 °CStart-temperature ceiling
80Independently valid live trials
16Fixed-relative contrasts across 8 workload/repetition groups
01

Thermal admission

2026-09-06T18:17:00.432510 image/svg+xml Audited L4 study report 0 2 4 6 8 10 12 14 16 Wait after cooling command (minutes) Failed probe run Warm-reference run Five-clock run 0.42 min 12.65 min 0.21 min 12.97 min 1.14 min 15.00 min A shorter start rule on saved histories 62 °C admission replay Original reference selection 0 25 50 75 100 125 150 175 Wait after cooling command (seconds) 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of saved segments admitted 62 °C admits every saved segment 60 °C · 26/28 admitted 62 °C · 28/28 admitted 65 °C · 28/28 admitted
The 62 °C rule would admit all 28 recorded idle segments from 6 acquisitions within 180 seconds. The blue circles replay a ten-second post-command window; open squares mark the original reference-selection waits, including the failed 900-second attempt. The right panel is the empirical distribution over individual segments, not a population confidence interval. Earlier workload dispatch would change subsequent temperatures; this is not a simulation of a new acquisition.
Export figure PDF SVG PNG
02

Prefill and decode diagnostics

2026-09-06T18:17:00.839193 image/svg+xml Audited L4 study report 210 660 Requested graphics clock (MHz) 0 100 200 300 400 500 600 Whole-request GPU energy (J) Prefill-heavy Concurrency 1 Concurrency 4 210 660 Requested graphics clock (MHz) 0 200 400 600 800 1000 1200 Decode-heavy 210 660 Requested graphics clock (MHz) 0 250 500 750 1000 1250 1500 1750 Long-context generation 210 660 Requested graphics clock (MHz) 0 5 10 15 20 Completion time (s) 210 660 Requested graphics clock (MHz) 0 5 10 15 20 25 210 660 Requested graphics clock (MHz) 0 10 20 30 40
24 independently valid diagnostic trials. Each point is one observed repetition; connected points share request shape, concurrency, repetition, and acquisition. Unpaired valid trials remain individual points. No error bars aggregate these few repetitions. Energy covers the whole request workload through final completion; client timing does not identify exact internal prefill or decode energy.
Export figure PDF SVG PNG
03

Probe clock realization and board power

2026-09-06T18:17:01.296718 image/svg+xml Audited L4 study report Decode C=1 Decode C=4 Long context C=1 Long context C=4 Prefill C=1 Prefill C=4 Diagnostic request shape and concurrency 0 100 200 300 400 500 600 Observed graphics clock (MHz) Requested clocks need not be realized Requested 210 MHz Requested 660 MHz Decode C=1 Decode C=4 Long context C=1 Long context C=4 Prefill C=1 Prefill C=4 Diagnostic request shape and concurrency 0 10 20 30 40 50 Telemetry-derived mean board power (W) Measured draw is reported separately Configured 40 W setting
Clock dots are time-weighted means; vertical segments show temporal minima and maxima, not uncertainty intervals. Board power is audited whole-workload energy divided by elapsed time, derived from NVIDIA telemetry without a physical power meter. The dashed 40 W line is the configured limit, not an assertion that reported board draw stayed below it. These data do not by themselves identify why a requested clock was not realized or explain a discrepancy between the configured limit and reported draw.
Export figure PDF SVG PNG
04

Five-clock workload comparison

2026-09-06T18:17:01.725820 image/svg+xml Audited L4 study report 1 2 3 4 5 Worst p95 latency / audited 660 MHz baseline 5.5 6.0 6.5 7.0 7.5 8.0 GPU energy through completion (kJ) 210 MHz 660 MHz 1125 MHz 1575 MHz 2040 MHz Code · rate_2x · 5/5 settings 0.75 1.00 1.25 1.50 1.75 2.00 2.25 2.50 Worst p95 latency / audited 660 MHz baseline 6.6 6.7 6.8 6.9 7.0 GPU energy through completion (kJ) 660 MHz 210 MHz 1125 MHz 1575 MHz 2040 MHz Conversation · rate_4x · 5/5 settings
Each point is one independently valid development workload trial. The horizontal coordinate is the worse of TTFT and mean-ITL ratios to the matching source's audited 660 MHz trial; the dashed line is the 1.1 benchmark-relative limit. A panel without a selection-eligible reference instead shows raw p95 TTFT, explicitly labeled descriptive only; a failed readiness gate cannot supply a target or selection evidence. Lower energy and lower latency are preferable. Each source's plotted screen is descriptive; selecting one common fixed setting requires all five clocks to be assessed on every development workload. No uncertainty estimate is attached to a single trial.
Export figure PDF SVG PNG
05

Requested versus observed clocks

2026-09-06T18:17:02.142217 image/svg+xml Audited L4 study report 210 660 1125 1575 2040 Requested graphics clock (MHz) 0 250 500 750 1000 1250 1500 1750 2000 Observed graphics clock (MHz) Code · rate_2x 210 660 1125 1575 2040 Requested graphics clock (MHz) 0 250 500 750 1000 1250 1500 1750 2000 Observed graphics clock (MHz) Conversation · rate_4x
Dots are time-weighted means of sampled clocks across each entire workload, including idle gaps. Vertical segments show the observed minimum and maximum over time, not statistical error bars. The diagonal marks exact clock realization. Power management and thermal constraints can reduce realized clocks; clock values alone do not identify a throttle reason. The configured 40 W limit is not an assertion about the measured board-power maximum.
Export figure PDF SVG PNG

Client-visible activity under fixed 660 MHz

Source timingStreaming-only timeActual 210 during streaming1–3 streams durationActual 210 during 1–3 streams
Code rate_2x8.73%90.46%8.245 s99.88%
Conversation rate_4x64.30%88.82%9.182 s100.00%

The GPU already delivered 210 MHz through nearly all recorded streaming-only intervals with one to three outstanding requests. These client-state exposures are not exact GPU phases, policy residency, an energy-savings bound, or an irreducible power floor. The policy also uses 0.2-second observations and 2-second holds; its effect requires measured commands, clocks, latency and energy. Held-out mixtures can differ.

06

Adaptive energy and latency

2026-09-06T18:17:02.511170 image/svg+xml Audited L4 study report −1.0 −0.5 0.0 0.5 1.0 1.5 GPU energy saved versus fixed setting (%) Code 8205-8268 · repeat 1 Code 8205-8268 · repeat 2 Code 8525-8588 · repeat 1 Code 8525-8588 · repeat 2 Conversation 17152-17215 · repeat 1 Conversation 17152-17215 · repeat 2 Conversation 18560-18623 · repeat 1 Conversation 18560-18623 · repeat 2 Adaptive candidate 0.98 1.00 1.02 1.04 1.06 1.08 1.10 Worst p95 latency / paired fixed setting −30 −25 −20 −15 −10 −5 0 GPU energy saved versus fixed setting (%) Code 8205-8268 · repeat 1 Code 8205-8268 · repeat 2 Code 8525-8588 · repeat 1 Code 8525-8588 · repeat 2 Conversation 17152-17215 · repeat 1 Conversation 17152-17215 · repeat 2 Conversation 18560-18623 · repeat 1 Conversation 18560-18623 · repeat 2 GPU-managed clocks 0.98 1.00 1.02 1.04 1.06 1.08 1.10 Worst p95 latency / paired fixed setting
Every point is one independently valid matched workload and repetition. Positive energy differences favor the comparison method. Blue circles satisfy the relative 1.1 latency limit for both TTFT and mean ITL; orange crosses fail at least one. Absolute development targets and fixed-setting qualification are checked separately in the audit before any qualified savings claim. These repeated blocks are descriptive observations, not independent request-level samples or population confidence intervals.
Export figure PDF SVG PNG
07

Clock changes during serving

2026-09-06T18:17:02.981474 image/svg+xml Audited L4 study report 200 300 400 500 600 Graphics clock (MHz) Demand / phase policy Observed graphics clock Requested clock at decision 0 50 100 150 200 Time since first scheduled arrival (s) 0 5 10 15 20 25 Observed requests Awaiting first token Streaming output
Source-verified observations from heldout_rep1_conversation_b1_adaptive_selected. The trace is selected deterministically from audited valid trials, preferring an adaptive method. Requested-clock changes are plotted at recorded decision times; actuation latency remains part of measured workload energy and latency. Awaiting/streaming client states are causal observations, not exact GPU phase labels.
Export figure PDF SVG PNG
08

What the measurements support

Energy is derived from NVIDIA GPU telemetry; it does not cover the whole server or provider bill and was not checked with a physical power meter. Short diagnostic requests do not isolate exact GPU prefill/decode energy. The bounded temperature rule establishes a start ceiling, not thermal equilibrium or matched starting temperatures. All comparison methods retain the configured 40 W power limit, 6,251 MHz memory setting, and 77/79 °C watchdogs. The configured limit is not an assertion that measured board power stayed below 40 W. New causal policies are separate from the existing governor/MPC, whose calibration requirements remain unmet. Repeated blocks share trace requests and acquisitions; a small exploratory study cannot establish a population-wide savings estimate.

Invalid, unstarted, and pending outcomes are retained in the downloadable summary. They supply no plotted energy or latency value. An unmeasured result is never displayed as zero.

Methods and interpretation

Primary replay timing: Code rate_2x; Conversation rate_4x. Each case retains its source request identities and complete token lengths.

Conversation gaps are multiplied by one half relative to the original rate_2x cases; code timing is unchanged. The original conversation rate_2x thermal failure remains secondary evidence and supplies no revised latency target.

The 73 °C post-trial gate checks completion and thermal margin after the first dense conversation trial. It is distinct from the unchanged 62 °C start admission and the 77/79 °C watchdogs. A completed valid trial above 73 °C remains descriptive evidence and cannot support selection.

5 predecessor code fixed-setting trials are marked for primary reuse; the independent import audit verifies exactly the five settings. Other predecessor trials remain diagnostic or secondary evidence.

The intended comparison has 8 global paired groups per comparator. Acquisitions separated by source do not reduce that total or add independent samples.

A 900-second allowance per conditioning attempt applies to adaptive-candidates-continuation-v3-20260906T174213Z, adaptive-heldout-code-rep1-v3-20260906T182420Z, adaptive-heldout-code-rep2-v3-20260906T210116Z, adaptive-heldout-conversation-rep1-v3-20260906T193105Z, adaptive-heldout-conversation-rep2-v3-20260906T201736Z. Earlier acquisitions retain their recorded 180-second allowances and outcomes. The extension gives more time to reach the same 62 °C, ten-second admission window; it does not establish thermal equilibrium. Allowances are recorded separately for each acquisition in the report data.

Trace attribution: Azure LLM Inference Dataset 2023, released under CC BY 4.0. Credit: Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, and Ricardo Bianchini, Splitwise: Efficient generative LLM inference using phase splitting, ISCA 2024. Official release: https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md. Our adaptations select 64-request blocks, rescale arrival timing, and replay recorded token lengths using synthetic repeated-token-ID prompts.

Azure traces supply request timing and input/output token counts, not original prompt text. The runner uses synthetic repeated token-ID prompts, temperature zero, ignored end-of-sequence tokens, and exact requested output lengths, with prefix caching disabled. These measurements do not validate semantic production workloads.

The fixed baseline is one common setting selected on development workloads. It is not an oracle that selects a different clock for each held-out block. The demand/phase versus idle-only policy comparison is a development ablation; only the selected policy enters held-out evaluation.

A null savings result when realized clocks collapse to the same values cannot distinguish an ineffective policy from inadequate clock-control authority. Requested-clock labels alone do not establish a distinct physical operating point.

Download trial summary CSVDownload report dataDownload full PDF report

Self-contained report. No external fonts, scripts, tracking, or content delivery network. Figures are exported as PDF, SVG, and PNG.