How to measure AI inference energy
Eight steps that turn "how much energy does this model use?" into a figure someone else can check.
The eight steps
- Fix the task set. Choose versioned tasks that represent the real workload.
- State the quality gate. Decide what counts as a passing answer; only passing tasks count.
- Declare the meter point. Chip, board, rack or facility. Comparisons must match.
- Use a proper analyzer. At M1 to M3, a SPEC-accepted power analyzer controlled through PTDaemon, with clocks synchronized by NTP (Tschand et al., MLPerf Power).
- Run a ranging pass. Find peak voltage and current, then fix the ranges for the measured run.
- Measure long enough. At least 60 seconds per window; loop short tasks.
- Close the ledger. Check that the accounts balance with the closure rule.
- Report fully. Decision Yield with meter point, accuracy, label and grade.
Why on-board counters are not enough
Counters on chips and boards are convenient but can drift from true power. JL-26 counts them as Measured only after they are validated against an analyzer on the same system; otherwise they are Estimated.
Why per-query figures disagree
Published "energy per query" estimates vary widely mainly because they use different boundaries, tasks and quality bars. Some count only the accelerator; others include the host, networking or the facility. The benchmark comparison shows what each benchmark includes.
Questions
Can we measure in the cloud?
Often only with on-board telemetry, which is Estimated unless validated. A rack-level measurement at M3 needs physical access or the provider's metered data.
Sources
Keep reading
Take the next step.
How to measure the energy of AI inference properly: task set, quality gate, meter point, analyzer, ranging, 60-second windows, closure check and reporting.