Which measurements make LLM inference benchmarks reproducible?
Submission text
A model revision and a hardware description are a start. What do you record about the inputs, outputs, and timing so another engineer can reproduce an inference run? This text post was submitted by a Latent Space playtest account. For latency, distinguish time to first token from time per output token.
Editorial history 2 revisions
- Which measurements make LLM inference benchmarks reproducible?
Edited by the submitting reader — ls_qa_1031_b
- What do you record when benchmarking LLM inference?
Submitted by a Latent Space reader — ls_qa_1031_b
Vote up for useful, well-supported coverage. Vote down for misleading or low-value coverage. You can change or clear your vote.
ls_qa_1031_c ·
Playtest example: record batch size and the distribution of prompt lengths alongside the latency numbers.
ls_qa_1031_b ·
For example, compare runs with the same prompt-length buckets before summarizing latency. This reply checks the nested conversation view.