Which measurements make LLM inference benchmarks reproducible?
A model revision and a hardware description are a start. What do you record about the inputs, outputs, and timing so another engineer can reproduce an inference run? This text post was submitted by a Latent Space playtest account. For latency, distinguish time to first token from time per output token.
Sign in to voteStory details & reporting
ls_qa_1031_c ·
Playtest example: record batch size and the distribution of prompt lengths alongside the latency numbers.
ls_qa_1031_b ·
For example, compare runs with the same prompt-length buckets before summarizing latency. This reply checks the nested conversation view.