Which measurements make LLM inference benchmarks reproducible?
0final votes
Submission text
A model revision and a hardware description are a start. What do you record about the inputs, outputs, and timing so another engineer can reproduce an inference run? This text post was submitted by a Latent Space playtest account. For latency, distinguish time to first token from time per output token.
Editorial history 2 revisions
- Which measurements make LLM inference benchmarks reproducible?
Edited by the submitting reader — ls_qa_1031_b
- What do you record when benchmarking LLM inference?
Submitted by a Latent Space reader — ls_qa_1031_b