Inference

Researchers demonstrate that AI inference on standard GPUs can reach speeds of 3,000 tokens per second, rivaling dedicated inference hardware, by optimizing the software stack through architecture/engine/kernel co-design.