Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

TL;DR: we show that AI inference on GPUs can be super-fast, reaching the speed regime of dedicated inference hardware cards when optimizing the whole software stack with architecture/engine/kernel co… [+23436 chars]