Benchmarking LLM Inference at Scale with AIPerf

You spin up a model. Tokens start streaming. Then hits the awkward silence of the real question: Is this actually fast, or is it just breathing?

Usually, engineers default to habits born of desperation. We hack together an asyncio loop, fire off curl commands in a bash script, or vibe-code another one-off utility that falls apart the moment concurrency climbs past three. Python's Global Interpreter Lock steps in, your measurement use becomes the bottleneck, and suddenly you are benchmarking your own terrible script instead of vLLM. It is embarrassing, but we have all done it. We build rickety testing shacks and wonder why they blow over in a windstorm.

Enter AIPerf. NVIDIA built it as a ground-up replacement for GenAI-Perf, and thankfully, they threw out the old architecture entirely. Instead of a single choking process, it uses a multi-process model wired together with ZMQ. Worker processes hammer the server while isolated record handlers crunch the results asynchronously, meaning the testing tool itself finally gets out of the way.

What I appreciate — and this matters — most here is the refusal to fake reality. You get fifteen other endpoint types, support for public datasets like ShareGPT, and trace replay formats pulled straight from output environments like Mooncake and Baseten. More importantly. You get fifteen other endpoint types, support for public datasets like ShareGPT, and trace replay formats pulled straight from output environments like Mooncake and Baseten. More importantly. By the way, you can shape arrival patterns using Poisson or gamma distributions rather than just blasting a flat, unrealistic stream of requests — surprisingly enough.

Drop the custom scripts. Setup a local loop with a tiny model, dial in your traffic shapes, and measure what your hardware can actually endure before everything catches fire.