How We Made a Text-to-Speech Model Respond in Sub-50 ms
Researchers have developed a custom implementation for the Qwen3-TTS model that achieves sub-50 ms time-to-first-audio latency on a single NVIDIA H100 GPU. The project aims to provide a high-performance, cost-effective alternative to existing commercial text-to-speech services.
Why it matters
Reducing latency in real-time AI voice synthesis is critical for creating natural, responsive human-computer interactions in voice assistants and customer service bots.
Our Qwen3-TTS 1.7B CustomVoice implementation achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM.
We compare five implementations: ours, vLLM-Omni, SGLang-Omni △ , VoxServe, and M*, under Poisson open-loop traffic. After tuning each implementation for low-latency streaming, ours is the only one to achieve sub-50 ms p95 TTFA . We maintain sub-50 ms p95 TTFA through 10 RPS and keep it below 100 ms even at 20 RPS .
Our system produces approximately 630 characters per second at 10 RPS. At $4.29 per hour for a 1 H100 SXM instance, this translates to ~$2 per 1M characters at full utilization 1 . For comparison , ElevenLabs V3 is $100 / 1M and Cartesia Sonic 3.5 is $49 / 1M at a higher TTFA .
We open source the implementation and benchmark . Our methodology is explained below.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in