Low Latency Voice: The Race to First Syllable

Voice interfaces live and die on latency. A text UI can tolerate a spinner for two seconds, but a voice assistant that pauses before answering feels broken. The number that matters most is time to first audio: how long between the user stopping speaking and hearing your first syllable.

Break the round-trip into stages. STT must start transcribing while the user is still talking, using partial transcripts instead of waiting for end-of-speech. Your application logic should act on the first confident partial, not the final one. TTS must stream its output incrementally so audio starts playing while the rest of the sentence is still being synthesized.

Caching is the hidden lever. Greetings, confirmations, and common responses can be pre-synthesized and stored, turning a 400ms generation into a 5ms lookup. For a support bot, the ten most common replies cover most calls, and those are exactly the ones users notice being slow.

Measure per-stage, not end-to-end. Instrument STT finalization time, your logic time, and TTS first-byte time separately. When a user complains about a laggy bot, the stage breakdown tells you immediately whether to blame the speech model, your code, or the network, instead of guessing.

At llm-kita, STT and TTS sit behind one OpenAI-compatible API, with streaming support on both sides. Wire both directions once, and keep every stage observable with per-request latency logs built in.