The Best AI Model Won’t Win Voice. The Fastest Loop Will.
Generative AI’s first commercial form resembled search: type a request, wait while it crosses the internet to a giant model, read the answer. A few seconds felt miraculous. Voice abolishes that tolerance. Human conversation runs on timing – interruption, hesitation, back-channel murmurs – and conversation research across languages puts the typical gap between speaking turns at roughly 200 milliseconds. An assistant that answers half a second late feels awkward; several seconds late, and dialogue collapses into walkie-talkie turn-taking. For leaders building voice products, that changes what “best” means: the constraint that decides the experience is not a benchmark score but where the computation physically happens.
The industry ran a natural experiment on exactly this question in 2025–26, in the same product category: live translation. Apple’s Live Translation, across Messages, phone calls, FaceTime, and AirPods, runs entirely on device – Apple’s support documentation says so explicitly, framing privacy as the benefit, though the architectural consequence matters as much: no round trip to a data centre just to understand each utterance. Google went the other way. Gemini 3.5 Live Translate, released in June 2026, is a cloud-streamed speech-to-speech model covering more than seventy languages and preserving a speaker’s tone – capabilities no phone-sized model matches – yet it translates continuously while running, by design, a few seconds behind the speaker, and Google DeepMind’s own model card treats that lag as a critical quality dimension. Each company bought a different half of the ideal product.
Which is why the edge-versus-cloud framing is too binary. The winning architecture is hybrid. The device should own the fast loop: speech detection, interruption handling, common translations, local context, and a small language model for immediate responses. The cloud should become an escalation layer for what genuinely exceeds the handset’s thermal and memory budget – deep reasoning, rare language pairs, fresh information. Google itself is building both halves: alongside its cloud model it ships LiteRT-LM, an open-source framework for running language models directly on phones, browsers, and even Raspberry Pi–class devices, with GPU and NPU acceleration and function calling for fully offline agents.
Hybrid design has a second, underrated benefit: data minimisation becomes an architectural property rather than a policy promise. A voice assistant need not stream a continuous recording of someone’s surroundings because one difficult sentence requires cloud intelligence; a capable local model can decide which fragment – or which abstracted representation – escalates. The constraint is capability: small models will not match frontier systems indefinitely, and the simultaneous-translation literature spends much of its effort trading quality against lag. But for conversational products the best model does not automatically produce the best experience. A three-point gain on a reasoning benchmark is commercially irrelevant if the customer keeps asking whether anyone is still there.
Three implications for product and technology leaders. First, set a latency budget as a first-class requirement – an experienced-response-time target the architecture must meet – before selecting any model. Second, design the routing explicitly: define which requests the on-device model owns, which criteria trigger escalation, and log the split. Third, evaluate candidates on the experience, not the leaderboard: measure time-to-first-word under real network conditions. Edge AI may prove one of the few arenas where smaller models hold a structural advantage over larger ones. They are already in the room.
Personal analysis. Views are mine, not my employer's.