Block’s new voice model sounds eerily human

Jack Dorsey’s company, Block, has introduced a new voice AI model named Dessa that operates locally and demonstrates a distinctively human-like quality, incorporating expressions such as laughter, pauses, and emotional tones. The model is full-duplex, enabling it to listen and speak simultaneously, a feature also seen in OpenAI's GPT-Live model. The development matters because it signals a push towards more natural, on-device conversational agents that don't rely on cloud processing for their core function.

Dessa's Architecture and On-Device Operation

According to Block, Dessa was trained entirely on human speech using a single NVIDIA GPU rack. The company claims the model's natural sound is a direct result of avoiding an intermediate text-to-speech pipeline, an architecture that it argues strips away crucial emotional information. Unlike models from firms like ElevenLabs which often use tags to guide emotional delivery, Block states that emotion is an inherent quality of Dessa's output. The model’s knowledge base is derived directly from its conversational training data rather than being paired with a separate large language model for information retrieval.

• Full-Duplex: Capable of simultaneous listening and speaking for more natural conversational flow.
• Local Processing: Designed to run on consumer-grade computers like a MacBook M4 Max, reducing latency and reliance on cloud infrastructure.
• End-to-End Speech: Avoids converting voice to text, which Block claims preserves emotional nuance.
• Integrated Knowledge: Relies on its conversational training data for knowledge, rather than an external LLM.

Strategic Implications for Block and the AI Market

While the technology is notable, Block has not released Dessa to the public, instead issuing a call for safety researchers to probe the model for potential misuse, such as creating deceptive voice interactions. This cautious approach highlights the risks associated with highly realistic voice agents. For Block, which focuses on financial services and small business solutions through products like Square, an in-house, low-cost, on-device voice model could significantly enhance its offerings, from automated ordering systems to more sophisticated customer service experiences, without incurring substantial cloud computing costs.

By developing an on-device, emotionally expressive voice model, Block is not just entering the generative AI race; it's building a vertically integrated tool to create a competitive moat in its core fintech and commerce businesses. Bypassing the cloud and text-based intermediaries points to a strategy focused on lower operational costs, reduced latency, and a more intuitive user interface, which could redefine user experience in point-of-sale and customer support applications.