Jack Dorsey introduces Dessa, an AI model that learns through speech

Dessa, developed by an AI research team at Block, takes a different approach to conversational artificial intelligence by learning directly from human speech rather than relying on conventional large language models and text-based processing.

Jack Dorsey, co-founder of Twitter, now known as X, has drawn attention to a new artificial intelligence research project called Dessa, which aims to make machine conversations more natural by learning directly from spoken language. Developed by a research team at Block, Dorsey’s financial technology company, the model departs from the approach used by many mainstream voice assistants.

Conventional voice AI systems typically convert a user's speech into text, process the words using a language model and then generate a spoken response. While effective, this method can lose conversational signals such as laughter, sarcasm, emotional shifts and pauses that help people interpret meaning.

Dessa instead uses a speech-to-speech approach, learning from human conversations without relying on a traditional large language model (LLM). Its developers say this architecture could offer a different way to build AI systems capable of understanding and responding to the nuances of spoken communication.

Real-time conversations without waiting for speech to end

One of Dessa’s distinguishing features is its ability to listen and speak simultaneously, a capability known as full-duplex communication. Unlike conventional voice assistants that generally wait for a user to finish before responding, the model can acknowledge speech, interrupt naturally and react while a conversation is still unfolding.

The system can reportedly use brief conversational responses such as “yeah” and “got it”, while also incorporating laughter into its replies when appropriate. Its developers say it can recognise emotional cues, sarcasm and changes in vocal tone that might be lost when speech is converted into written text.

Demonstrations shared by the team on its website and social media have showcased the model’s conversational behaviour, attracting attention from observers interested in alternative approaches to AI development.

The developers have also highlighted its local computing capabilities. According to the team, Dessa can run on a MacBook Pro equipped with an M4 Max chip or a more powerful machine, without requiring an internet connection for its operation. The model was trained using a single NVIDIA GB200 NVL72 system, according to information released by its creators.

Early research opens new questions for AI development

Dorsey confirmed the model’s underlying approach in responses to questions on X. When asked which language model powered its voice capabilities, he described it as “speech to speech. no llm.” He also confirmed that the system could operate locally.

The project remains at an early research stage, with its developers inviting a limited group of AI safety researchers to test the technology and identify potential risks or weaknesses.

Dessa’s approach raises broader questions about whether AI systems can develop language understanding and learn factual information primarily through sound rather than text-based training. Its progress could offer insights into alternative methods of building conversational systems.

However, the project’s long-term capabilities, reliability and practical applications remain to be established. For now, Dessa represents an experimental effort to rethink how machines listen, interpret speech and participate in human conversations.