With the rising integration of speech front-ends and enormous language fashions (LLM),
there’s a must discover architectures that combine these modalities.
Whereas end-to-end fashions have been explored extensively, cascaded fashions that stream outputs from LLMs to TTS appear to be oddly under-explored, regardless that they’re doubtlessly a lot easier.
Utilizing conventional text-to-speech programs to transform LLM outputs to audio, nonetheless, poses a technical drawback as a result of they want total utterances to generate sytlistic audio.
On this paper we current a ‘streaming’ TTS that may generate audio from streaming textual content utilizing a novel decoder-only structure that interleaves textual content and speech.
The mannequin is educated utilizing next-step prediction on interleaved information that’s generated from force-alignment of textual content transcripts to speech.
Duing inference our system processes textual content incrementally whereas producing constant speech output, making it appropriate for real-time purposes like conversational AI brokers the place an LLM can stream textual content to a TTS system.
Outcomes reveal that our strategy matches the standard of batch TTS programs whereas enabling streaming capabilities.
