Accent and Emotion Control in Zero-Shot Text-to-Speech

Summary: Text-to-speech (TTS) systems these days are incredibly powerful. With just three seconds of someone's recorded voice, these systems can clone that voice to say anything - without the need to train new models. This technology is already making a difference in people's lives. For example, individuals with speech disorders can record their voices while they are still able to speak and later use TTS to "speak" in their own voice. It's also being used to create personalised audiobooks such as bedtime stories in parents' voices for kids, or to dub movies into different languages to reach wider audiences.
But despite these exciting possibilities, TTS systems come with serious challenges around fairness and inclusivity. When asked to mimic the voice of someone with a less common accent, like Scottish or Nigerian English, these systems often get it wrong. Instead of keeping the original accent, they produce a mix of more common accents - like General American or Southern English accents. We term this problem accent hallucination.
There is also another issue with regard to explainability and controllability. These systems sometimes generate speaking styles that introduce emotions not present in the original voice, causing the speaker to sound either overly dramatic or unduly understated, potentially altering the intended meaning. We refer to this phenomenon as emotion hallucination.
This PhD research project aims to make TTS more inclusive, controllable, and transparent by looking at accent and emotion control in Zero-Shot TTS. Everyone deserves to be heard - not just in any voice, but in their voice, with their accent and intended style. As speech technology becomes more embedded in our daily lives, we believe it must represent the full diversity of how people speak, and do so responsibly.