Towards Efficient Tone-Aware Discrete Speech Representations

Tone distinctions are essential for lexical meaning in many world languages, yet they remain underrepresented in current self-supervised speech representation frameworks. Discrete speech units derived from models like HuBERT through unsupervised quantization are particularly susceptible to tone loss, as traditional methods such as k-means clustering prioritize phonetic structure over tone variation. This thesis investigates the extent to which discrete representations from
self-supervised models preserve lexical tone in tonal languages such as Mandarin and Yoruba. Through a series of probing experiments, we demonstrate that tone information is degraded during standard quantization, leading to reduced accuracy in tone-sensitive downstream tasks. To address this, we propose a series of tone-preserving strategies, including Residual Vector Quantization (RVQ), Finite Scalar Quantization (FSQ), and pitch-aware fine-tuning of HuBERT. These
approaches are evaluated through classification and reconstruction tasks, with results showing that deeper RVQ hierarchies and FSQ might perform better comparably similar with k-means in retaining tonal distinctions without significantly increasing representation complexity. Finally, we apply our tone-aware representations to Automatic Speech Recognition (ASR) and propose future extensions to Automatic Diacritic Restoration (ADR), demonstrating practical improvements
for speech technologies in low-resource tonal languages. Our findings highlight the importance of incorporating prosodic structure into speech tokenization pipelines and provide a path toward more inclusive and linguistically improved speech models.