level: research
text-to-speech systems often mispronounce rare proper nouns, loanwords, and technical terms because text alone is ambiguous. even models that use phonemes lack a direct way to specify how a single word should sound. graft solves this by conditioning the pronunciation of a chosen word on a short spoken sample of that word. the audio hint is encoded with the model's own speech tokenizer and tied to the word's position in the prompt.
the system uses voice conversion during training to separate the hint speaker's voice from the target speaker's voice. this means the pronunciation hint can come from any speaker, while the output stays in the desired target voice. in a blind english listening study, human raters ranked graft first by a clear margin. they judged its rendering of difficult words as closest to a reference recording of that word.
graft was also tested on a five-language objective benchmark, showing its effectiveness across languages. the approach works with neural codec language models, which are a recent type of text-to-speech model. by adding fine-grained control, graft makes synthetic speech more accurate for names, technical jargon, and other challenging words. this reduces errors in applications like audiobook narration, voice assistants, and accessibility tools.
why it matters: accurate pronunciation of rare words is critical for voice assistants, audiobooks, and accessibility tools, and this method improves it without extra training data or speaker restrictions.