Have you ever heard a text-to-speech (TTS) voice read an English sentence perfectly, only to stumble badly on an embedded Spanish name or a Hindi phrase? This common problem highlights the central challenge of multilingual TTS: handling text that mixes languages, a phenomenon known as code-switching.
This article explains the technical reasons why even advanced voices "break" on mixed-language text. You will learn the specific steps where errors occur and how to test and control these systems for better results.
Key takeaways
- Multilingual TTS voices often fail on mixed-language text by defaulting to the primary language's accent and pronunciation rules for foreign words.
- The core challenges are accurate Language Identification (LID) for each word and correct Grapheme-to-Phoneme (G2P) conversion, which turns spelling into sounds.
- Simply supporting many languages does not guarantee quality code-switching. Performance varies hugely between language pairs and depends heavily on the training data.
- For reliable results, you often need to provide explicit language tags (e.g., using SSML) instead of relying on automatic detection.
- Evaluating a system requires testing for naturalness and correctness across pronunciation, accent, rhythm, and speaker consistency, especially with names and transliterated words.
Why Do Multilingual Voices 'Break' on Mixed-Language Text?
A multilingual TTS voice breaks on mixed-language text because it often misidentifies a word's language and applies the wrong pronunciation rules. The most common failure is when the model defaults to the dominant language’s accent for a foreign phrase, failing to apply an accent appropriate to that phrase (Source). This makes the speech sound unnatural or even unintelligible.
Code-switching is the practice of alternating between two or more languages within a single conversation or sentence. It’s common worldwide, from "Hinglish" (Hindi-English) in India to "Spanglish" (Spanish-English) in the United States. A user might write, "I need to go to the bodega to buy some pan."
A simple TTS model might process this sentence and identify English as the dominant language. When it encounters "bodega" and "pan," it may not recognize them as Spanish words. Instead, it attempts to pronounce them using English sound rules, resulting in an awkward, anglicized output.
The model fails because it lacks the context or training data to handle the switch gracefully. It has learned to associate certain letter patterns with specific sounds in one language, and it applies that knowledge incorrectly to another. The system understands the letters but misunderstands the linguistic context.
How Multilingual TTS Handles Code-Switching: A Step-by-Step Breakdown
To understand why multilingual TTS fails, you need to look at its internal pipeline. An error at any early stage will cascade, leading to a flawed audio output. The entire process is a complex journey from text to sound, and you can learn more about the fundamentals in our guide on how text-to-speech works.
Here are the key steps and where code-switching can go wrong:
1. Text Normalization
The system first cleans the input text, expanding abbreviations, numbers, and symbols. For example, it turns "Dr." into "Doctor" and "$10" into "ten dollars." This stage is language-dependent. Is "1/2" pronounced "one half" (English) or "un demi" (French)? An incorrect initial language guess causes errors before pronunciation even begins.
2. Language Identification (LID)
This is the most critical step for code-switching. The system must identify the language of the text. There are two main approaches:
- Utterance-level LID: The model assigns a single language to the entire sentence. This is simple but fails badly for code-switching, as it forces all words into one language's pronunciation model.
- Word-level LID: A more advanced model identifies the language of each word or phrase. This is essential for good code-switching but is much harder to do correctly, especially with short or ambiguous words.
3. Grapheme-to-Phoneme (G2P) Conversion
Once the language is identified, the system converts the written letters (graphemes) into sound units (phonemes). A phoneme is the smallest unit of sound that distinguishes one word from another. This mapping is language-specific. For example:
- The letters "ch" are pronounced differently in church (English), Loch (Scottish), and Bach (German).
- The letter "j" sounds different in jump (English), jalapeño (Spanish), and Jean-Paul Jacques (French).
If the LID step assigned the wrong language, the G2P converter will select the wrong phonemes, leading to a completely incorrect pronunciation.
4. Linguistic and Acoustic Prediction
The model takes the phoneme sequence and predicts acoustic features like spectrograms or speech tokens. In modern neural systems, factors like language, accent, and speaker identity are often learned together (Source). This entanglement can cause issues; for example, weak language conditioning in the acoustic model can cause pronunciation leakage from the dominant language, even if the phonemes are correct. It also means that switching the language might unintentionally alter the speaker's voice or accent.
5. Vocoder Synthesis
Finally, a vocoder synthesizes the acoustic features into an audible waveform. The vocoder creates the human-like sound you hear. However, it cannot fix mistakes made earlier. If the acoustic model predicted the wrong features or the G2P conversion was wrong, the vocoder will faithfully synthesize the wrong sounds, producing perfectly clear but incorrect speech.
6. Evaluation
The final step in the pipeline is assessing the output. A robust evaluation tests for more than just intelligibility. For code-switching, it must check for both language correctness and naturalness at the switch boundaries. Key criteria include pronunciation accuracy, appropriate accent, consistent rhythm and speaker identity, and robustness to spelling variations, transliteration, and names.
Worked Example: Synthesizing a Hinglish Sentence
Let's walk through a concrete example to see how different systems handle a common Hinglish sentence.
Input Text: "Please confirm your OTP, aapka code hai 1234."
This sentence starts in English and switches to Romanized Hindi ("your code is 1234").
Scenario A: A Naive TTS System
This system uses simple, utterance-level language detection.
- LID: The system scans the whole sentence, sees more English words than Hindi, and tags the entire utterance as
English. - G2P Conversion: It now applies English pronunciation rules to every word.
- "Please confirm your OTP" is fine.
- "aapka" is read as "ap-kuh" or "a-a-p-k-a."
- "code" is fine.
- "hai" is read as "hay."
- Output Audio: The result is a heavily anglicized and unnatural-sounding sentence that is difficult for a Hindi speaker to understand. The voice sounds confident but is completely wrong.
Scenario B: A Sophisticated TTS System
This system uses advanced, word-level language detection.
- LID: The system analyzes the text word by word. It correctly identifies the language of each segment:
[en, en, en, en, hi, hi, hi, en]. - G2P Conversion: It applies the correct G2P model for each language.
- It uses the English G2P model for the English words.
- It uses the Hindi G2P model for "aapka," "code," and "hai," generating the correct phonemes for a native Hindi pronunciation. Transliterated text can still introduce ambiguities, but a well-trained model can handle common cases (Source).
- Output Audio: The result is a natural-sounding sentence. The speaker switches smoothly from English to Hindi and back, just as a bilingual human speaker would.
This example shows that high-quality code-switching depends almost entirely on the system's ability to perform accurate, word-level language identification and apply the correct pronunciation rules.
Key Approaches to Code-Switching TTS: A Comparison
Engineers use several strategies to build multilingual TTS systems. Each approach has different trade-offs in terms of complexity, data requirements, and the quality of the final output. The scale of data required is significant; one large-scale benchmark for code-switching included 420,000 text samples across 12 languages and over 80 hours of audio from 174 speakers representing 63 racial or ethnic backgrounds (Source).
| Approach A | Approach B | Key Difference |
|---|---|---|
| Utterance-Level Conditioning | Word-Level Conditioning | Utterance-level is simpler but causes accent leakage. Word-level is better for code-switching but needs more precise language detection and training data (Source). |
| Concatenated Monolingual Synthesis | End-to-End Multilingual Synthesis | Concatenation stitches audio from separate systems, which can create audible breaks. An end-to-end model learns smooth transitions but requires mixed-language training data (Source). |
| Implicit Accent Learning | Explicit Accent Control | Implicit models learn accent as part of the voice, making it hard to change. Explicit controls allow you to select the language and accent separately, offering more flexibility (Source). |
| Natural Code-Switched Data | Synthetic Code-Switched Data | Natural recordings are authentic but scarce. Synthetic data can fill gaps but may contain unnatural patterns. For example, one study used synthetic data to reduce an ASR system's mixed error rate from 12.1% to 10.1%, but these figures evaluate the downstream ASR task, not universal TTS quality (Source). |
What Are the Common Pitfalls of Multilingual TTS?
When evaluating or implementing a multilingual TTS system, it's easy to make false assumptions. Here are common pitfalls to avoid:
- Assuming language support equals code-switching quality. A system might support 100 languages but only handle code-switching well for the 5 most common pairs it was trained on.
- Relying on automatic language detection. For short, ambiguous, or technical text, automatic detection is unreliable. Names and loanwords are frequently misidentified.
- Ignoring accent control. A system might offer a Hindi language option but only with an Indian English accent. Accent is a distinct feature from language, involving different pitch patterns and sound realizations (Source).
- Failing to test with transliterated text. Many users write languages like Hindi, Urdu, or Arabic using the Latin alphabet. Your TTS system must be tested with this kind of input, not just native scripts.
- Overlooking proper nouns and brand names. These are often the first words to be mispronounced. A good system needs a way to handle them, like a custom lexicon.
- Not checking for speaker consistency. When switching languages, some models can cause the speaker's voice to change in pitch or quality, breaking the illusion of a single speaker.
How Can You Control Language and Pronunciation?
Instead of relying on imperfect automatic detection, you can take explicit control over the TTS engine's output. This is the most reliable way to achieve high-quality results for mixed-language content.
Use Speech Synthesis Markup Language (SSML)
SSML is a standard XML-based markup language that lets you control aspects of speech synthesis like pronunciation, rate, pitch, and volume. For multilingual text, its most important feature is the <lang> tag.
You can wrap any word or phrase in a <lang> tag to tell the engine exactly which language to use. The tag uses standard BCP-47 language codes (e.g., en-US for American English, hi-IN for Indian Hindi).
Here is how you would mark up our Hinglish example for perfect pronunciation:
<speak>
Please confirm your OTP, <lang xml:lang="hi-IN">aapka code hai</lang> 1234.
</speak>By explicitly tagging the Hindi phrase, you remove all ambiguity. The TTS engine is no longer guessing; it is being instructed to use its Hindi G2P and acoustic models for that specific segment.
Use Custom Pronunciation Dictionaries
For words that are consistently mispronounced—such as brand names, technical jargon, or unique family names—you can use a custom dictionary, also known as a lexicon. Most professional TTS platforms allow you to upload a lexicon where you map a specific word to its desired phonetic spelling. This gives you fine-grained control over your brand's voice and terminology.
When Should You Not Rely on Automatic Code-Switching?
While automatic code-switching is improving, there are situations where its unreliability poses too great a risk. You should avoid relying on it in the following cases:
- High-stakes applications. For medical instructions, emergency alerts, or financial information, any mispronunciation is unacceptable. Use explicit SSML tags to guarantee correctness.
- Brand voice consistency. If your brand requires a specific accent or delivery, automatic systems may introduce unwanted variability. Explicit controls are necessary to maintain a consistent persona.
- Text with many ambiguous words. If your content contains a high density of names, neologisms, or loanwords that could belong to multiple languages, automatic detection will likely fail.
- Short, fragmented text. Language detection models need context. For very short phrases or single words (like in a navigation app or user interface), there isn't enough information to make a reliable guess.
In these scenarios, the work of manually tagging your text with SSML is a small price to pay for accuracy and reliability.
What to learn next
- SSML Prosody: A Practical Guide to Controlling TTS - Learn how to use SSML prosody to control text-to-speech (TTS) output. This guide covers how to adjust pitch, rate, and pauses using tags lik Understanding the pitfalls of multilingual TTS is the first step toward building better voice experiences. To continue your journey, consider exploring these related topics:
- Challenges in Speech Recognition: Understanding the challenges of recognizing mixed-language speech provides a complete picture of the problem from both the synthesis and recognition perspectives. Learn more about the challenges of ASR for Indian languages.
- Modern TTS Architectures: Modern TTS architectures are better equipped to handle the complexities of multilingual speech. Dive deeper into the models that power today's most advanced voices by reading about codec language model TTS.
- Speaker Identification: In real-world conversations where code-switching occurs, identifying 'who spoke when' is another key challenge. Explore how AI solves this with our guide to speaker diarization.
Sources and further reading
- Code-Switching Text-to-Speech Synthesis
- Language and Accent-specific Prompting for Controllable Code-switched TTS
- A Controllable Text-to-Speech System for Generating ASR Training Data for Code-Switching Speech
- CS-TTS: An End-to-End Code-Switching TTS with Cross-Lingual Language Model
- A Controllable, Multi-Accent, Multi-Speaker Text-to-Speech System
- A Large-Scale Code-Switching Speech Synthesis Dataset with High-Quality Textual and Acoustic Information
- Acoustic-Prosodic, Articulatory, and Perceptual Correlates of Regional and Foreign Accent in Speech
- TACL: A Unified Transliteration-based Framework for Multi-Accent Speech Synthesis
