First — What Is an AI Voice, and Why Are We Testing It?
An AI voice (also called a TTS model — Text-To-Speech) is software that reads any written text aloud in a human-like voice. You have heard it everywhere. The Google Maps navigation voice? That is TTS. The Siri voice on iPhone? TTS. The automated voice when you call a bank's helpline? Also TTS.
Many big companies — Google, OpenAI, ElevenLabs, Microsoft, Cartesia, and others — have built their own AI voice products. Each one sounds a little different. Some sound very natural. Some sound robotic. Some handle Hindi well. Some do not.
VoiceArena is a platform that tests these AI voices against each other using real human listeners — not machines — to decide which one sounds better. Think of it like a blind taste test, except instead of food, people are listening to voices.
Imagine you are a judge on a singing show like Indian Idol. Two contestants sing the same song. You listen to both — you do not know their names or backgrounds. You pick who sounded better, and you write down exactly what went wrong with the other one. VoiceArena does this same thing, except the "contestants" are AI voice products, and 44,387 people are the judges.
The Problem With Just Looking at Rankings
VoiceArena already publishes a ranking — which AI voice is number one, number two, and so on. But a ranking alone is like a cricket scorecard that only shows the final score. It tells you who won. It does not tell you why they won or what the losing team did wrong.
The more valuable information was hidden in the written comments. Nearly half the people who voted (21,756 out of 44,387) also wrote a sentence or two explaining exactly what they heard that was wrong. We read all those comments, grouped them by type, and counted how often each type of problem came up.
The result is a ranked list of the 6 most common ways AI voices go wrong. We call this the defect list — the specific mistakes that make people choose one voice over another.
Think of Zomato restaurant reviews. The star rating tells you the restaurant is rated 3.2 out of 5. But it is the written reviews that tell you why — "biryani was cold", "waiter was rude", "chicken was undercooked". Those written complaints are far more useful to the restaurant owner than the star rating alone. Our analysis does the same thing for AI voices.
"Knowing your AI voice ranked 5th is useless. Knowing that 40% of the people who disliked it said it mispronounced words — and that in Japanese, nearly half of all complaints were about pronunciation — is something you can actually fix."
The 6 Most Common Ways AI Voices Go Wrong
These six categories came directly from what 21,756 real listeners wrote in their comments. Each percentage tells you how often that specific type of complaint appeared out of all the complaints collected.
When it happens most: Names of people and places, technical words, and sentences that mix two languages (like an English word inside a Hindi sentence).
Why it matters: This is the single biggest complaint across all 6 languages and all 9 AI voice products. Nearly 4 in every 10 complaints were about this. It is the number one problem the AI voice industry needs to fix.
Where it is worst: English and Vietnamese listeners complained about this the most. One specific AI product — OpenAI's voice — had 36 out of every 100 complaints about it being robotic. That is the highest robotic rate of any product in the test.
Why it matters: People stop trusting a voice that sounds fake. Even if every word is correct, a robotic tone makes listeners uncomfortable and they will prefer the competitor.
Example: Imagine someone reading "Please meet me at the station... tomorrow morning" but pausing after "the" instead — "Please meet me at the... station tomorrow morning." It sounds strange and breaks the listener's concentration.
Interesting finding: Cartesia's AI voice had this as its number one complaint (24 out of every 100 complaints about Cartesia were about bad timing). This is the most model-specific finding — it points to a specific design issue in that product.
Example: You typed "The meeting is at 3 PM." The AI read "The meeting is at 3 thousand PM" — adding the word "thousand" that was never there. Or it added a complete extra phrase at the end that you never wrote.
Why this is the most serious problem: This is not a quality issue. This is a reliability issue. If an AI voice reads a medical instruction and adds or changes words, it could cause real harm. Any listener who notices this immediately loses trust in the product — and will almost always prefer the other voice.
Worst offender: xAI's Grok voice product had the highest rate of this problem. About 11 in every 100 complaints about Grok were about it adding words that were not there.
Example: You wrote "Please do not press the button." The AI read "Please do press the button." — it dropped the word "not" completely. The sentence now means the exact opposite of what you wrote.
Why this is also very serious: Just like problem #4 above, this is not about sound quality — it is about accuracy and trust. A voice assistant that drops words cannot be used in any important situation. Arabic language listeners experienced this the most in this dataset.
Interesting finding: ElevenLabs — one of the top-ranked products — had the highest rate of this problem among the competitive products (15 in every 100 ElevenLabs complaints). This suggests a technical issue in their audio processing pipeline, separate from how natural or accurate the voice sounds.
Why even a small rate is a big deal: Any evaluation where the listener hears a glitch is almost always a loss. There is no forgiveness for this. It is like a singer hitting a wrong note on the highest note of the song — it ruins everything.
Two things that were claimed but are NOT in the data
Some earlier discussions about AI voice quality mentioned "emotional mismatch" (the voice sounds emotionless) and "accent problems" (sounds like a foreigner speaking Hindi) as major categories. When we looked at the actual complaint data from 44,387 real listeners, neither of these appeared as structured complaint categories. We have not included them in this list because they are not supported by the actual data. Good research only reports what the data actually shows.
How Each AI Voice Product Fails Differently
Every AI voice product has its own specific weak point. Think of it like cricketers — every batsman has a different weakness. One gets out on short-pitch deliveries. Another struggles against spin. The table below shows each product's main problem and how often it wins head-to-head against other voices.
| AI Voice Product | Biggest complaint about it | Second biggest complaint | Win rate (how often it beats the other voice) |
|---|---|---|---|
| Google Gemini 3.1 Flash | Sounds robotic (28.4%) | Mispronounced words (25.9%) | 53.8% — best in the test |
| ElevenLabs v3 | Mispronounced words (27.7%) | Crackling / audio glitches (15.0%) — highest noise rate among top products | 41.6% |
| Sarvam Bulbul-V3 (Hindi only) | Mispronounced words (24.1%) | Sounds robotic (23.6%) | 33.3% |
| Cartesia Sonic-3 | Bad timing / weird pauses (24.3%) — this product's signature problem | Mispronounced words (24.0%) | 24.8% |
| MiniMax Speech 2.8 | Sounds robotic (24.3%) | Mispronounced words (24.0%) | 24.3% |
| OpenAI gpt-4o-mini-tts | Sounds robotic (36.2%) — highest robotic rate of any product | Mispronounced words (20.6%) | 23.8% |
| Fish Audio S2 Pro | Sounds robotic (28.8%) | Bad timing / weird pauses (26.0%) | 22.4% |
| xAI Grok TTS | Mispronounced words (25.2%) | Severely mispronounced (23.0%) — and highest hallucination rate | 21.9% |
| Microsoft Azure Dragon | Sounds robotic (30.8%) | Mispronounced words (24.5%) | 18.6% — lowest in the test |
Table 1 — Each AI voice product's specific problem areas, from 44,387 human listening tests. Percentages are out of that product's own complaints.
The clear winner is Google Gemini 3.1 Flash TTS — it wins more than half of all head-to-head comparisons. It is the benchmark that other products are being compared against. Even Google's product, however, still has complaints about sounding robotic and mispronouncing words — showing there is still room to improve even at the top.
Does the Same Problem Exist in Every Language?
No — and this is one of the most important findings. Different languages have very different main problems. A product that works well in English might fail badly in Japanese. A problem that is rare in English might be very common in Arabic.
Think of a chef who is excellent at making Italian food. If you ask them to make a South Indian thali, they might struggle with completely different problems — different spices, different techniques, different expectations from the people eating it. AI voice products have the same challenge across languages.
| Language | Biggest problem in that language | Gap between best and worst product | Google's win rate in this language |
|---|---|---|---|
| English | Robotic voice (33%) + bad timing (30%) — pronunciation is actually the least bad here | 27% gap | 41% |
| Hindi | Robotic voice (27%) + mispronunciation (24%) | 35% gap | 52% |
| Japanese | Mispronounced words (49%) — nearly half of all complaints! Pronunciation is the dominant issue. | 55% gap — biggest gap in the entire dataset | 63% |
| Arabic | Mispronounced words (52%) — similar to Japanese, pronunciation dominates | 42% gap | 61% |
| Portuguese | Robotic voice (30%) + bad timing (15%) | 34% gap | 50% |
| Vietnamese | Robotic voice (33%) | 35% gap | 51% |
Table 2 — The main problem differs by language. Japanese and Arabic are primarily pronunciation problems. English is primarily a quality/naturalness problem.
The most surprising finding here is Japanese. If you only looked at the global data, you might think "robotic voice is the biggest problem." But Japanese listeners said something very different — nearly 1 in 2 complaints about Japanese AI voices was about pronunciation. The AI voices are saying Japanese words wrong. That is a completely different problem that needs a completely different solution.
The India finding — Sarvam Bulbul-V3
Sarvam is an Indian AI company that built a voice product specifically for Hindi. In this test, it was only evaluated in Hindi (2,473 tests) and won about 1 in 3 comparisons (33.3%). Its main problem areas — pronunciation and robotic voice — are the same as the global average. For companies building voice products specifically for Indian languages, this is a direct, honest benchmark of where an India-focused product currently stands against the global competition.
The Two Most Serious Problems (Problems 4 and 5)
Problems 1, 2, and 3 above are quality problems. The voice sounds bad, feels unnatural, or has bad timing. These are annoying, but they are improvement opportunities.
Problems 4 and 5 — making up words and skipping words — are in a completely different category. These are accuracy and trust problems.
Imagine if Google Maps occasionally added or removed a turn from your directions. Not very often — maybe 1 in 12 trips. You might not even notice at first. But the moment you notice it once, you stop trusting it completely. That is exactly what hallucination and missing words do to an AI voice product.
For companies using AI voices in important settings — medical instructions, legal documents, financial information, educational content — these two problems need to be fixed before anything else. A beautiful-sounding voice that occasionally changes the meaning of sentences is worse than a slightly robotic voice that is always accurate.
What Should AI Voice Companies Do Next?
Why Listening to Real People Beats Computer Testing
There are automated computer programs that test AI voice quality — they give a score based on acoustic measurements and technical analysis. These are useful for fast testing. But they have a fundamental limitation.
A computer cannot tell you that it noticed the word "thousand" was added. A computer cannot tell you that the pause felt wrong at that specific point in the sentence. A computer cannot tell you that the Hindi voice sounds "like it was trained on non-Indian data."
Only a real human listener, writing down what they actually heard, can give you that signal. The 21,756 written comments in VoiceArena's data contain the most detailed, honest, and useful feedback on AI voice quality available anywhere. This analysis makes that feedback easy to act on — by converting 44,387 individual human opinions into six clear categories of problems, ranked by how serious they are.
How we did this analysis: We analysed 44,387 real human votes collected on VoiceArena across 6 languages (English, Hindi, Japanese, Arabic, Portuguese, Vietnamese). Each vote involved a real person listening to two AI voices and picking the better one. About 49% of voters also wrote a comment explaining their choice. We extracted structured complaint labels (called defect tags) from these comments and counted how often each type of complaint appeared. All percentages in this article come from that count. The 9 AI products tested were: Google Gemini 3.1 Flash TTS, ElevenLabs Eleven v3, Sarvam Bulbul-V3, Cartesia Sonic-3, MiniMax Speech 2.8 HD, OpenAI gpt-4o-mini-tts, Fish Audio S2 Pro, xAI Grok TTS, and Microsoft Azure Dragon HD Omni. Sarvam was only tested in Hindi.