Arabic Voice Agents That Actually Work: Dialects, Code-Switching, and Voices

We build voice agents from Montgomery, Alabama and Manama, Bahrain. This is what we have learned about making them work in Arabic.

The first Arabic voice demo we ever ran went badly in a way we did not expect. The speech recognition was fine. The language model was fine. The voice was clear. The problem was that the caller said "ok" and "invoice" and "next week" in the middle of an Arabic sentence, and the system treated those words as noise. It answered a question nobody had asked.

That is the whole challenge in one call. Arabic voice AI is not "English voice AI with a different language setting." It is a different set of problems, and most of them show up in the first minute of a real conversation.

Here is what we have learned building Arabic voice agents for a Gulf telecom, a meeting-notes product for GCC investment teams, and a research project where every piece of the pipeline runs on local hardware.

Modern Standard Arabic is not what people speak on the phone

Most Arabic language tools are trained on Modern Standard Arabic (MSA). News broadcasts, documents, formal writing. Nobody calls a company and speaks MSA.

A customer in Manama says "شلونك" and "الحين". A customer in Riyadh says "كيفك" and "دحين". A customer in Cairo says something else again. The words for "now," "want," and "how are you" change across the Gulf, and the pronunciation shifts with them. A speech model that scores well on MSA benchmarks can still miss a Bahraini caller's third sentence.

What worked for us:

• Pick a speech-to-text provider with real Gulf dialect coverage. We have had the best results with Speechmatics for Gulf Arabic, with Deepgram as the fallback for English-heavy calls. Whisper large-v3 is a solid baseline when everything has to run locally, but it needs a good reference set to tune against.
• Build a test set from real audio, not scripts. Ten minutes of recorded calls from the actual customer base beats an hour of staff reading lines. People mumble, talk over hold music, and switch topics.
• Measure word error rate on names and numbers separately. A model can be 92% accurate overall and still get the account number wrong every time, which is the one thing that matters in a collections call.

People switch languages mid-sentence, and your agent has to follow

In the Gulf, a normal sentence sounds like "أبي أغير الـ plan حقي قبل الـ billing date." Arabic grammar, English nouns. Linguists call this code-switching. Your voice agent calls it a problem unless you plan for it.

Three things made the difference:

1. Run recognition in a mode that expects both languages. Some providers let you set Arabic as primary and English as secondary. If yours does not, you end up transcribing "billing date" as a string of Arabic letters that mean nothing.
2. Let the language model answer in the language the customer used for the question, not the language of the last word. If the customer asked in Arabic and said one English noun, the answer should be Arabic. We put this rule in the system prompt and tested it with a few hundred mixed sentences.
3. Keep product names and amounts in the form the customer expects. For our telecom work, that meant saying the currency as "دينار" and not "دينار بحريني" every single time, because that is how people actually say it. Small, but callers notice.

For Majlis, our meeting-notes product for GCC investment teams, the same code-switching shows up in a different form. A partner says "خلونا نراجع الـ IRR قبل الاجتماع الجاي" and the notes need to come out as "Review the IRR model before the next meeting." The transcript has to keep "IRR" as "IRR," and the summary has to understand that the Arabic around it is an instruction, not small talk.

The voice matters more in Arabic than in English

English text-to-speech has had a decade of polish. Arabic text-to-speech is newer, and the gap between "understandable" and "sounds like a person" is wider.

Two things we watch for:

• Diacritics. Arabic writing usually leaves out the short vowels. A text-to-speech model has to guess them, and when it guesses wrong, a word comes out mispronounced in a way that sounds foreign. Good models handle common words well. Names, product names, and anything unusual need to be spelled out with diacritics in the script, or replaced with a phonetic form.
• Register. A collections call should sound calm and respectful. A retail assistant should sound warm. The same voice model can do both, but the prompt has to tell the language model to write in that register, because the voice only reads what it is given.

We build most of our production voices on ElevenLabs, which handles Arabic well and gives us control over pacing. For the sovereign project, where no audio can leave the building, we use an open Arabic text-to-speech model running on a local GPU. It is a step behind the hosted voices in naturalness, and a step ahead on the one thing that project cares about: the audio never leaves the network.

Turn-taking is a separate problem

A voice agent has to know when the caller has finished speaking. Get it wrong one way and the agent interrupts. Get it wrong the other way and there is a long, awkward silence.

Arabic made this harder than we expected. Gulf speech has pauses in different places than English, and a caller who says "يعني..." is thinking, not done. The fixed "wait 700 milliseconds of silence" rule that works in English cut people off constantly.

What fixed it was a turn-detection model that looks at the words and the audio together, not just the silence. We use Silero for voice activity detection and a small turn-taking model on top. It is not perfect, but the agent went from interrupting on roughly one call in five to roughly one in forty.

Hand off to a person early, and say so in the caller's language

Every Arabic voice agent we have shipped has one rule above all the others: the moment a caller asks for a person, they get a person. Not after one more question. Not after a menu.

We also found that saying it clearly in Arabic matters. "راح أحولك لأحد زملائي الحين" ("I will transfer you to one of my colleagues now") lowered the number of callers who hung up during the transfer. Silence during a transfer reads as the call dropping.

What a working setup looks like

For the telecom project, the shape of the system was:

• Pipecat for the real-time audio pipeline
• Speech-to-text tuned for Gulf Arabic, with English as a secondary language
• A language model with a tightly written system prompt covering tone, currency wording, and when to hand off
• ElevenLabs for the voice
• A supervisor transfer that keeps the call connected and attaches the transcript

Every call ended with a full transcript the team could read. Before the agent, the collections team could review about five percent of calls. After, they could review every one.

The honest summary

Arabic voice AI works in production. We have run it with real customers, on real calls, about real money. But it works because of a hundred small decisions about dialect, mixed-language sentences, pronunciation, and timing, not because of one big model choice.

If you are planning a voice agent for Bahrain, Saudi Arabia, or the UAE, start with a recording of ten real calls and build the test set before you build anything else. That one habit would have saved us our first bad demo.

We build voice agents in Arabic and English from our office in Manama. If you want to hear one, talk to us.