What if the only thing you changed was the language of your question and your AI gave a completely different answer? That’s the hidden challenge of bilingual prompting, and it’s a growing concern as teams begin using large language models (LLMs) for critical decision-making.

LLMs are increasingly used in multilingual environments, but they aren’t always tested that way. When a model is given input in one language and a question in another, its reasoning can break down, quietly and unexpectedly. New research shows that this small shift in how we prompt AI systems can lead to significant drops in accuracy, revealing a gap in how we evaluate and deploy them across global use cases.

The Multilingual AI Blind Spot

Imagine this scenario: a global company rolls out an AI tool to scan contracts and flag legal risks. In English, it works flawlessly. But when the same tool is used on contracts in Spanish, with legal questions asked in Arabic or Turkish, the results become inconsistent. Risk assessments change. Outputs feel off. And yet, the content hasn’t changed; only the language of the question has. It’s a reminder that multilingual performance can’t be taken for granted, even in systems that seem reliable.

New research from our Research Labs at Welo Data reveals that LLMs don’t always reason equally across languages. Specifically, when models receive prompts that mix two languages, a technique known as bilingual prompting, their accuracy drops. Subtly in some cases. Significantly in others.

If you’re deploying LLMs in multilingual environments, especially for decision-making tasks like customer support, contract review, or clinical reasoning, you need to understand how language affects model behavior.

This blog explores findings from the paper “Diagnosing Performance Gaps in Causal Reasoning via Bilingual Prompting in LLMs“, offering practical guidance for leaders and product teams working with AI across borders.

Why Language Mixing Matters for Your Business

In many real-world settings, people don’t interact with AI in neatly siloed language environments. A customer writes a complaint in Korean, and support follows up in English. A medical system receives patient history in Arabic and a follow-up query in French. A team translates a report into Spanish but prompts ChatGPT with questions in English.

These scenarios aren’t exotic, they’re normal. Yet most AI testing workflows only check models in English or a single local language.

This research introduced bilingual prompts across six languages: English, Spanish, Japanese, Korean, Arabic, and Turkish, and tested eight leading LLMs on four causal reasoning tasks. Each task required the model to infer or explain cause-and-effect relationships in bilingual story-question pairs.

Key findings:

• Bilingual prompts reduced model accuracy by 4.6% on average.
• The question language affected performance more than the story language.
• English did not always produce the best results, despite its advantage in training data.

This is not about fluency or translation. It’s about reasoning. And that makes it a new category of risk for global AI applications.

Three Insights Every Leader Should Know

1. The Question Language Affects the Answer

One of the strongest findings was that models were more influenced by the language of the question than by the language of the content. This means two people can ask the same question about the same input, but if one asks in Korean and the other in English, the model might give different answers.

Why? A phenomenon known as the recency effect. In bilingual prompts, the last language the model “sees” (the language of the question) has disproportionate influence over its reasoning and response.

Business implication: In multilingual workflows, your AI system’s output may depend more on the language of the question, even when the content stays the same. If a legal assistant tool gives different advice based on whether the question is asked in English or Arabic, that’s not just inconsistent, it’s risky.

2. English Isn’t Always the Best Bet

LLMs are usually trained on massive English datasets. So it’s natural to assume English prompts will lead to the most accurate answers. But the research shows this assumption doesn’t hold.

This is likely due to learned biases during training. In English, models were more likely to say “no” when asked whether one event caused another, even when “yes” was correct. This aligns with prior research showing that LLMs exhibit a negative bias in binary reasoning tasks, often defaulting to “no” when uncertain.

Business implication: Relying on English prompts may reinforce hidden model biases. Language diversity in testing may actually improve system reliability.

3. How Reasoning Task Type Influences AI Performance

Another key insight from the study: performance drops from bilingual prompting aren’t uniform. The impact depends heavily on the type of reasoning the model is asked to perform.

Researchers evaluated four distinct reasoning tasks across eight leading LLMs:

• Causal Discovery – Cause: Determining whether one event directly caused another
• Confounder Detection: Determining whether one event did not directly cause another  
• Language Variation: Testing the model’s ability to recognize differently worded prompts that mean the same thing (e.g., caused vs. led to)
• Norm Violation Detection: Identifying which social, moral, or legal violation most directly contributed to an event

The key takeaway? Bilingual prompting affected each task differently.

For example:

• Causal Discovery – Cause was especially sensitive to prompt language. In some cases, English questions performed worse than random guessing.
• Confounder Detection showed smaller but consistent drops, suggesting that reasoning about confounders is vulnerable to prompt phrasing.
• Language Variation exposed how question language significantly influenced accuracy, with Korean and Turkish prompts yielding lower performance.
• Normality Violation was the most stable across languages, though question language still influenced performance.

Why This Matters for Businesses:

If you’re using AI to support decision-making, whether in legal, clinical, or compliance settings, both the type of reasoning task and the language of the prompt can significantly affect accuracy.

Some tasks are more vulnerable to language shifts than others, which means testing across reasoning types is just as critical as testing across languages.

How to Build a Language-Aware AI Strategy

Most organizations don’t evaluate their AI systems for cross-language reasoning. But given the growing reliance on LLMs across sectors, this is an emerging blind spot.

Start With These Questions:

• Do we use AI systems where prompts and data might be in different languages?
• Have we tested model outputs across language pairs?
• Are key decisions being made based on AI outputs in non-English prompts?
  

Watch For These Warning Signs:

• Unexpected differences in AI outputs by geography or language
• Lower performance in translated or localized environments
• Confusing or contradictory reasoning when switching languages
  

What to Do:

• Add bilingual prompt tests to QA workflows for critical use cases
• Run scenario-based evaluations using your company’s real multilingual data
• Prioritize multilingual validation in high-risk areas (legal, healthcare, finance)
• Document known biases in your model’s behavior across language combinations
  

This doesn’t require a complete overhaul; it just requires a shift in how you test and monitor your systems.

The Path Forward

Language bias in AI is measurable, and if left unchecked, can undermine user trust, decision quality, and global product performance.

This new research doesn’t just highlight a problem. It offers a practical diagnostic: use bilingual prompts to test how robust your model’s reasoning is across languages.

For companies deploying AI in global contexts, this is more than a technical detail. It’s a chance to lead by building systems that are inclusive, transparent, and reliable for everyone, no matter what language they speak.

Want to Learn More?

This blog shares the big takeaways. For full details on the dataset, methods, and statistical modeling, check out our original research paper: Diagnosing Performance Gaps in Causal Reasoning via Bilingual Prompting in LLMs.