A model performs well across its multilingual evaluations, and the team moves ahead with a launch in four new markets. Within weeks, problems start to emerge in two of them. The answers are technically accurate, but the tone feels unnatural, sometimes too formal or unexpectedly blunt. In other cases, the model confidently answers questions it hasn’t fully understood. 

These issues weren’t necessarily missed because the evaluations were poorly designed. The problem may be that the training data and evaluation benchmarks were developed using similar guidelines, assumptions, and definitions of what constitutes a good response. When both reflect the same cultural or linguistic blind spots, strong evaluation scores can create a false sense of confidence. 

We see this challenge in multilingual training data, particularly when evaluation methods fail to reflect how people actually use language in different markets. When problems surface after deployment, teams often respond by collecting more data, adding languages, or running additional evaluations. Those steps can be useful, but they won’t necessarily uncover weaknesses already embedded in the evaluation process itself. 

The Evals Share the Blind Spot of the Data 

In April, Microsoft Research published an audit of 51 multilingual benchmarks covering 242 datasets and 219 languages. Its main finding is that 56% of dataset-language instances are translated from English instead of built natively, which the authors say brings in artifacts and English-centric framing. 

The audit also found that 36% of evaluated languages appear in only one benchmark, and that low-resource languages are tested on one to three task categories where high-resource languages get fourteen. Contamination runs through the set as well, in the form of translated benchmark leakage and parallel corpus overlap that surface-form detection doesn’t catch. 

The authors recommend evaluation that is natively constructed, culturally grounded and contamination-aware. We read that as a research-side case for in-market origination, and for treating benchmark design as something you build for a domain instead of adapting from a public set. 

Most of the degradation traces back to four failure modes. Each one can be caught before a model ships, though few buyers know to ask about them during procurement. 

Failure Mode One: Translated English with Local Vocabulary 

The cheapest route to multilingual training data is to build it in English and carry it over. Sometimes that means explicit translation. More often the English is built into the structure: prompts written in English, task instructions derived from English, annotation guidelines whose hard cases were reasoned through in English, and rubrics that encode English conventions and then score another language against them. 

Two problems follow, and the second is more costly. On the surface, the output can be linguistically correct and still feel hollow to a native speaker. Underneath, translation can only give you an English speaker’s questions in another language. It can’t tell you what a user in São Paulo or Jakarta would have asked to begin with, which tasks they would bring to a model, or how they would phrase a complaint. A reward model trained on that signal learns to prefer the wrong response with high confidence, which is worse than learning nothing. 

Standard metrics don’t show this, because they come from the same source. Annotators agree with one another and gold tasks pass, so the pipeline looks healthy while being consistently off. Given the Microsoft audit numbers, this is the majority case rather than the exception. 

PolyWorkBench offers some of the strongest evidence that translation can’t carry this work. Its authors state that machine translation is never used to produce a source-language file. Documents are used verbatim or drafted from scratch in the target language by the same annotator, and ground-truth facts are injected back as verifiable anchors so that no task can be solved by copying strings. 

What surfaces it: Ask where the prompts in each language came from and what share of the set is translated. Translated data is, in effect, English data in local packaging. 

Culture Goes Beyond Translation 

Culture matters as much as vocabulary. Clients often ask for broad categories like Arabic or Spanish, but as our VP of Global Operations and Quality, MK Blake, explains, successful models need far more precision. Dialect, regional vocabulary and cultural expectations all shape how users experience AI, and a model trained for one region may not resonate in another. 

That is why Welo Data starts with detailed assessments of contributors’ regional expertise, dialect knowledge and specialization before matching them to projects. Client conversations also begin with where the product will launch and how people will use it, so the training reflects the right audience and use case. The result is AI that better reflects the cultures and communities it is designed to serve. 

What surfaces it: ask where the prompts in each language originated, and what share of the set is translated. Translated means English data in local packaging.

Culture Goes Beyond Translation

Another theme throughout the conversation was the importance of cultural understanding. Organizations often request support for broad language categories such as Arabic or Spanish, but MK explained that successful AI models require much greater precision. Dialects, regional vocabulary, and cultural expectations all influence how users experience AI. A model trained for one region may not resonate with users somewhere else. 

For that reason, Welo Data emphasizes detailed assessments to understand contributors’ regional expertise, dialect knowledge, and specialization before matching them to projects. Client discussions also begin by identifying where a product will launch and how users will interact with it, allowing training to reflect the appropriate audience and use case. 

The result is AI that better reflects the cultures and communities it is designed to serve. 

Failure Mode Two: In-Language Coverage Without In-Market Depth

The second failure is easier to sympathize with, and just as costly. Contributors speak the language but aren’t in the market, haven’t been for years, or are evaluating for a region whose conventions differ from their own. 

Regional variation within a language is where models tend to embarrass their owners, and it is also where most pipelines average the signal away. Two cohorts from different markets disagree, the disagreement reads as inconsistency, and the quality system resolves it by majority. The majority encodes one market’s conventions, and the model carries them everywhere. 

In-market depth means origination, adjudication and calibration all happen inside the market being served, with cross-market disagreement adjudicated instead of averaged. Welo Data runs this across 155+ locales in 8 global regions, with 14 secure facilities where a program’s sensitivity calls for them. That scale is demanding: once the work has to hold across dozens of markets at once, the operational load grows faster than most pipelines are built to handle. 

What surfaces it: Ask for the market distribution of contributors within each language, not just the language count. Then ask what happens when two markets disagree. 

Failure Mode Three: A Training Signal That Isn’t Verifiably Human 

Before going further, it’s worth saying that this isn’t a multilingual problem. Fraud and substitution affect every data pipeline in the industry, English-only ones included, and nothing about working in another language makes a contributor more likely to misrepresent themselves. A vendor who suggests otherwise is telling a story about the people doing the work instead of describing the risk. 

What is specific to multilingual programs is that verification gets harder as language and market depth increase, and the usual shortcuts cost you the coverage you’re paying for. Confirming that someone is a Marathi speaker currently living in Pune is a different operational problem from confirming a US-based English speaker. Limiting a program to easily verified markets, a small set of payment rails, or contributors with one kind of documentation quietly pulls coverage back toward the markets that needed the least help. That is the real tension, and it comes down to operations. 

The risk that matters most for generative AI programs is a submission written by a model and presented as human judgment. It is plausible, fast and internally consistent, which is the opposite of what the task exists to produce. A reward model then trains on model output while the program believes it is training on people. For RLHF and evaluation work, that corrupts the training signal itself. Standard metrics won’t flag it, because gold tasks and inter-annotator agreement measure consistency, and a well-organized fraud is very consistent. 

NIMO is the tool we’ve built for this. Instead of scoring only the final output, it monitors behavior across every contributor session, and it gives governance teams an audit trail showing how a dataset was produced and by whom. It was recognized at the 2026 AI Excellence Awards for fraud detection and prevention. We see it as one answer to the problem, not a fix for the whole industry, because fraud in AI data is far from solved. The standard to aim for is verifying identity throughout a program, not just at intake, with detection designed to catch model-generated submissions. 

What surfaces it: Ask how contributor identity is re-established in week nine. Then ask how model-generated submissions are detected and what has actually been caught.

Failure Mode Four: Judgment with No Structure Around It 

The fourth failure is structural. A program hires capable people, hands them an instruction document and scores their output against a rubric, but leaves out everything that turns individual judgment into a reliable signal. That includes calibrated examples for hard cases, a defined escalation path for ambiguity, and feedback that builds judgment instead of just scoring compliance. 

Without that, every contributor quietly invents a policy for the hard cases, and the aggregate ends up looking like data while behaving like noise. The task definition says the work takes judgment, while the operating model treats it as execution, and the data ends up reflecting the operating model. 

MK Blake wrote about this gap in MultiLingual Magazine. Titles like annotator, labeler and rater describe mechanical work, when the job done properly is closer to applied linguistics at scale. Organizations that hire, pay and support people against the mechanical description get mechanical results, and find out in production. 

Our quality systems and human-in-the-loop model are built around this: gold tasks and calibration sets for each market, root cause analysis of disagreements instead of suppressing them, and adjudication that goes back to the cohort so the next thousand examples carry the resolution. The same setup made a dual-track AI safety evaluation workable across policy compliance and hallucination auditing at the same time. 

What surfaces it: Ask to see the calibration set for the hardest category in one target language. Ask what happens to a contributor’s hard call after they make it. 

Long-Horizon Work Compounds All Four 

PolyWorkBench, from Beijing Jiaotong University and Tencent’s Weixin AI, built 67 multi-step workflow tasks across ten languages in July. Each runs six to twelve tool-use steps, and eight models were evaluated across eighteen model-and-harness combinations. 

The most telling result is the spread within a single model. A strong model holds 0.82 to 0.96 across all ten languages, while mid-tier models swing more than 30 grade points between their best and worst language. One configuration scored 0.305 in German against 0.926 in Chinese. The authors conclude that language variation is a real axis of failure, not a flat penalty on general capability. 

They name the mechanism cross-lingual coordination errors. The model comprehends each instruction correctly, but source-language content drifts out of alignment with target-language output over the course of the task, so the final artifact is wrong even though every intermediate step looked plausible. For anyone debugging a multilingual agent, this is worth remembering: the trace reads clean while the deliverable is wrong, which is why agentic evaluation has to assess intermediate reasoning steps and not only the final output. 

MAPS, accepted to EACL 2026 Findings, translated four established agentic benchmarks into eleven languages, producing 805 tasks and 9,660 language-specific instances. Both performance and security degrade outside English, and severity tracks how much of the input was translated. The security half deserves more attention than it gets in red teaming conversations. 

One more point for anyone running automated multilingual quality gates. PolyWorkBench reports that its LLM-as-judge score correlates with the executable grade at only r = 0.18, and that the correlation falls to roughly zero once you look only at tasks that passed. If your quality gate is one model scoring another model’s output, that is the correlation you’re relying on, and it is the argument for human-in-the-loop evaluation at the points that matter. 

Three Conditions Underneath the Failure Modes 

Three conditions run through all four failure modes. The first is in-market origination, which governs what goes in. The second is a verifiably human signal, which governs whether what went in is what you think it is. The third is traceability to model behavior, which is about measurement and not data: name the language, the task shape, the metric and the delta, or the spend was overhead. 

Keeping all three steady across dozens of markets at once takes operational discipline, and that is where providers really differ. 

Six diligence questions

For any multilingual data provider, ours included: 

1. Is my evaluation data built by the same cohorts, against the same guidelines, as my training data? 
2. Where did the prompts in each language originate, and what share of the set is translated rather than natively authored? 
3. What is the market distribution of contributors within each language? 
4. How is contributor identity re-established in week nine rather than week one?
5. How do you detect model-generated output submitted as human judgment, and what have you caught? 
6. What did your last quality intervention change about model behavior, in which language, on which task shape? 

Why It Matters Beyond the Pipeline 

People like to say AI is changing everything and that multilingual matters more than ever, but that doesn’t tell you much. 

Here are the numbers that do. According to Microsoft’s AI Diffusion Report, generative AI usage is 28.8% in the Global North and 16.2% in the Global South. Much of the growth still ahead is likely to come in languages that benchmarks cover least well and test with translated English. Whether AI actually reaches those users comes down to how well models work in the languages they speak. Nobody should get less out of AI because they didn’t grow up speaking English. Who gets what is largely settled in the training signal, the reward models and the evaluation sets, so that’s where the work has to happen. 

Working these questions through a live program? Bring us the two languages where your evals look fine and your users disagree. We will walk through how we would originate, verify and calibrate them. Talk to our team or read how other programs handled it.