A model clears its multilingual evaluation suite. Scores hold across every target language. The program ships into four new markets, and within a few weeks two of them are producing a different story: answers that are accurate and somehow wrong, a register that reads curt or oddly formal, confident responses to questions the model has misread outright.

Nothing broke. The evaluation measured what it was built to measure. It was built by the same cohorts, against the same guidelines, with the same assumptions about a good answer as the training data it was scoring. An eval cannot surface a gap its authors share.

This is the most expensive pattern in multilingual AI training data, and it rarely gets diagnosed as what it is. It presents as a model problem, so teams respond with more data, more languages, more evaluation volume. The pattern survives all three, because volume was never the variable.

There is now enough published evidence to say that precisely rather than anecdotally.

The arithmetic that makes this a first-order problem

Microsoft’s AI Diffusion Report put generative AI usage at 18.8% of the world’s working-age population in June 2026, against a working-age population of roughly 5.3 billion. More than 1.2 billion people have used an AI tool at least once in under three years.

The same report splits current usage 28.8% in the Global North against 16.2% in the Global South. Around 5% of internet users are in the United States.

The remaining growth is therefore overwhelmingly non-English. Our reading of those figures puts close to 90% of the incremental adoption opportunity outside English, which is our inference rather than a published statistic. A frontier lab’s valuation is an implicit bet on model behavior in the languages its evals interrogate least.

The instrument has the same blind spot as the data

In April, Microsoft Research published an audit of 51 multilingual benchmarks covering 242 datasets and 219 languages. Its central finding: 56% of dataset-language instances are translated from English rather than natively constructed, which the authors note introduces artifacts and English-centric framing.

Alongside that, 36% of evaluated languages appear in only one benchmark. Low-resource languages are assessed on one to three task categories where high-resource languages get fourteen. And contamination runs through the set via translated benchmark leakage and parallel corpus overlap that evades surface-form detection.

The authors’ recommendation is evaluation that is natively constructed, culturally grounded and contamination-aware. Read that as a research-side argument for in-market origination, and for treating benchmark design as something you build for a domain rather than adapt from a public set.

Four failure modes produce most of the degradation. Each is detectable before a model ships, and each shows up in procurement as a question most buyers do not currently ask.

Failure mode one: translated English wearing local vocabulary

The cheapest route to multilingual training data is to build in English and move it. Sometimes that is explicit translation. More often it is structural: prompts authored in English, task instructions derived from English, annotation guidelines whose hard cases were reasoned through in English, evaluation rubrics that encode English conventions and then score another language against them.

Two distinct things go wrong, and the second is the expensive one. The first is surface: the output is linguistically correct and behaviorally hollow, and a native speaker notices. The second is that translation can only ever give you an English speaker’s questions rendered in another language. It cannot tell you what a user in São Paulo or Jakarta would have asked in the first place, which tasks they would bring to a model at all, or how they would phrase a complaint when something went wrong. A reward model trained on that signal learns to prefer the wrong response with high confidence, which is worse than learning nothing.

The failure is invisible to the standard metric set, because the metrics derive from the same source. Agreement is high. Gold tasks pass. The pipeline is consistent, and consistently off. On the Microsoft audit numbers, this is the majority case rather than the exception.

The strongest evidence that translation will not carry this work comes from the researchers building the hardest multilingual benchmarks. PolyWorkBench states that machine translation is never used to produce a source-language file. Documents are used verbatim or drafted from scratch in the target language by the same annotator, with ground-truth facts back-injected as verifiable anchors so no task can be solved by string-copying.

What surfaces it: ask where the prompts in each language originated, and what share of the set is translated. Translated means English data in local packaging.

Culture Goes Beyond Translation

Another theme throughout the conversation was the importance of cultural understanding. Organizations often request support for broad language categories such as Arabic or Spanish, but MK explained that successful AI models require much greater precision. Dialects, regional vocabulary, and cultural expectations all influence how users experience AI. A model trained for one region may not resonate with users somewhere else. 

For that reason, Welo Data emphasizes detailed assessments to understand contributors’ regional expertise, dialect knowledge, and specialization before matching them to projects. Client discussions also begin by identifying where a product will launch and how users will interact with it, allowing training to reflect the appropriate audience and use case. 

The result is AI that better reflects the cultures and communities it is designed to serve. 

Failure mode two: in-language coverage without in-market depth

The second failure is more sympathetic and similarly costly. Contributors speak the language. They are not in the market, have not been for some years, or are evaluating for a region whose conventions diverge from their own.

Regional variation inside a language is where models embarrass their owners, and it is also where most pipelines average the signal away. Two cohorts from different markets disagree, the disagreement reads as inconsistency, and the quality system resolves it to a majority. The majority encodes one market’s conventions and the model carries them everywhere.

In-market depth means origination, adjudication and calibration happening inside the market being served, with cross-market disagreement adjudicated rather than averaged. Welo Data runs this across 155+ locales from 8 global regions, with 14 secure facilities where program sensitivity requires them. The complexity is the point: once the work has to hold across dozens of markets simultaneously, the operational surface grows faster than most pipelines are built to absorb.

What surfaces it: ask for market distribution of contributors within each language rather than the language count. Then ask what happens when two markets disagree.

Failure mode three: a training signal that is not verifiably human

First, the honest framing. This is not a multilingual problem. Fraud and substitution affect every data pipeline in the industry, English-only ones included, and nothing about working in another language makes a contributor more likely to misrepresent themselves. Any vendor implying otherwise is selling you a story about the people doing the work rather than a description of the risk.

What is specific to multilingual programs is that verification gets harder as language and market depth increase, and the usual shortcuts cost you the coverage you were paying for. Confirming that someone is a Marathi speaker currently living in Pune is a different operational problem from confirming a US-based English speaker. Restricting to easily verified markets, to a small set of payment rails, or to contributors holding one particular kind of documentation quietly collapses coverage back toward the markets that needed the least help. That is the real tension, and it is an operational one.

The risk that matters most for Gen AI programs is a submission produced by a model and presented as human judgment. It is plausible, fast, internally consistent, and the precise inverse of what the task exists to generate. A reward model then trains on model output while the program believes it is training on people. For RLHF and evaluation work that corrupts the training signal itself, and the standard metric set will not flag it, because gold tasks and inter-annotator agreement measure consistency and a well-organized fraud is extremely consistent.

NIMO is what we point at here: behavioral monitoring across every session rather than scoring final output alone, with the audit trail a governance team needs to show how a dataset was produced and by whom. It was recognized at the 2026 AI Excellence Awards for fraud detection and prevention. We would present it as the specific thing we have built rather than as a solved problem for the category, which it is not. The standard worth holding is identity established continuously through a program rather than at intake, with detection designed for model-generated submissions.

What surfaces it: ask how contributor identity is re-established in week nine. Then ask how model-generated submissions are detected, and what has actually been caught.

Failure mode four: judgment with no apparatus around it

The fourth failure is structural. A program hires capable people, issues an instruction document, and scores output against a rubric. What it omits is the apparatus that converts individual judgment into a reliable signal: calibrated examples for the hard cases, a defined escalation path for ambiguity, and feedback that develops judgment rather than scoring compliance.

Without it, every contributor privately invents a policy for the hard cases and the aggregate is noise shaped like data. The task definitions say the work requires judgment. The operating model treats it as execution, and the data reflects the operating model.

Our VP of Global Operations and Quality, MK Blake, wrote about this gap in MultiLingual Magazine. Titles like annotator, labeler and rater describe mechanical work, when the job done properly is closer to applied linguistics at scale. Organizations that hire, compensate and support against the mechanical description get the mechanical result, and then discover it in production.

This is what our quality systems and human-in-the-loop model are built around: gold tasks and calibration sets per market, root cause analysis on disagreement rather than suppression of it, and adjudication that returns to the cohort so the next thousand examples carry the resolution. The same apparatus is what made a dual-track AI safety evaluation workable across policy compliance and hallucination auditing simultaneously.

What surfaces it: ask to see the calibration set for the hardest category in one target language. Ask what happens to a contributor’s hard call after they make it.

Long-horizon work compounds all four

PolyWorkBench, from Beijing Jiaotong University and Tencent’s Weixin AI, built 67 multi-step workflow tasks across ten languages in July, each running six to twelve tool-use steps, and evaluated eight models across eighteen model-and-harness combinations.

The spread inside a single model is the finding. A strong model holds 0.82 to 0.96 across all ten languages, while mid-tier models swing more than 30 grade points between best and worst, one configuration scoring 0.305 on German against 0.926 on Chinese. The authors’ conclusion is that language variation is a genuine axis of failure rather than an additive penalty on general capability.

Their failure taxonomy names the mechanism as cross-lingual coordination errors: comprehension is correct, but source-language content drifts out of alignment with target-language output across the trajectory, so the final artifact drifts even though every intermediate step looked plausible. For anyone debugging a multilingual agent, that is the sentence to internalize. The trace reads clean and the deliverable is wrong, which is why agentic evaluation has to assess intermediate reasoning steps rather than final output alone.

MAPS, accepted to EACL 2026 findings, translated four established agentic benchmarks into eleven languages, producing 805 tasks and 9,660 language-specific instances. Both performance and security degrade moving out of English, with severity correlating with how much of the input was translated. The security half of that deserves more attention than it currently gets in red teaming conversations. One aside for anyone running automated multilingual quality gates. PolyWorkBench reports that its LLM-as-judge score correlates at r = 0.18 with the executable grade, and that correlation collapses to roughly zero once you condition on tasks that passed. If your multilingual quality gate is one model scoring another model’s output, that is the correlation you are relying on, and it is the argument for human-in-the-loop evaluation at the points that matter.

What the three conditions cost you

Underneath the four failure modes sit three conditions. In-market origination, which governs what goes in. A verifiably human signal, which governs whether what goes in is what you think. And traceability to model behavior, which is a measurement condition rather than a data one: name the language, the task shape, the metric and the delta, or the spend was overhead.

Each is easy to claim. The difficulty is that they pull against each other. Verification done crudely shrinks coverage toward the markets easiest to verify, which is the opposite of in-market depth. Chasing language count pushes toward availability over adjudication capability. Optimizing for a single model metric rewards whatever data moves that number this quarter.

Holding all three steady across dozens of markets at once is an operational discipline, and it is where providers actually differ.

Six diligence questions

For any multilingual data provider, ours included:

1. Is my evaluation data built by the same cohorts, against the same guidelines, as my training data?
2. Where did the prompts in each language originate, and what share of the set is translated rather than natively authored?
3. What is the market distribution of contributors within each language?
4. How is contributor identity re-established in week nine rather than week one?
5. How do you detect model-generated output submitted as human judgment, and what have you caught?
6. What did your last quality intervention change about model behavior, in which language, on which task shape?

Question one catches more than the rest combined.

Why it matters beyond the pipeline

There is a comfortable version of this conversation about AI changing everything and multilingual mattering differently now. It is true and it helps nobody.

The sharper version is the arithmetic. AI usage sits at 28.8% in the Global North and 16.2% in the Global South, and nearly all the remaining growth is in languages the benchmarks cover worst and measure with translated English. Whether AI reaches people is substantially a question of whether models behave well in the languages those people actually use. Nobody should be disadvantaged in what AI can do for them by virtue of not having been born speaking English. That outcome runs through training signal, reward models and evaluation sets, which is unglamorous and is where it is decided.

Working these questions through a live program? Bring us the two languages where your evals look fine and your users disagree. We will walk through how we would originate, verify and calibrate them. Talk to our team or read how other programs handled it.