Welo Data partnered with a global leader in artificial intelligence research and development to execute a 4-week multilingual pilot evaluating large-language-model performance across text and speech tasks. 

The engagement spanned 20 locales and four advanced evaluation workflows — Error Rewrite, Listening Comprehension, Preference Ranking, and Speaking — producing 36,000+ high-quality annotations by culturally fluent Trainers. 

Through rigorous human-in-the-loop quality systems, native-language training, and adaptive milestone design, Welo Data delivered measurable improvements in accuracy, consistency, and cross-lingual performance. 

The pilot validated Welo Data’s ability to mobilize expert contributors rapidly while maintaining strict auditability and quality alignment — establishing a foundation for long-term multilingual evaluation at enterprise scale. 

The Challenge

A leading AI Lab required a highly specialized, globally distributed team to execute four advanced evaluation workflows — each testing a different aspect of large language model (LLM) behavior — across 20+ locales and under aggressive timelines. 

Success depended on flawless instruction-following, single-pass precision (no rework), and consistency across linguistic and cultural contexts. 

The project also demanded rapid sourcing of qualified Trainers, precise calibration across time zones, and emerging frameworks for spoken-task evaluation, where few industry standards existed. 

The Approach

Welo Data deployed a 4-week multilingual evaluation program across 20 locales, combining LLM-specific training modules, live calibration sessions, and native-language audits to ensure precision, cultural fidelity, and scalability. 

Four Core Workflows 

1. Error Rewrite / Discovery 

Designed to test a model’s self-correction and text-generation fidelity. 

Trainers identified factual or grammatical errors in model outputs, explained each correction, and rewrote the passage accurately in the target language. 

This workflow strengthens SFT (Supervised Fine-Tuning) and RLHF (Reinforcement Learning from Human Feedback) pipelines by providing clear “before and after” examples of corrected reasoning. 

Result: Overcorrection rates decreased by 28% following rubric calibration. 

2. Listening Comprehension 

Evaluators listened to audio samples and answered contextual questions in natural language, testing the model’s ability to map spoken input to accurate, concise text responses. 

This workflow generates gold-standard references for multimodal instruction-following, vital for speech-to-text alignment and factual recall. 

Result: Accuracy and conciseness scores improved after introducing localized examples. 

3. Preference Ranking (Triads) 

Participants ranked multiple model outputs (best, middle, worst) and justified their reasoning, surfacing fine-grained human judgments on helpfulness, fluency, and tone. 

This preference data powers RLHF reward-model training, offering comparative human feedback signals that steer models toward better alignment and instruction adherence. 

Result: Trainers reached over 90% agreement in calibration rounds, and Welo Data achieved the highest vendor accuracy across this workflow. 

4. Speaking / Spoken Response Generation 

Native speakers recorded 90–120-second responses to prompts, testing natural delivery, register, tone, and completeness. 

This dataset supports speech-based model alignment, enabling conversational systems to generate or evaluate spoken responses naturally and culturally appropriately. 

Result: Trainers improved fluency and natural delivery metrics after live coaching; participation levels reflected differing comfort with voice-recording tasks. 

Supporting Systems 

• Purpose-Built Workforce Funnel: 195 sourced, 159 active; 8.6% pass rate after multilayer linguistic, cultural, and audio screening. 
• Human-in-the-Loop Quality: A native Quality Controller in each locale audited approximately 50% of all completed tasks, applying standardized rubrics to ensure consistency and accuracy. 
• Localized Feedback Loops: Native-language audit comments and office-hour calibrations enabled real-time production corrections. 
• Agile Coordination: Continuous reporting and milestone-based incentives sustained throughput above 2,000 tasks/day. 

The Results

• 36,389 tasks completed in 4 weeks across 20 locales (21 targeted; all except Farsi) 
• Zero rework cycles and continuous daily reporting 
• 87% overall completion rate, 89% among active Trainers 
• Top vendor accuracy in Error Rewrite and Preference Ranking tasks 
• Upward trend in zero-error rates across all task types, validating rubric calibration and real-time feedback loops 

The Insights 

• Localized Guidelines Improve Consistency: Translating and adapting English-centric rubrics increased clarity and performance in non-alphabetic languages. 
• Spoken Tasks Require Sensitivity: Contributor hesitation around voice data improved through optional participation and clearer consent guidance. 
• Balanced Sampling Enhances Reliability: Manual audits created uneven coverage; next phase introduces systematic sampling APIs. 
• Fraud Mitigation Is Foundational: Strengthened identity, IP, and behavior checks prevented low-quality or automated submissions, particularly in high-risk locales. 

The Outcome 

The pilot confirmed Welo Data’s ability to deliver auditable, culturally fluent training data for both text and speech models. 

By project close, the client verified that Welo Data’s quality, reporting rigor, and responsiveness met or exceeded internal benchmarks — paving the way for expanded multilingual evaluation programs at scale. 

Future phases will extend automation, localized rubric development, and multimodal sampling systems to support next-generation LLM and voice-model evaluation. 

Project Snapshot 

Welo Data

The multilingual data & evaluation partner for model builders and enterprises.