A leading AI safety and evaluation company required expert human evaluation to validate AI model behavior against real-world corporate policies. The engagement demanded a trained, calibrated workforce capable of operating across two structurally distinct evaluation tasks simultaneously: assessing AI-generated responses for policy compliance, and auditing AI-generated rules for hallucination against source documentation. 

Welo Data deployed a 15-person specialist workforce, delivered both tracks in parallel across a phased program, and exceeded the client’s quality threshold on all three policy frameworks.  

The Challenge

Policy Compliance at Scale

The client needed a trained workforce to determine whether AI-generated responses complied with, or violated, the specific codes of conduct of three different companies operating across distinct regulatory and cultural environments. Evaluators had to identify not just direct violations but also subtle circumventions and prompt-rephrasing attempts, without being misled by the user’s intent. 

Rule Hallucination Detection 

The client’s AI system auto-generated policy rules from source documents. Each rule required expert auditing to determine whether it was fully grounded in the source policy, partially hallucinated (adding constraints not present in the text), or fully fabricated. This required deep reading of complex corporate and regulatory policy documents and precise cross-referencing, a cognitively demanding task structurally distinct from the compliance track. 

The Approach

Welo Data designed and executed a dual-track quality program with independent guidelines, annotation logic, and quality control (QC) mechanisms for each task stream. 

• Per-company calibration gating: Each rater completed company-specific calibration quizzes as a prerequisite before accessing batch tasks, preventing cross-policy framework drift when raters moved between policy environments. 
• Three-layer QC architecture: Consensus scoring, blind test tasks with pre-defined correct answers, and senior expert audits ran concurrently across all three company tracks, with a 30% audit target. 
• Targeted coaching loops: Per-rater agreement tracking with individual ground truth deltas enabled early identification of outliers and targeted re-training before batch quality was affected. 
• Guideline intelligence: Systematic misalignment patterns were identified, quantified, and resolved with documented intervention plans, and delivered back to the client as structured guideline update recommendations. 

Key Project Components

Task 1: Policy Compliance Evaluation 

Evaluators assessed AI responses against company-specific codes of conduct, selecting On-Policy or Off-Policy labels and, where applicable, categorizing violation type as Direct Violation, Bypass, or Prompt Rephrasing Suggestion. Justification required citation of the exact source policy language, not section headers. 

Task 2: Rule Hallucination Audit 

Evaluators cross-referenced AI-generated rules against source policy documents, labeling each rule as Correct, Partially Hallucinated, or Fully Hallucinated, with a reason category and optional source citation. 76% of hallucinated rules involved plausible-sounding constraints not present in the source, making detection non-trivial. 

Quality Assurance Infrastructure 

A dedicated QC team ran consensus validation, blind test task scoring, and rater audits. Raters falling below threshold received coaching, re-training, and formal escalation if issues persisted. Three systematic misalignment patterns were identified and mitigated during production. 

Outcomes and Impact

• All three policy frameworks exceeded the client’s 85% ground truth agreement production threshold, with peak performance reaching 91.7%. 
• 690 AI-generated rules audited across three company frameworks, with grounding accuracy and hallucination type documented for each. 
• 3 systematic misalignment patterns identified, documented, and resolved with estimated quality lift of  8-15% per intervention. 
• Structured guideline update recommendations delivered with 7 specific instruction changes, priority ratings, root cause analysis, and impact projections. 

Key Results

• All three policy frameworks exceeded the 85% ground truth agreement threshold, validating the evaluation program across distinct regulatory and cultural environments. 
• Peak GT agreement of 91.7% achieved on the policy compliance track. 
• 690 AI-generated rules audited across three corporate policy frameworks, with grounding accuracy and hallucination type documented for each. 
• 3 systematic misalignment patterns identified, documented, and resolved, with estimated quality lift of 8 to 15 points per intervention. 
• Structured guideline update recommendations delivered: 7 prioritized instruction changes with root cause analysis and projected quality impact. 
• Dual-track evaluation framework established and ready to extend across additional policy domains and languages. 

Why It Matters

As AI systems are deployed in regulated, high-stakes environments, the ability to validate model behavior against real-world policy frameworks becomes critical. This engagement demonstrated that complex, dual-track AI safety evaluation can be executed at production scale with rigorous quality controls, and that systematic annotation insights can directly improve the next iteration of model evaluation programs. 

Welo Data

The human layer behind enterprise AI evaluation.