A Fortune 100 cloud technology company selected Welo Data to develop a benchmark dataset for evaluating AI systems built to analyze, visualize, and interpret data through code-driven analytical tasks designed by highly specialized domain experts.

The initiative centered on coding and analytical reasoning tasks that reflected real-world business scenarios, ensuring that models could not only execute code correctly but also generate insights that align with professional data science and engineering workflows.

The client needed a robust benchmark to evaluate the accuracy and reasoning capabilities of their AI agents, as existing datasets failed to capture the complexity, diversity, and ambiguity of actual data science challenges.

Welo Data’s approach combined technical precision with thoughtful workflow design — applying creative problem-solving to structure benchmarks that reflected the realities of analytical reasoning in business contexts.

The Challenge

Evaluating the reliability of coding agents requires more than testing syntax or runtime accuracy. The client lacked diverse, high-fidelity Golden Sets grounded in realistic analytical tasks — the kind that require multi-step reasoning, business context, and strong code–to–visualization alignment.

Previous datasets were too narrow, focusing on straightforward problems that did not represent the variability and complexity of real-world data or the nuanced decision-making process behind successful data analysis.

Welo Data was tasked with designing a benchmark that could test whether these AI systems understood not just how to code — but why they were coding.

The Approach

Welo Data executed two complementary task types — visualization tasks and insight tasks — blending human prompt engineering expertise with technical rigor in Python scripting.

Each task type represented a distinct workflow. For visualization tasks, contributors designed datasets, prompts, and Python code to produce final visual outputs. For insight tasks, the workflow culminated in a rubric and a golden model response, capturing the analytical reasoning and answer quality expected from high-performing AI systems. Together, these two structures mirrored how real data scientists move from concept to conclusion, creating a multidimensional benchmark for evaluation.

Key Project Components:

• Task Volume: 140 total tasks (70 visualization, 70 insight)
• Two complementary contributor profiles:
  • Prompt Engineers and Analysts – Experts in designing realistic business scenarios, crafting prompts, and defining evaluation rubrics to ensure contextual and linguistic accuracy.
  • Developers and Data Scientists – Professionals with 5+ years of Python experience responsible for coding, dataset creation, and generating outputs that aligned with defined rubrics and visual standards.

This division of roles balanced creativity with technical precision, ensuring quality across both natural-language and code-based components.

The design emphasized flexibility within structure — giving contributors room to apply domain creativity while maintaining consistent coding and evaluation standards.

• Onboarding & Guidelines: Live training with follow-up recordings, reference materials, and standardized templates for coding structure and evaluation criteria
• Role Design: Clear division between prompt writers and coders to balance creativity with technical precision

Outcomes and Impact

The project successfully delivered a reusable benchmark for AI-generated code and visualization outputs, setting a new standard for how model reasoning is evaluated.

Through cross-time zone collaboration, contributors maintained consistent delivery while reducing turnaround delays. Engineers and developers worked in tandem — one team driving Python accuracy, the other shaping contextually grounded prompts.

Key Results:

• 140 tasks delivered across 10+ business domains
• Consistent quality across both natural-language and code-based components
• Established a scalable framework for coding evaluation that can extend to future 1,000+-task datasets
• Created reusable examples that now inform new approaches to reasoning-based model assessment

The client recognized Welo Data’s ability to pair technical expertise with adaptive, solution-oriented workflows—an approach that improved both efficiency and benchmark fidelity. By combining human oversight, domain expertise, and methodological structure, the collaboration moved model evaluation beyond surface-level correctness into genuine reasoning fidelity. The focus on completion times and pricing reflected strong delivery quality and scope alignment, with internal tracking showing a ~50% reduction in average handling time (AHT) from project initiation to final delivery.

Why It Matters

This engagement proved that coding-based AI evaluation can evolve beyond surface accuracy to measure true analytical reasoning. By combining human expertise with structured methodology, Welo Data helped the client build a benchmark that reflects the real-world decisions data scientists make every day.

The result is a new standard for evaluating AI agents capable of coding, reasoning, and communicating insights — reliably, transparently, and at scale.

By uniting prompt engineering precision with coding accuracy and scalable human review, Welo Data demonstrated that structured evaluation frameworks can redefine what reliability looks like for coding-based AI agents.

The project underscored how thoughtful workflow design and adaptive problem-solving can bring creativity into even the most technical layers of AI evaluation — driving both consistency and innovation in how coding agents are tested.

Welo Data

The human layer behind enterprise AI evaluation.