A leading global technology company sought to enhance the performance of its large language model (LLM) for data integrity and quality of its ad and search rating processes.

To reassure themselves of a reliable AI data training program, clients require an ongoing solution to ensure that the quality raters involved in assessing their data were not only qualified and verified but also free from fraudulent activities and monitored continuously throughout any rating exercise.

The solution? Partnering with Welo Data to leverage its innovative risk mitigation platform, NIMO (Network Identity Management and Operations).

---

The Challenge

The client needed human raters to review and grade internet advertisements, which would help influence how ads are delivered to end users in search results.

The primary challenge faced by the client was the need to guarantee the accuracy and integrity of data generated by raters, who play a critical role in evaluating ads and search results. The client required a solution that would address two key concerns:

Rater Quality: Ensuring that all raters involved were verified professionals capable of contributing accurate, high-quality data. The client needed to be confident that the raters were performing their tasks without any fraudulent interference.

Threat Detection: Mitigating potential threats such as cash-grabbing schemes, automated task completion without human involvement (use of bots), account sharing or selling, malicious actors accessing material, etc. These activities could severely compromise the quality of data and lead to significant financial and reputational risks.

The Solution

Welo Data’s NIMO platform was introduced as a comprehensive solution to address these challenges. Launched in 2024, NIMO is a pioneering risk mitigation system designed to ensure data integrity by enabling trust in the data pipeline and ensuring delivery success. The platform integrates multiple methods to safeguard the rater management process:

Rater Verification: NIMO utilizes a multi-step verification process to confirm that raters are who they claim to be. By thoroughly vetting each rater, NIMO ensures that only legitimate individuals are onboarded.

Behavior Analysis: NIMO utilizes automated systems to monitor rater activity for any irregular patterns. For example, if a rater exhibits unusual task completion rates or accesses the system from an unexpected location, these actions are flagged for review. Any inconsistencies in behavior trigger further investigation to ensure compliance with expected standards.

Risk Monitoring and Mitigation: The platform includes mechanisms designed to identify potential risks related to fraudulent activity. By analyzing various behavioral factors, NIMO assesses the likelihood of improper actions and assigns risk scores to participants. Raters with elevated risk scores may be subject to further review or restricted from continuing their activities.

Automated Notifications and Reporting: When NIMO detects potentially malicious activities, it notifies the client so they can take immediate action, such as blocking a rater from their systems. Over time, Welo Data has enhanced NIMO’s processes, automating much of the data collection and reporting, making it easier for the client to monitor rater activity and ensure compliance with their standards.

As Tasos Panagis, Threat Modelling and Intelligence Leader at Welo Data, stated,

“This Welo Data client needed to ensure that any individuals involved in rating their data are verified and vetted to maintain the highest level of accuracy. By utilizing Welo Data’s innovative risk mitigation platform, NIMO (Network Identity Management and Operations), they feel confident in the security of the raters onboarded to work with their data and the quality of the work delivered.”

The Results

Despite being in operation for just a short period of time, NIMO has already delivered promising results for the client. The platform has proven effective in both onboarding and monitoring raters, ensuring that only qualified individuals contribute to the client’s ad and search data processes.

Some of the key outcomes include:

Onboarding Blocks: NIMO successfully blocked several candidates during the onboarding process, preventing potentially fraudulent individuals from participating in the client’s systems.

In-Production Monitoring: A significant number of raters were flagged as suspicious and subsequently blocked while in production. This has minimized the risk of fraud and ensured the integrity of the client’s data.

Tasos Panagis reflects on the program’s success:

“We are talking about potentially tens of thousands of raters. What’s critical is that we guarantee our clients the best teams and also block anyone before they can impact the client’s system and AI model negatively. The results so far have been very positive.”

Conclusion

Welo Data’s NIMO has proven to be a crucial asset for this leading tech giant, helping them mitigate risks and ensure the quality of their ad and search rating processes. By leveraging advanced identity verification, behavioral monitoring, and threat analysis, NIMO has provided the client with the confidence that their data is accurate and secure.

As the program continues to evolve, Welo Data remains committed to delivering innovative solutions that address the unique challenges faced by global technology leaders working with the latest AI models and LLMs.iving into the realm of generative AI (GenAI), a large foundational model developer teamed up with Welo Data to improve the accuracy and fluency of large language model (LLM) output.