24 Questions You Need to Ask Every Data Annotation Company

There exist several data annotation companies to choose from, each with their own operational workflow, quantitative evaluation, training methodologies and tools. It is the combination of these factors that determine the quality of your training datasets and ultimately the performance of your algorithm.


At Takt, we have developed unique processes based on years of experience dealing with datasets of varying complexities and need for scalability with affordable pricing. We know exactly how to achieve a dataset-annotator fit having studied the psychology behind the annotation process. In order to ensure that you make an informed decision in selecting the data annotation company, we have condensed our expertise into 23 questions across 5 distinct categories each of which represent key areas - 


1. Methodology for Quality

Aim: Understand how they derive quality through uncertainty

2. Pricing and Alignment of Incentives

Aim: Ensure the annotators are motivated for quality and you get the maximum per dollar spent

3. Scalability 

Aim: Confirm you can count on their quality and delivery when your volume increases

4. Data Security

Aim: Guarantee your Data’s Privacy 

5. Technology and Tools

Aim: Learn about quality and efficiency metrics, proprietary in-house tools and APIs and real-time annotation



Task Highlights

Complexity
Accuracy
Volume
Price
Task Type
You NEED to ask these questions to every data annotation company before you consider working with them.

Methodology for Quality

Your machine learning algorithms are driven by the quality of the data they train on. We know that small differences in the quality can make or break your algorithms. We also know that training an algorithm is not a one time task. It is a continuous process where you test and validate your models and enrich your corpus based on the results. A data annotation team needs to be able to adapt to your requirements, keeping quality as the priority. 


The following questions will help you understand their importance placed on quality - 


  1. How do you select annotators for a dataset annotation task? Are there internal quality matrices for each annotator to determine dataset-annotator fit? If you do not have the right annotators in-house, do you have domain experts who handle the hiring, screening and testing of annotators?  
  2. What structure do your annotators work under? Are there supervisors to ensure that they are monitored and trained on a daily basis? Do you have project managers to ensure consistency across your supervisors?  
  3. The annotation guideline I have made might not be fully understood by your annotators. How do you ensure 100% transfer of knowledge?
  4. How do you review the work of your annotators? Do you start with a 100% review? What metric do you follow when deciding to reduce the review %? 
  5. If my annotation guideline changes during an existing iteration, how do you handle the changes? Do you have a mechanism in place to create divisions in the dataset based on different guidelines?

Pricing & Alignment of Incentives

We know the harsh truth: Every annotation task has a maximum budget. Every annotation company can work within that budget, however, at the cost of quality. It is imperative that pricing in a company incentivizes annotators for quality rather than volume of output. It is also important that as a client you gain economies of scale as your volume requirements increase. 


The following questions verify these requirements - 


  1. Do you pay your annotators per annotation or based on the quality of their work? How do you ensure that they are not aiming to produce higher volume but rather incentivized for higher quality?
  2. In order to validate your quality before committing to a large contract, can we run a pilot program?
  3. What is your pricing model and how do you customize the pricing for my unique dataset? Do I pay per annotation or per hour?
  4. How does your pricing model scale with volume? How do you decide the drop in price per annotation with higher volumes?

Fix your hiring process with questions we have devised over years of experience.

Download Now