What does atSpoke do?

atSpoke, using an AI-powered engine, automates the servicing of internal IT and HR requests for companies so that they reach the right department and solution.

‍

What was the Annotation Task?

Classify IT and HR queries, generated from the internal Slack messages and emails within companies, into 13 different categories such as Requests, Technical Assistance, Informational Questions and sub-categories within them.

‍

Task Highlights

Complexity

High

Accuracy

98%

Volume

36,700 data points

Price

$0.13 per data point

Task Type

Classification (13 Categories)

“Takt did a great job with our labeling task - they learned a domain-specific labeling task with a high level of accuracy and maintained a high level of accuracy throughout the task. They stand above and beyond other services due to their attention to detail and commitment to quality!”

-David Lim, Machine Learning Engineer

Problem

Complex Dataset with the Need for Domain Expertise - atSpoke customized their NLP engine to each client’s internal Slack messages and emails. This resulted in the datasets having highly contextual data with software and hardware products as well as lingo specific to certain companies.

Multiple Classification Categories with Intersections Resulted in Edge Cases - atSpoke needed exceptions in the classification of certain categories for specific client datasets. This level of customization needed continuous reorienting of annotator decision making. For example - The query “Can you help me add Excel to my Okta?” could be classified as “Request for a software” or “Help with a software”.
atSpoke’s Internal Annotation Team Fell Short Quality and Cost - Without instant feedback loops, training methodologies and workflows in place to identify, correct and prevent errors in classification there was a huge drop in quality. Additionally, implementing a majority vote protocol was far too expensive.
High Level of Data Security - Since atSpoke customized their NLP engine for each client, they built their models on internal company third-party data which needed to be securely stored and annotated.
Short Term High Volume Contract - atSpoke’s tasks were intermittent high volume tasks and hence there was a need for rapid scaling of the annotation team.

‍

Solution:

Hired full-time annotators with BAS in computer science and HR backgrounds to handle the IT and HR queries respectively. Since they understood the context and application of the dataset there was a high dataset-annotator fit.
Takt developed internal operation workflows through it’s in-house platform that accounted for edge cases. A repository of terms and concepts that were foreign to the annotators was also built over time.
Feedback loops helped annotators flag issues and doubts during annotation rather than pick with uncertainty. This ensured errors did not propagate and a mistake of one annotator was a learning for every annotator.
Every Takt annotator that deals with private data works at our on-site secure facilities ensuring the highest privacy standards.
Takt built monthly contracts for atSpoke which perfectly suit their needs. We also achieved majority votes within atSpoke’s budget.

‍

Learn about atspoke's experience and why they continue to partner with Takt

Download Now

How did Takt Overcome 5 Critical Problems with atspoke’s Conversational AI Training Data?