<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=555865791448670&amp;ev=PageView&amp;noscript=1">

Build Winning Teams with iMocha

Company News, Diversity & Inclusion, iMocha Engineering Product Updates Remote Hiring Skills Assessment

All Posts

The phrase Artificial Intelligence, sweetly abbreviated as AI when put in front of a product, raises the expectations from a product beyond limits.  

During the early pre-launch surveys with our clients on our AI-EnglishPro, we knew that our clients expected magic, and we had to provide the magic. Obviously, no rabbits were getting conjured from black hats but surely there was a problem to be solved: a reasonably accurate score of a candidate’s English Proficiency within reasonable time as a candidate speaks, writes, reads, and listens.  

The other challenge was that our test, with an ability to automatically evaluate a candidate’s proficiency, would be compared against other manual variants of it. Unlike in some cases, wherein the prefix of AI is used as a marketing gimmick, we genuinely wanted to increase the trust of our Hiring Managers, Recruiters, Learning & Development Leaders, and other stakeholders in AI. We also wanted to ensure that AI actually helps them cut down their hiring or upskilling assessment time.  

We started breaking the of a mammoth task at hand. While one team of experts started deep-diving on CEFR Compliance, another worked on the content of the initial variation of the test. The product team worked on the UX and candidate experience; and we, the Engineering team, started brainstorming the AI techniques, Reporting Engine, APIs, and the overall Integration and Architecture 

Being an English Proficiency Test, surely the major department of AI/ML we were to use was the NLP (Natural Language Processing). However, that didn’t cut down on the brainstorming efforts.  

AIENGLISH-PRO-1

The initial mornings started with alternatives. The questions that rose were should we go with models of our own, should we use services of Cognitive Service Providers like Microsoft. And if we go with our own models, where do we build, train, and test the models. 

Furthermore, we had multiple enabling skills against which we were planning to assess the candidatesFor example, we started with Oral Fluency, Vocabulary, Grammar and Spelling, and Sentiment Analysis for speaking. For writing, we started with Vocabulary, Grammar and Spelling, Sentiment Analysis, and Writing Fluency with road-map items of Pronunciation Check and Context Check Analysis. 

We decided to do a mix and match for our beta version. The plan was to use our own models for few evaluation points; and for few, check with third party libraries or pre-built cognitive services.  

We knew that the technology stack, the models, the architecture, etc. would keep evolving. Hence, we decided to start with very basic models in our initial beta version, gathering data on the beta version of the tests, and using that data to train our models for subsequent phases. 

So we started with VADER (from the NLTK library) for Sentiment Analysis, a third party library for grammar and spelling check, heavy pre-processing and analysis using NLTK and spaCy, the most popular NLP libraries for other evaluation points. The initial check for vocabulary was through a NLP concept called Named Entity Recognition, only to later realize that it definitely doesn’t come close on its own 

AIENGLISH-PRO-2

This basic selection of models or third-party libraries helped with creating a very initial framework of our NLP Engine, integrate it with our test platform as the upstream and reporting Engine as the downstream, containerize, deploy on our Microsoft Azure subscription, and initiate the beta version of the test to some of our friendly clients.

This initial beta version proved to be the key to our success. It not only helped with validating our end-to-end integration flow, test our exception and error handling, test our architecture, test the look and feel of the reports, candidate experience, etc. but it also helped us correct our assumptions on the data. Post the initials assessments, we went back to the whiteboard for few weeks.  

The outcomes were many, the four prominent ones being: 

  1. We were able to replace VADER with our own model based on BERT and Binary classifier 
  2. We successfully replaced the NER based scoring for vocabulary with frequency distribution analysis of one of the largest corpus 
  3. We attempted to replace the third-party library without own model again based on BERT 
  4. And finally, multi-label classifiers were tested on GEC 

On the architecture front, there were heavy changes as well. For example, the underlying messaging medium was changed, container configuration, changes to the base docker images, etc. The UX changed, the content of the tests changed a bit, and a lot more changes were done on candidate experience. 

nlp-ARCHITECTURE

Coming back to the AIwe started thinking more and more on the ML Ops, dataset curation, hyper parameters tuning, heavy training, validation and testing of our models, etc.  

However, one amazing thing that happened during these efforts was that we got connected with the Microsoft’s specialists and engineering team. Detailed brain-storming with Microsoft engineering team and architects helped us change our direction slightly. We started looking at Microsoft Azure Machine Learning Services to build, train, and test our models in order to help with ML Ops mainly. Parallelly, we also started looking at Microsoft Cognitive Services offerings. The Microsoft team, by this time, well understood our use case; and because of our frequent calls followed by POCs and tests, we were able to close down the use of Azure Cognitive Services like Text Analytics, Speech Services, Custom Text, Azure Video Indexer, and other offerings for the phase two of the product 

We were then on a mix of our own models, third party libraries, NLTK, spaCy, and Azure Cognitive services. It seemed perfect.  

We rolled out the phase two changes to some clients of ours. Though the results were great, there was a lot of calibration required. Our main clientfor AI-EnglishPro are based in the USA and India, and hence we had to recalibrate keeping both audiences in mind. This exercise took few weeks, each of us were taking multiple tests ourselves, regeneration of reports post each change in logic and comparing if the results of the logic are acceptable, taking client feedbacks, going over candidate feedbacks from the tests, etc.  

Fast forwarding, our improved version of the test is now actually helping some of our clients assess their candidates on Business English, and the efforts seem to have paid off based on our client feedbacks. However, we are not stopping here for sure. We are constantly learning and improving 

We assess the data and how our models and Azure Cognitive Services religiously on a daily basis, and any discrepancies are fixed in our models or sent to the Microsoft Engineering team for their inputs.  

Moving to the roadmap, one of the key question our clients ask is if they can customize the test. We understand that our clients have various use cases, for example, checking the English Proficiency in a support role, in a back office role, in a management role, and hundreds of other use cases. Due to the requirement of data set and training and testing of the models for each of this use case, we surely cannot open hundreds of these use cases; however, we are in the process of opening some of these key use cases.  

We also have been asked by our clients about our pronunciation check. However, AI-EnglishPro is CEFR compliant and, as per the changes in the Phonological Control aspect of CEFR 2018, which stresses on intelligibility rather than ascent and pronunciation, we have decided to make another product that checks pronunciation of a candidate against a reference text. 

C:\Users\Vishal Madan\Pictures\tempsnip.png

The results of this undertaking have been overwhelming to say the least; we have completed the initial 1000 assessments for a client of ours in Edu-Tech and the results are great, but a lot needs to be done still 

We love to see our clients happy and we love to see them astonished by the magic. We haven’t been able to bring out a rabbit through AI; however, I believe we surely have been able to magically bring out a right unbiased assessment of proficiency of their candidates.  

You can learn more about AI-EnglishPro here.

Vishal Madan
Vishal Madan
Head of Engineering, iMocha, Vishal is a senior techno-functional leader, who takes pride in being fearless, decisive and works only with one motto in mind – 'Value Addition' in any task at hand, be it the primary portfolio in Delivery, Managed Services or Program Management or proactively taken secondary portfolio(s) which helps the organization succeed.
Find me on:

Topics: Tech Recruitment, iMocha Engineering

Related Posts

Major Announcement: Interview Mocha and all its domains are now iMocha.io

2020 ushered a change - the new normal (as netizens lovingly call it), virtual interactions, innumerable calls over Zoom, Teams, Google Meet, what-have-you. Phew!  

HR Conferences in 2021 You Cannot Miss

Attending HR conferences is a great way to broaden your understanding, learn the new ticks that keep the HR world turning, and network. They often deep-dive into specific topics and discussion areas, so more often than not, you’ll go back with actionable insights and practices that all can follow. Moreover, with the world shifting to online in 2020 and beyond, these conferences are more accessible than ever. 

iMocha's 2020 At a Glance

2020 is a year that tested all of us. We were forced to leave our everyday routines, activities behind. Our way of life changed; our way of work changed.