PyData Seattle 2023

Sample this – two cities in India; Mumbai and Pune, though only 50 miles apart have a distinctly varied spoken dialect. Even stranger is the fact that their sign languages are also distinct, having some very varied signs for the same objects/expressions/phrases. While regional diversification in spoken languages and scripts are well known and widely documented, apparently, this has percolated in sign language as well, essentially resulting in multiple sign languages across the country. To help overcome these inconsistencies and to standardize sign language in India, I am collaborating with the Centre for Research and Development of Deaf & Mute (an NGO in Pune) and Google. Adopting a two-pronged approach: a) I have developed an Indian Sign Language Recognition System (ISLAR) which utilizes Artificial Intelligence to accurately identify signs and translate them into text/vocals in real-time, and b) have proposed standardization of sign languages across India to the Government of India and the Indian Sign Language Research and Training Centre.

As previously mentioned, the initiative aims to develop a lightweight machine-learning model, for 14 million speech/hearing impaired Indians, that is suitable for Indian conditions along with the flexibility to incorporate multiple signs for the same gesture. More importantly, unlike other implementations, which utilize additional external hardware, this approach, which utilizes a common surgical glove and a ubiquitous camera smartphone, has the potential of hardware-related savings of as much as US$100mn+ at an all-India level. ISLAR received great attention from the open-source community with Google inviting me to their India and global headquarters in Bangalore and California, respectively, to interact with and share my work with the Tensorflow team.

Outline

Background of the problem - understanding the problems faced by the deaf and mute community.
14 million people in India have speech and hearing impairment.
Current solutions are neither scalable nor ubiquitous.
Defining a strong problem statement
Key aspects while designing the application.
Building a low resource consuming machine learning model that can be deployed on the edge.
Eliminate the need for external hardware.
Phase 0 : Localizing just hand gestures.
Phase 1 : Adding your facial key points along with hand localization.
Phase 2 : Adding sequential information each frame for carrying the context this enabling the model to pick up the entire context.
Getting resources from Google and TensorFlow.
Results and conclusion
Future aspects

Demonstrations
Preparation [15 mins]
ISLAR Phase 0 [5 mins]
ISLAR Phase 1 [5 mins]
Presentation at Google, Bangalore [5 mins]
Presentation at Google, California [5 mins]

Target audience and outcome : This tutorial is aimed at machine learning practitioners who have relevant experience in this field with basic understanding of neural networks and image processing would be highly appreciated. By the end of the session, the audience will have a clearer understanding of the problems being faced by an underrepresented community in India therefore, catalyzing the thought process of the attendees to address social issues in India as well as other developing countries.