Signo

The problem statement: About 70 million people have speech and hearing impairment, making it tough for them to communicate with their loved ones and coworkers. A platform that makes communication easier for them over the internet without the other person knowing sign language would help them in their everyday lives.

The solution: A software that converts ASL to plain English and displays it as subtitles on your everyday conference apps like G-Meet and zoom would make it very easy for people with a speech impairment to talk to everyone, without worrying about the other person not knowing sign language. Features like adding your own slang and phrases for signs, and a section to learn sign language makes it although more useful to use and acts as the complete suite of an online communication platform for the speech impaired.

Timeline : The initial stage of this project might be considered the ideation phase, as not a lot of work had been done on the project. We had a clear idea of the machine learning models and the type of data to use in order to implement it. We had a very basic RNN model trained with 5 words and 30 videos each. Our project uses Mediapipe to map out the data points for each hand and converts those points into a numpy array which is then fed to the model to predict the hand sign

Now the model is trained with 15 classes with 60 videos of each class and 30 frames per video, with 126 data points in every frame and 3 coordinates per data point. we created the dataset over the course of this hackathon and it's entirely trained by us. Now the model can work on any conferencing app by creating a virtual camera using pyvirtualcam. We optimized the entire software, to use lesser data points and reduce the drop in the frame rate when used in the browser.

What's not working: The create your own sign language feature is almost working. The heavy lifting (aka data collection and preprocessing) is being done, but you have to update the classes and train the model again to see the changes.