AI-Powered Bidirectional Sign Language Translator

Abstract

The communication between deaf and hearing people still raises obstacles in different settings such as education, work, and even public places. This is mainly due to the shortage of sign language interpreters and the low level of sign language skills among the general public. As a result, these obstacles often lead to unequal participation, delayed information access, and exclusive communication. However, the continuous integration of AI and computer vision technologies presents a possibility of creating automated translation tools that could make communication easier and more accessible. This project, driven by this requirement, offers an AI-enhanced two-way sign language translator that features real-time Sign-to-Text and Text-to-Sign translation utilizing common hardware and a web-based interface. The Sign-to-Text component of the system makes use of a standard webcam to peg hand movements and sends each of the sequential frames through the MediaPipe Hands algorithm in order to detect the 21 main hand points. The identified points are then mapped into normalized feature vectors and subsequently categorized using a CNN (convolutional neural network) that has been trained on appropriate ASL alphabet datasets. Such a setup enables prompt detection of non-verbal random letter gestures without making use of particular sensors like data gloves or depth cameras. The result is gradually built up into a highly readable text, thus opening up the world of sign language to hearing users who can then follow the conversation of deaf users. The Text-to-Sign module acts as a reversal by issuing a transformation of written text into a series of ASL gesture images. This image-oriented approach gives a lightweight and resource-saving alternative to 3D animation, thus allowing it to work under low-power conditions while being clear to deaf users. The entire system is developed as a web application using the Django framework, which means it is accessible from any device and browser, therefore, no installation is needed. To test the performance of the system regarding recognition accuracy, latency, usability, and compatibility, extensive tests were performed. The Sign-to-Text classifier during controlled evaluations reached an accuracy of about 95%, with an average time of fewer than two seconds per prediction. Alongside this, the system was perceived as very easy to use by the deaf and the hearing participants; they all concluded that the interface is user-friendly, the translation results are easy to follow, and the system allows for basic communication to run smoothly. Compatibility tests also confirmed that the system performs reliably across the most popular web browsers and operating systems