This project focuses on the development of a user-friendly application tailored specifically for visually impaired individuals. The application leverages voice commands to enable users to send emails, check email status, search through messages, and read inbox content, all without the need for visual interaction. This project uses Whisper model for speech to text conversion and GTTS library to vocalize text response, thus facilitating a voice based interaction with users.
- Whisper Model
- gTTs library
- Python
- Flask
- HTML/ CSS/ JS
- User Registration Enable 2-step verification in gmail and generate app password then use your email id and generated app password along with username and password of your choice for registration.
- Login with voice Users can login by speaking their id and password which is matched with hashed password for authentication.
- Voice based response gTTs python library is used to convert text to speech in order to provide voice based response.
- Audio Recording
- The app uses PyAudio to record audio in WAV format which continues till user press space bar.
- Audio parameters: 16-bit PCM, mono channel, 44.1 kHz sample rate.
- Speech to Text Conversion The app uses the Whisper tiny.en model for English transcription (ie. converting voice to text)
- Email functionalities
- Compose and Send Email: Users can dictate emails, which are confirmed and sent securely using Gmail's SMTP server.
- Mailbox Status: The app retrieves the status of email folders via IMAP, announcing the number of messages in each.
- Search Email: Users can search emails by criteria like sender or subject, with results read aloud or skipped as desired.
- Retrieve Latest Emails: Users can fetch and summarize the latest emails from selected folders. Summaries are created using NLTK python library.
- Install the necessary packages
pip install playsound
pip install PyAudio
pip install gTTS
pip install nltk
-
Installing whisper
- Installing ffmpeg: To install whisper you will first need to download and install ffmpeg. You can follow this tutorial for its installation Install ffmpeg.
- Install whisper:
pip install -U openai-whisper
You can also refer to this whisper installation guide Whisper reference
-
Installing flask for frontend Use the file app.py
pip install Flask
-
To run the app use
python -u "app.py"
Open http://127.0.0.1:8000 to view the app.