A real-time interactive game that leverages an ASL (American Sign Language) recognition model to form words. Watch as your hand signs come to life on screen, with letters automatically connecting when they form real words!
intro.mp4
Bubbly uses your webcam to track your right hand in real-time and recognizes ASL finger-spelled letters. As you sign, letters appear on screen. When the letters you've signed form actual words or word prefixes, they automatically connect together visually, creating a satisfying and educational feedback loop. This visual connection system helps learners to see immediate feedback on their signing, improve awareness of specific gestures and angles, and understand word formation in a unique, interactive way.
When signed letters are spatially close and spell a valid word, they visually snap together.
make_words.mp4
This update introduces spatial hashing for efficient neighbor detection among letter bubbles. Previously, finding neighboring letters required comparing every bubble against every other bubble — an O(n²) operation. With spatial hashing, the canvas is divided into a grid of cells, and each bubble is registered into the cell it occupies. Neighbor lookups then only check bubbles in adjacent cells, reducing the cost of neighbor queries dramatically as the number of bubbles grows.
This creates richer, more performant force interactions between letters, making the simulation feel more alive without sacrificing frame rate.
Each letter bubble is now influenced by a Perlin noise field, giving them organic, flowing movement rather than static or purely physics-driven behavior.
noise.mp4
Letters now respond to local density: when too many letters crowd together, repulsive forces push them apart.
density.mp4
If too many of the same word crowd together, they merge into one. This feature is still being implementated, as you can see the merge direction isn't correct
weird.mp4
For this project, I also spent a lot of time looking into data visualization techniques such as Delauray triangulation and Vonoroi diagram, as well as particle simulation which involves GPU programming and texture manipulation. These were difficult ideas to wrap my head around, and my first attempt in implementing these features wasn't super successful. However I think these are important skills to have as I scale up my project in the future, especially for force calculations on the letters.
- p5.js - Creative coding framework for visualization and interaction
- TensorFlow.js - Machine learning model for hand pose detection and ASL recognition
- MediaPipe - Real-time hand landmark detection
- Real-time Recognition - Instant ASL letter detection using your right hand
- Smart Word Detection - Automatically identifies when signed letters form valid words or prefixes
- Visual Connections - Letters dynamically connect on screen when they form meaningful combinations
- Educational & Entertaining - Learn ASL through interactive, visual feedback
To play with this program, you will need a modern web browser, webcam access, and preferably good lighting for optimal hand tracking.
- Position yourself in front of your webcam with good lighting
- Hold your right hand in view of the camera
- Sign ASL letters one at a time
- Watch as letters appear on screen
- When your letters form words or word prefixes, they'll automatically connect!
- Keep practicing and watch the bubbles connect
- Improve recognition accuracy
- Expand vocabulary recognition
- Implement custom word-based or letter-based forces
Hope you enjoy a little game of sign language!