Karen's project

What if your camera could understand sign language?

That was Karen's question, and it became ASL Live: a free web app that reads American Sign Language from a laptop or phone camera, and helps anyone learn their first signs. Karen came up with the idea, and she and her dad built it together.

Try it: learn a sign Sign and see the words

250everyday signs it recognizes

79%of fingerspelled letters read correctly on phrases it had never seen

100,000+fingerspelled phrases used to train the model

0videos uploaded: everything runs on your own device

Why it matters

Most hearing people know almost no sign language, so everyday conversations with Deaf and hard of hearing people can be harder than they need to be. ASL Live is a small step toward closing that gap. It helps people practice signs with instant feedback, and it shows how a computer could one day help Deaf and hearing people understand each other.

How it works

1
Track the handsThe camera finds 543 points on your hands, face and body in every frame, right in your browser.
2
Read the signA model trained on thousands of signers matches your movement to one of 250 signs.
3
Spell it outFor names and new words, a second model reads fingerspelling letter by letter. Karen and her dad trained it on Kaggle using Google's ASL Fingerspelling data.
4
Learn and practicePick a sign, watch how it is made, try it on camera, and see right away whether you got it.

What's next

Next up are a stronger fingerspelling model and a way to type or speak so the computer can answer back in sign. Feedback from Deaf signers is the most valuable kind; you can send it to joe@thinkmorphic.com.

ASL Live builds on open work by others: the winning model from Google's sign recognition competition on Kaggle, the PopSign ASL dataset from Georgia Tech, RIT's National Technical Institute for the Deaf and the Deaf Professional Arts Network, and Google MediaPipe. Full credits are on the ASL Live page. A ThinkMorphic project, ThinkMorphic.com.