How a hacker used Kinect to help his disabled mom send emails

If we had a series called “Awesome stuff people can do with a Xbox Kinect”, this would be the latest instalment. Also, this guy should win some award for being a good son. Chad Ruble’s mother has suffered from aphasia, a disorder which impairs the ability to process language, since she had a stroke 12 years ago, making it virtually impossible for her to communicate with friends and family using a traditional keyboard. So her son developed a system which lets her send simple email messages, using Microsoft’s Kinect.

Ruble “Kinect-ified” certain simple emotions and quantities to let his mother express how she is feeling without text. She just holds her hand up in the air and hovers over an emoticon and bar graph, to express an amount, which translates into things like “I am feeling very happy”.

How did he do it? Ruble explains:

I used a Kinect with the SimpleOpenNI library for Processing along with some gesture recognition code from Matt Richardson to track the position of my mom’s hand. I then used a sample Processing sketch from Daniel Shiffman to generate and send the email by using the green arrow button. The red “X” resets the screen.

He admits that the system is quite simple at the moment, but he plans to add extra screens and options at a later stage so that his mom can share more detailed messages, which allows the system to take a snapshot of her using the Kinect’s camera and attach it to the email.

Yes, there’s a video… and just look at how happy she is to send her first email in 12 years.