It’s that time again! A bunch of really eager computer scientists have a prototype that will translate sign language to speech! They’ve got a really cool video that you just gotta see!
They win an award! (from a panel that includes no signers or linguists). Technology news sites go wild! (without interviewing any linguists, and sometimes without even interviewing any deaf people).
…and we computational sign linguists, who have been through this over and over, every year or two, just *facepalm*.
The latest strain of viral computational sign linguistics hype comes from the University of Washington, where two hearing undergrads have put together a system that … supposedly recognizes isolated hand gestures in citation form. But you can see the potential! *facepalm*.
Twelve years ago, after already having a few of these *facepalm* moments, I wrote up a summary of the challenges facing any computational sign linguistics project and published it as part of a paper on my sign language synthesis prototype.
But since most people don’t have a subscription to the journal it appeared in, I’ve put together a quick summary of Ten Reasons why sign-to-speech is not going to be practical any time soon.
- Sign languages are languages. They’re different from spoken languages. Yes, that means that if you think of a place where there’s a sign language and a spoken language, they’re going to be different. More different than English and Chinese.
- We can’t do this for spoken languages. You know that app where you can speak English into it and out comes fluent Pashto? No? That’s because it doesn’t exist. The Army has wanted an app like that for decades, and they’ve been funding it up the wazoo, and it’s still not here. Sign languages are at least ten times harder.
- It’s complicated. Computers aren’t great with natural language at all, but they’re better with written language than spoken language. For that reason, people have broken the speech-to-speech translation task down into three steps: speech-to-text, machine translation, and text-to-speech.
- Speech to text is hard. When you call a company and get a message saying “press or say the number after the tone,” do you press or say? I bet you don’t even call if you can get to their website, because speech to text suuucks:
-Say “yes” or “no” after the tone.
-I think you said, “Go!” Is that correct?
-My mistake. Please try again.
-I think you said, “I love cheese.” Is that correct?
- There is no text. A lot of people think that text for a sign language is the same as the spoken language, but if you think about point 1 you’ll realize that that can’t possibly be true. Well, why don’t people write sign languages? I believe it can be done, and lots of people have tried, but for some reason it never seems to catch on. It might just be the classifier predicates.
- Sign recognition is hard. There’s a lot that linguists don’t know about sign languages already. Computers can’t even get reliable signs from people wearing gloves, never mind video feeds. This may be better than gloves, but it doesn’t do anything with facial or body gestures.
- Machine translation is hard going from one written (i.e. written version of a spoken) language to another. Different words, different meanings, different word order. You can’t just look up words in a dictionary and string them together. Google Translate is only moderately decent because it’s throwing massive statistical computing power at the input – and that only works for languages with a huge corpus of text available.
- Sign to spoken translation is really hard. Remember how in #5 I mentioned that there is no text for sign languages? No text, no huge corpus, no machine translation. I tried making a rule-based translation system, and as soon as I realized how humongous the task of translating classifier predicates was, I backed off. Matt Huenerfauth has been trying (PDF), but he knows how big a job it is.
- Sign synthesis is hard. Okay, that’s probably the easiest problem of them all. I built a prototype sign synthesis system in 1997, I’ve improved it, and other people have built even better ones since.
- What is this for, anyway? Oh yeah, why are we doing this? So that Deaf people can carry a device with a camera around, and every time they want to talk to a hearing person they have to mount it on something, stand in a well-lighted area and sign into it? Or maybe someday have special clothing that can recognize their hand gestures, but nothing for their facial gestures? I’m sure that’s so much better than decent funding for interpreters, or teaching more people to sign, or hiring more fluent signers in key positions where Deaf people need the best customer service.
So I’m asking all you computer scientists out there who don’t know anything about sign languages, especially anyone who might be in a position to fund something like this or give out one of these gee-whiz awards: Just stop. Take a minute. Step back from the tech-bling. Unplug your messiah complex. Realize that you might not be the best person to decide whether or not this is a good idea. Ask a linguist. And please, ask a Deaf person!
Note: I originally wrote this post in November 2013, in response to an article about a prototype using Microsoft Kinect. I never posted it. Now I’ve seen at least three more, and I feel like I have to post this. I didn’t have to change much.
This has been shared by kind permission from Angus’s blog.
Angus Grieve-Smith is a hearing linguist and programmer. He created one of the early sign language synthesis prototypes in 1997, and has recently been working on information extraction projects in New York University’s Computer Science Department. He teaches Linguistics at Saint John’s University.
The Limping Chicken is the UK’s deaf blogs and news website, and is the world’s most popular deaf blog. It is edited by Deaf journalist and filmmaker Charlie Swinbourne.
Please note that the views of the writers are their own, and not necessarily the views of the Editor or site as a whole. Read our disclaimer here.
- Phonak: innovative technology and products in hearing acoustics
- Ai-Media: Should sign language be taught in schools?
- Bellman & Symfon: home alerting solutions
- Deaf Umbrella: sign language interpreting and communications support
- Clarion: BSL/English interpreting and employment services
- Appa: Communication services for Deaf, Deafblind and hard of hearing people
- Signature: Find out about the Signature conference here.
- SignVideo: Instant BSL video interpreting online
- 121 Captions: captioning and speech-to-text services
- Hearing Direct: Online hearing aids
- The National Theatre: Captioned and BSL accessible theatre in London
- Doncaster School for the Deaf: education for Deaf children
- SignLive: Online video interpreting for Deaf people
- RAD Tax Advice: Tax and Tax Credit info for Deaf people
- Deaf Independent: Deaf care and support services
- Signworld: online BSL learning and teaching materials
- National Deaf Children's Society: The leading charity for deaf children
- DCAL: Find out how to study at the Deafness Cognition and Language Research Centre, London
- cSeeker: Deaf-led educational communication support service
- Signed Culture: Advocating for BSL access to arts and culture
- Sarah Gatford: BSL interpreting, training and consultancy
- SignHealth: healthcare charity for Deaf people
- CJ Interpreting: communication support in BSL
- Sign Solutions:, language and learning
- Action Deafness Communications: sign language and Red Dot online video interpreting
- BSLcourses.co.uk: Provider of online BSL courses
- British Society for Mental Health and Deafness: Promoting positive mental health for deaf people
- deafPLUS: Money advice line in BSL
- Happy: Microsoft Office courses taught in BSL and SSE by a Deaf trainer – all abilities catered for
- Hamilton Lodge School in Brighton: education for Deaf children
- Lipspeaker UK: specialist lipspeaking support
- Ozen: Australian hearing aid specialists
- Elmfield School, Bristol: Inclusive education for Deaf pupils
- Exeter Deaf Academy: education for Deaf children