New Zeland based Kara Technology can provide sign language digital humans through the clever use of Epic Games Meta Humans. Their use of Metahumans blended with their own technology can provide accurate Sign Language translation extremely quickly, such as in the case of an emergency public service announcement.
Auckland moves to alert Level 3 from 11:59pm Tuesday for at least two weeks. The rest of New Zealand stays in alert Level 2. For more information visit https://t.co/uQI1aNQSGn pic.twitter.com/xqeSy3gZch
— Kara Technologies (@Kara__Tech) September 20, 2021
The video above is demonstration this technology. Kara’s technology can provide sign language access to emergency messages through API calls. Their signing avatar Vasilisa (above) is providing crucial information regarding New Zealand’s COVID-19 lockdowns in New Zealand Sign Language (NZSL). For the emergency avatars, the base input is text, but it could be audio.
The company started about three and a half years ago when the four founders became involved with the deaf community. Arash Tayebi, one of the co-founders became partially deaf while doing his Ph.D. He came to realize that there’s not a lot of deaf students at Auckland University where he was studying. “He realized that obviously there’s no difference between the hearing and the deaf, and that there should be no barrier to entry into University for the idea,” explains his co-founder Farmehr Farhour. “He looked around and started talking to the deaf education centers in New Zealand and it turns out there’s a lack of resources for deaf children.” Not only is there a lack of resources, but many rely on the written word and deaf children often develop reading skills at a slower rate than hearing children, further disadvantaging deaf children. “Even as later in life, deaf adults usually has a lower literacy age than an equivalent hearing person. So it also disadvantages them when their older,” he adds.
The Kara process is not real-time, but it is able to generate a clip very quickly and the company understand the significance of being able to target real-time signing, but it is not just a function of graphics power. The intent of the sentence needs to understand. The Kara team has respect for actual live interpreters and does not seek to replace them. Rather they seek to offer the option of a personalized scalable option that only computerization can provide. As well as providing a service that may complement traditional sign language when either a human is not present to facilitate or it is not cost-effective. They are actively working with the signing community to help increase inclusion and access.
“We’ve designed our platform is modular. So we can use a transcription model, for example with machine learning transcription to take voice, audio, or video and transcribe that into plain English and then use that to feed our avatars,” explains Farhour.
To be able to take a script and produce a sign language version, the Metahuman’s were chosen as they provide such a wealth of extremely high quality and different digital humans, yet they all have the same core rig and underlying base structure. The Kara team had started their own digital human project and swapped to the Metahuman solution.
Coupled with the character is the technology to input the range of motion capture data and then blend with various techniques to produce a smooth and relaxed visually consistent presentation. “We’ve developed algorithms at a high level that works with a dataset of vocabulary and these are combined based on the input.” explains Farhour. “The transitions are created on an automatic basis. The animation that’s generated is based on motion capture. and all stitched together to produce the final output,”
Expressing concepts in your preferred language
Perhaps if you are hearing and reading this story you may ask, ‘why not just post the written message?’, This is not an uncommon response, at fxguide, we naively asked the same question. In addition to the literacy issue, the deaf prefer to have everything, and their first language of choice which in New Zealand is NZSL. The simplest way to understand this is to imagine that you are in a different country, which natively speaks a different language. You might prefer to hear your native language than the spoken language of the country you are in. You can understand both, but you consider your native language the easiest to deal with, and the simplest to understand. Similarly, if you are deaf in New Zealand then your native language might reasonable be NZSL. Sign language can spell out words as they are written but most communication is not in this form. Thus for a deaf person, their own sign language is not just a literal signing of the written words.
Respect for actual interepters
In film and television much has been rightly made about the ethics of someone of a different race representing someone from a different background. This is an issue in signing as well. Is it culturally appropriate to have a white middle-aged guy produce the sign language translation of a native Mori speaker? In the context of Kara’s Metahumans, the actual interpreter that produces the training data could present as any of the vast array of Metahuman’s Epic games supports.
“We’ve noticed is there is a lot of requirements for different ethnic characters – different avatars with clearly different backgrounds,” says Farhour “We’ve actually looking into this interesting issue of a Mori avatar being driven by a non Mori person, is that acceptable? That’s something that we’ve exploring and we get gathering information from the community, from the different perspectives and making sure what is culturally okay.” The issue is further complicated in a small country such as New Zealand where there may not be enough signing experts from all the various ethic groups that one might want to represent that actually sign NZSL. Imagine a film has a Jamaican character, it might be hard to find a Jamaican NZSL expert, should the presenter’s ethnicity be changed or should a persona with a different background be used to drive a Jamaican Metahuman?
Not only is the origin of the interpreter an issue, but each country and different sign languages. While Kara’s technology could be applied to any particular sign language community, the company would seek to partner with the local interpreters and build a bespoke training data set and character for that community.
But even with all the challenges, the company is made huge steps forward providing technology that promotes inclusion and balances the media landscape for those in the deaf community.