A pioneering tool from UPF allows deaf people to do radio

BarcelonaThe fight for everyone's accessibility to the media, not only as receivers but also as emitters, has taken an important step. Pompeu Fabra University (UPF) has developed SignarIA, a tool that enables deaf people to acquire a synthetic voice with which to make radio. The project was born from the need to solve a challenge that arose in the journalism faculty of this institution: how to involve Berta, the first deaf student to enroll in their degree, in the radio workshops?

SignarIA is based on the construction of an electronic voice as natural as possible. The deaf person expresses themselves in Catalan Sign Language and a device called Computer Vision decodes the message. Then the voice donor comes into play, who in many cases is expected to be a relative of the person with hearing loss. The underlying idea is to combine the latent sound imprint of the deaf person with the prosody and articulation of the donor, who will have trained an artificial intelligence system. The result is a personalized synthetic voice that has the potential timbre of the person using it. A cousin of Berta, of the same age, has been the donor.

The main contribution of the system is that this conversion from sign language to voice is done live and in real-time, so that the person can participate in a radio program and see how their signed speech is converted into sound that can be heard in audio, through the airwaves or the internet. At the same time, through the laptop, the user can read what the other participants in the program are saying, also live. From here, everything is ready for next year, when she will be in her third year, Berta to take on the fifteen weekly hours of the radio workshop that is part of the educational program, for ten weeks.

The way it works allows for very precise messaging. For each signed phrase, the system proposes three messages: the first is literal, the second provides context, and the third contains more creativity. With a machine with pedals, the user can choose which of the three best suits what they want to say (or can discard them and generate three new proposals). From here, artificial intelligence learns the style and way of doing things of the person using it, and refines its predictions more and more.

The project has been driven by UPF, but has been done in alliance with We Are Mortensen, the company that has provided the technology capable of making it possible. For the moment, it is in the prototype phase, and the university has already created two discussion groups: one with deaf signers to analyze the potential uses of the tool and assess its impact, and another with faculty from the communication department to explore the application of this technology both in the classroom and in radio studios. Among the most immediate applications are, for example, its use in trials, to guarantee deaf witnesses that what is recorded faithfully reflects their wishes.

Work is also underway on the inverse process: that what the other participating voices say is transformed into sign language, through an avatar created by artificial intelligence. UPF has initiated contact with other research centers working on this to promote progress in this direction. The underlying idea is to achieve that people like Berta do not have to give up their mother tongue, which is Catalan Sign Language, in her case.

Open access data

Another benefit of the SignarIA project is that it has generated an important corpus of data that is made freely accessible, with the aim of stimulating research in the recognition and synthesis of Catalan Sign Language (LSC). As a result of this work, combined with the work of the LSC Lab research group at the Department of Translation and Language Sciences at the same university, and that of the cooperative Laifari SCS, an innovative methodology has been created for the creation of synthetic data that includes, for example, the conjugation of verbs in Catalan Sign Language. Thus, linguistic examples can be generated systematically and scalably, which can facilitate the learning of this language.

The tool was presented this Tuesday at a conference that included presentations by the two main researchers, Marcel Mauri and Roger Cassany, as well as the project engineers, Xavi Vinaixa and Inés Broto. During the event, linguist Gemma Barberà and coordinator Cecilia Barba presented the qualitative results of the project. The meeting also included a round table titled "Journalism and inclusion, the pending issue", during which the lack of representation of deaf people in the media was discussed.

"It is a technology that adapts to us and not the classic case of when we have to adapt to technology," assesses Cassany. "There is, therefore, also a claim for technological sovereignty, which is born from our needs and is not imposed by large corporations. That is why it is an open technology, at the service of universities and research centers.