At a university school, undergraduate classes are taught in Spanish, but many of the students come from abroad, mostly from Asia. The school wanted to know whether technology could help each student follow the class in their own language, and we prepared a proof of concept.
The problem
The teacher speaks in Spanish and uses the vocabulary of the subject: techniques, tools and proper names you won’t find in a general dictionary. In the same classroom there are students who speak Chinese, English or other languages, and not all of them understand English well enough to use it as a bridge language. And the goal is for students to listen to the explanation while watching the teacher, not to read a screen.
How speech technology solves it
The solution chains three technologies. Speech recognition turns what the teacher says into text, sentence by sentence, as they speak. Machine translation turns each sentence into each student’s language. Speech synthesis reads it aloud in that language, so the student hears it through their headphones almost at the same time as the original explanation.
Since only one voice needs transcribing, the teacher’s, adding languages doesn’t slow down the translation. Each student also gets the text, in case they’d rather read or want to review it later.
The proof-of-concept app
We developed a web app that needs no installation. The teacher speaks into a lapel microphone, which gives clean audio even with noise in the classroom. Each student joins from their phone or laptop with the room code, chooses their language and starts listening. They can change the voice they hear and how fast it speaks.
In the first tests we saw where it failed: some subject terms were transcribed wrong, and very long sentences took a while to arrive. That’s why the plan included tuning speech recognition to the school’s glossary, with help from the teaching staff, and better deciding when a sentence is complete enough to translate.
What we learned
With nearly a hundred classes a week, the cost of the service matters as much as the quality of the translation. When most students need the same language, the architecture can be adjusted to make it much cheaper.
You also have to think about the conversation, not just the translation. Listeners don’t have a microphone, so the design has to work out how a student asks a question when they don’t understand something.
This work is the basis for the live translation for classrooms and conferences we keep exploring in the lab.
