Mei Zheng, Data Scientist at Smartling, was selected as the winner of the World-Readiness Competition at the TAUS Massively Multilingual Conference & Expo in San Jose, CA, on October 13, 2022. Participants were tasked to present a solution to reach the last human on earth in their native language.
Take a look at how Smartling proposes to quantify the quality of translation memories!
What is Translation Memory?
Translation Memories (TM) are files consisting of previous translation pairs that have been translated in the past across various locales and content types.
Why Is It Important to Quantify the Quality of Translation Memories?
Human translators leverage translation memories as a starting point for their translations through fuzzy matches. If a high fuzzy match string is paired with a low-quality translation, it dramatically reduces the human translators' productivity.
From the machine translation perspective, maintaining a high-quality TM can enable Smartling to help customers use their TM to train custom machine translation engines that tailor specifically to their brand's terminology.
High-quality TM means cost-savings and higher-quality translation in the long run!
Current Challenges with Translation Memory
Files in TM are collected over time, and users often don’t know if the translations are still relevant. If multiple unvetted language service providers or linguists have worked on your translations in the past or machine translations sneaked into your TM unknowingly, your TM quality could remain questionable.
Wouldn’t it be great if there was a straightforward way to quantify the quality of translation memories?
Smartling has set out to build an automatic tool that can detect current translation memory issues and automatically score the quality of translation memory.
How to Score Translation Memory
Smartling is taking a two-part approach to quantifying TM quality:
Relevance
As our customers utilize our TMS, we can gather and utilize the following data:
- How much TM they’re leveraging for their translation
- The percentage of their strings with high fuzzy match scores (over 90% match)
- How many repeated matches on a source text they have
Translations
From the semantic similarity perspective, we’re converting each of the pairs of translations into vector embeddings. In 3D vector space, we can mathematically calculate their cosine similarities. The higher the score, the more similar the string pairs are in terms of their meaning.
And from the textual similarity perspective – if there is a high fuzzy match, we can assign it to human post-editing. Now that we know exactly what the resulting string is, we can then calculate the translation edit rate (known as the human post-translation edit rate). By calculating this edit rate, we can then know the quality of our customers’ translation memory.
According to our TM scoring tool based on these methods, we saw that, on average, high-quality TM scores 80.4 points and low-quality TM scores 53.8 points out of 100 points.
Use Case/Applications for Quantifying the Quality of Translation Memory
- Use the scores to quantify penalization percentage to specific TMs
- Help customers automatically filter out bad strings and clean up their TM
- Better corpus for training custom machine translation engines
Smartling is working on building this prototype into a product as part of our translation memory offering for our customers. Stay tuned for the exciting news!
In the meantime, learn more about Smartling’s Neural Machine Translation Hub.
