Word transcription: homophones and typos

[Qianchu L]

When we ask listeners to perform word transcription tasks, do we first convert their transcriptions into phonetic symbols, and evaluate the accuracy of this phonetic transcription instead? Because I would assume that homophones will be hard to be transcribed exactly the same as the text especially in SUS.

[Simon]

This is a good point, and one we do indeed have to confront in a practical test. In a SUS test, we compare words (not their pronunciations). It is therefore appropriate to allow listeners to type in homophones, or indeed to mis-spell words.

There is usually either some pre-processing of the typed-in responses, before we compute the Word Error Rate (WER), or we allow for these mismatches when performing the dynamic programming alignment as part of the WER computation. This might be achieved by creating lists of acceptable matches for each word in the correct transcription, such as

correct word: your
allowable responses: your, you’re, yore youre

Such lists need updating for each listening test (after gathering the listeners’ responses) because listeners seem to be very good at finding new ways to mis-spell or mis-type words!

I’ve attached an example list of acceptable variants for a set of Semantically Unpredictable Sentences, taken from the tools used to run the Blizzard Challenge.

Attachments:

You must be logged in to view attached files.

[Author]

Posts