[Georgia M]

Scoring SUS results using WER, I guess we should still allow for homophones, as Benoit et al suggested in their way of scoring. Is this right?
The argument should always be that in real (semantically meaningful) speech we actually distinguish homophones through context, which is absent in the SUS stimuli.

[Georgia M]

A follow-up on that…

If we use the Latin Square method, then we cannot score each listener’s results relatively (for the listener and the test), as one listener will no longer have given responses for a full set of sentences. I believe that we will need to score our results somehow objectively.

In this case, could we insert a reference sample of natural speech uttering SUS sentences among our synthetic voices ? If this natural sentence is not transcribed correctly, then we would need to exclude the listener from our sample…or adjust the rest of his scores. I am not sure if this makes sense and how exactly the fact that we don’t have responses from the same individual for all the questions should be controlled.

Any articles/guidance on scoring results of Latin Square questionnaires?

[Georgia M]

Thank you for these clarifications. Now I can clearly make the distinction.

[Author]

Posts