Evaluation of comprehension and working memory

[Chiara]

When designing an evaluation test, e.g. listening to a long passage, since cognitive skills such as working memory largely influence comprehension, what other ways there are to control for this effect other than “measuring” the participants’ “working memory”? It does not sound like a completely reliable approach.
Would it be convenient to include a control trial in which the same passage is read, for example, by a human? Or would the comparison between natural and synthesised language just not be fair?

[Simon]

Including a control trial sounds like the right thing to do. In almost all types of evaluation, we need something to compare to:

1. We explicitly ask listeners to make a comparison and tell us their judgement
2. We compare the responses of listeners across several conditions

The second case applies here. If we cannot control for unknown factors (e.g., working memory) then we have to make sure those factors have the same value across all conditions that we wish to compare. In your example, we would have the same subjects perform the same task, once on the synthetic speech, and once on natural speech, then we would quantify the difference in their responses (e.g., accuracy in a comprehension test).

[Author]

Posts