Evaluation phases and comparing systems

[Giorgia M]

Shall the system go through more than one evaluation, eg. one early evaluation to act upon and a final one, hopefully driving to higher scores?

Is it mandatory to compare systems, or the evaluation can be on one’s voice only? If mandatory, would we be comparing against another Festival voice, or what else?

[Simon]

Multiple rounds of evaluation would be normal when developing a large system over a period of time. For the “Build your own unit selection voice” exercise, that would probably take too much time though.

In general, it’s difficult to evaluate a single system in isolation: this is because most types of evaluation provide a relative judgement compared to one or more other systems or references. Even in the case of intelligibility testing, where evaluating a single system sounds reasonable, we still need to interpret the result: for example, is a Word Error Rate of 15% good or bad? One way to know would be to measure the intelligibility of natural speech under the same conditions.

When comparing multiple systems, we would normally use the same speaker and in fact the exact same database (unless we were investigating the effect of database size or content). Trying to compare two systems built from different speakers’ data would not enable us to separate the effects of speaker from those of the system.

[Author]

Posts