Quiz

Let’s test your recall of the video material. Make sure you’ve watched all the videos and read through the corresponding slides. Don’t refer back to them whilst doing this quiz!

Why do we need to evaluate TTS systems at all?

Some reasons are: as a diagnostic test to guide future development (e.g., to decide which research direction to pursue next); to compare against another TTS system, or a baseline; as a pass/fail test to determine if our TTS system is ready to release as a product (e.g., it is sufficiently intelligible).

Why/how would we evaluate during the development of a TTS system?

Individual components can be evaluated before integration (e.g., NSW normalisation can be evaluated using text input/output). The components can also be tested in combination with other components to identify unwanted interactions (e.g., does our improved letter-to-sound model now generate previously-unseen phoneme sequences).

What are some good and bad ways to evaluate intelligibility?

The best way is to measure listener ability to recognise the words being spoken, which is usually done by asking them to transcribe the synthetic speech. A bad way is to ask their opinion of how intelligible the speech was- how would they know, without being given the correct transcription?

Why might we get a "ceiling effect" when measuring the intelligibility of synthesised text from a newspaper?

Because the text is semantically predictable, listeners will use context to predict words that were synthesised poorly. Thus they will tend to recognise all the words, tending to a 100% intelligibility score.

What do these acronyms stand for: MOS; MUSHRA; MRT; DRT; MCD; RMSE

MOS = Mean Opinion Score; MUSHRA = MUltiple Stimuli with Hidden Reference and Anchor; MRT = Modified Rhyme Test; DRT = Diagnostic Rhyme Test; MCD = Mel Cepstral Distortion; RMSE = Root Mean Square Error.