Introduction
It's probably obvious that we need to evaluate any speech synthesiser, but let's pause and ask why that is.
Why evaluate?
What are we trying to get our of our evaluation? Do we need to know how to improve the system, or do we just want to know if it's better or worse than a competing system or a baseline?
What to evaluate?
Depending on our goals, we may need to evaluate the whole end-to-end TTS system, or just some of its components.
Which aspects?
It's important to be very specific about which aspects of the system we are evaluating: do we want to measure naturalness, intelligibility or something else?
How to evaluate
In general, we are going to need some listeners, but what exactly shall we have them do?
Test design
Careful design will make sure listeners do the task we want them to, and that there are no unwanted effects.
Materials
The choice of appropriate text materials needs to be guided by what we are trying to measure, and what kind of listeners we are using.
Evaluation
How do we evaluate a speech synthesiser? Almost always, we will need to play samples of synthetic speech to listeners and obtain some response from them.
This is the new version. Still under construction.