Testing during system development

[Ida S]

What is the best way to perform evaluation during system development? I don’t think researchers regularly recruit human evaluators to listen to the outputs of their system after each small design change, and yet some way of measuring progress seems necessary. It would be neat to have some automatic objective scoring methods, based on statistical properties of human speech, even if they could only serve as rough guides. What I imagine as a desirable scenario is knowing what features are most distinctive of synthetic speech as compared to natural speech and being able to evaluate how your system fares with respect to those features. Such features would need to be relatively easy and quick to calculate. They could include for instance some measure of smoothness of the pitch contour, or of how natural spectral changes are or of how human-like is the placement and length of pauses.
What I’d like to know is (a) whether we know what computable properties of speech correlate reliably with perceived naturalness; (b) if so, do researchers use these properties for development-stage evaluation and if not, what do they do apart from listening to the synthesised outputs themselves?

[Simon]

If we don’t want to perform a listening test after every minor change to the system, then we need to rely on either

1. Our own listening judgement
2. Objective measures

We’ll cover objective measures in the lecture.

Objective measures are widely used in statistical parametric synthesis. In fact, the statistical model is essentially trained to minimise a kind of objective error with the training data. We can then measure the error with respect to some held-out data.

[Author]

Posts