Report Write-up and Marking Guidance

Sections 1-4: Festival Error Analyses [4 x 10 marks]

For the each of the error analysis sections (1-4), you should at least:

• [3 marks] Clearly describe exactly the error you have found. That is, tell us:
  • What the text input to Festival was
  • What the expected (i.e., correct) output should have been
  • What the actual output was
• [3 marks] Clearly explain what TTS pipeline component is responsible for the error and why the error happened in that component.
  • Provide evidence to support your explanation.
• [4 marks] You can get more marks for showing more depth of analysis, for example:
  • Further analysis/more evidence of mistakes’ origins, e.g. if it’s due to interactions of modules
  • Analysis of severity of mistakes, e.g., Would it affect output frequently or only rarely?
  • Further analysis of the error from a phonetics perspective, e.g., why does it sound bad? Why might the error affect speech intelligibility or naturalness? Is the error something a human speaker might also make?
  • Discussion of potential solutions
  • Use of citations to further support your claims/argument
  • Other relevant insights into the errors, especially those that highlight differences between theory and practice.
  • Use of figures, tables, other visualisations to make your answer more informative. It will be much easier to report some types of errors with tables, but some will require figures (e.g., waveform errors).

Try to be precise in pointing out how the error appears in the linguistic specification and/or the generated waveform (as appropriate). You may find it useful (in some cases necessary) to use tables or figures to illustrate the errors or explanations of where they came from.

You can describe more than one error per category, but note that you’ll likely get more marks for showing more depth of understanding about the category as a whole. So, several very similar examples will probably not get you as much as an in-depth analysis of a specific example that really shows that you understand what that particular part of the TTS pipeline does (and potentially how it interacts with other parts of the TTS process).

Section 5: Comparison with a recent TTS system [10 marks]

You can focus on one example from a more recent (non-Festival) TTS system in this section.

• [3 marks] Clearly describe exactly the error you have found. That is, tell us:
  • What the text input to the non-Festival TTS system you are using
  • What the expected (i.e., correct) output should have been
  • What the actual output was
• [2 marks] Clearly show what Festival generates with the same input and explain if it generates the same (or different) errors
• [3 marks] Explain and compare the two outputs:
  • If Festival creates the same error, explain which TTS pipeline component is responsible for the error and why the error happened in that component. Provide evidence to support your explanation.
  • If Festival doesn’t create the same error, explain why this is the case.  For example, did one of the pipeline components correct for the error?
• [2 marks] As for the previous sections, you can get more marks for showing more depth of analysis.
  • You are not expected to fully understand how the more recent TTS systems work at this point.  However, you may get bonus marks for a well reasoned guess for why the error occurred and highlighting how the more recent systems differ from the Festival setup you have for this assignment.

Caution: Remember that this section as a whole is only worth 10 marks, so reading a lot of papers about newer TTS models for this section will probably not give you a lot of gains!  Make sure you finish the rest of the assignment before going for extra bonus marks.

Section 6: TTS pipeline discussion [10 marks]

In the TTS pipeline discussion section, you should discuss the overall implications of your error analysis. When formulating your answer, consider the questions raised in the task description.

When writing up your discussion, imagine that you are a consultant and have been asked to provide recommendations to the company that produced this voice. They have a limited budget and limited time to spend on improving the voice. What should they focus their attention on to improve the voice? What are the most important problems to fix? What are the potential tradeoffs between the usefulness or generalisability of potential solutions, and the effort that would be needed to implement those solutions?

Assume here that the company is committed to sticking with a unit selection voice, so focus on recommendations regarding this voice (i.e., don’t just say change to use a neural network based approach!)

You can consider errors that you found that you didn’t write up in the first part of the report here. However, if you do introduce new error examples in this discussion, you will need to explain them clearly.

As this is a discussion, this section will be marked more holistically (out of 10 marks), but here is a rough guide for mark bands:

• 1-3 (poor): Something has been submitted for this section, but it’s only marginally relevant or has a lot of incorrect statements (hence don’t count under the positive marking scheme)
• 4 (fair): You’ve had a go at outlining the overall problems in the pipeline but it’s more of a straight summary of what has been found than a discussion over the overall implications.   The analysis is lacking additional insights into what should be prioritized to improve the voice. Recommendations are perhaps vague or very generic.
• 5 (good): You’ve identified some overall issues with the pipeline and have attempted to make some recommendations that show understanding of the pipeline as a whole. Recommendations are reasonable and appropriate but might lack further justification or evidence, or might lack in a discussion of generalisability or effort required by the recommendations.
• 6 (very good): Clear and reasonable recommendations with a good amount of justification and evidence. The discussion shows you understand the unit selection pipeline in both theory and practice. This is a good discussion but you go further with, e.g. discussion of generalisability or analysis of the costs/benefits of the proposed approach. Claims are reasonable but it may be missing citations or clear examples that would make the argument stronger.
• 7-8: (excellent): Recommendations show excellent, extra insight into issues with the TTS pipeline and demonstrate an excellent  understanding of the gaps between theory and practice. Recommendations are supported by strong argumentation and evidence, perhaps including more error analysis. Claims are supported by evidence from the report or good citations. Errors not discussed in the previous sections should be clearly explained and evidenced. There is some discussion of the generalisability of their recommendations as well as potential drawbacks.
• 9-10 (excellent++): Everything for the excellent category, plus something extra that goes beyond the scope of the course. The argumentation is flawless and the writing always clear and concise. For a 10, the markers are thinking there is no way that this could be improved given the word limit.

Bibliography

You should provide references for any claims you make outside of what has been covered in the essential course materials. It is not necessary to cite course lectures or videos, though you may wish to reference the readings and other related works.  Similarly, if you are drawing on a paper referenced in the lectures or the course readings, you should reference the actual paper (having had a look at it yourself!)

You can gain extra marks for bringing in insights from related work to strengthen your argumentation (i.e., academic papers that are not on the essential reading list). However, you should be able to complete this assignment to a good standard working just with the core course materials.