In this practical exercise, you’re going to build a neural text-to-speech synthesiser using recordings of your own voice.

Before starting, be a proper engineer and

• keep a logbook to record every single step

You’ll find this invaluable if you need to repeat any steps, and your notes will also be useful for writing up a lab report at the end.

To build your synthetic voice, you will follow step-by-step instructions and use a variety of existing tools. Currently, we only support the University of Edinburgh “Eddie” compute cluster, because some steps require GPUs.

Here are the main stages in this exercise:

1. Get access to the necessary computing facility and set-up your environment
2. Learn how to train the model, using some pre-existing data
3. Create your own data
   • Select or design a recording script
   • Make the recordings in the studio
   • Prepare the data for training the model
4. Train the model on your own data
5. Evaluate the model(s) you have trained
6. Write up.

Read all the way through the instructions before you start!

Related forums

IMPORTANT: you must never directly contact the University computing helpline for assistance, unless we ask you to! Everything that you need (e.g., filesystems, compute access) has already been pre-configured for you. If you need help, ask in the lab or on the forums.

What you will learn in this part:

• Working with a compute cluster:
  • Logging in
  • How to run a job
  • Skills for working on a remote system
  • Where files are stored

Logging in to Eddie in the terminal

We will be using the Edinburgh Compute and Data Facility that provides the “Eddie” compute cluster which has GPUs that we need.

Terminology:

Local computer – the computer you are sitting at, perhaps your own laptop or a PPLS lab computer

Login node – a computer that is part of Eddie, which you can log in to remotely

Compute node – a computer that is part of Eddie, which you cannot directly log in to, but can run jobs on. Some nodes contain a GPU, which is required for running the models that we will use in this exercise.

GPU – “graphics processing unit”, a specialised type of computer that performs a large number of computations at the same time (“in parallel”): well-suited to the type of computations needed for neural networks

Job – running a program on one of the compute nodes, by scheduling it

Scheduler – a program running on the cluster that decides which job to run next

Filesystem – a place to store files. There are multiple filesystems which you’ll learn about shortly.

To log in to Eddie, you need to be either on the campus network, or connected to the VPN. In a terminal on your local computer:

ssh s1234567@eddie.ecdf.ed.ac.uk

Where s1234567 is your username and the password is your EASE password. This will log you in to one of Eddie’s login nodes. Important: you must never perform any heavy computation on login nodes – they are shared between all users (and they don’t have GPUs).

For convenience, you can add the following to your ssh configuration file on your local computer (usually this is ~/.ssh/config)

Host eddie
  HostName eddie.ecdf.ed.ac.uk
  User s1234567

which will allow you to log in using

ssh eddie

and that will also be convenient when using VS Code.

Running your first job

Remember: do not run substantial jobs directly on a login node! There are two ways to run a job on the cluster:

1. Schedule the job (add it to the queue) – this is the most common way you will use
2. Request an interactive session on a compute node, then run the job directly – this is useful for development and debugging

Let’s run our first job!Create a shell script that prints “Hello world!” and save it as hello.sh in your home directory. Since this is such a simple script, it’s OK to try running it on the login node to check it works:

./hello.sh

now add the job to the queue

qsub hello.sh

You will get a message like: Your job 51978720 ("hello.sh") has been submitted

Now, how do we know whether our job is running, or has finished? We can ask the scheduler:

qstat

When run with no arguments, this will list all the jobs that you currently have in the queue. If you get no output, that means nothing is in the queue – so perhaps your job already finished. Since it ran on a compute node, it will not have printed “Hello world!” to our session on the login node. Instead, all output will be placed in a pair of files, one each for stdout and stderr:

hello.sh.e51978720
hello.sh.o51978720

The names for these files are constructed from the name of the job (which defaults to the name of the program or script that you submitted), e or o, and the job number (assigned by the scheduler). They will be saved in whichever directory you were in when you ran qsub.

Take a look at their contents. Which one contains “Hello world!”? Why?

Next, start learning the useful skills in the following section. Take your time and don’t worry if at first they seem difficult or confusing.

Related forums

Korin’s slides from the first lab session are available in the forums.

What you will learn in this part:

• The data required to train our speech synthesis model
• The toolkit for training, and performing inference with, our model
• Working with a compute cluster:
  • debugging and testing using an interactive session
  • submitting your first substantial job
  • monitoring training progress

Any errors you encounter during this part will be due to human error (yours!) and not the tools or the data. You must successfully complete this part before trying it with your own data.

Do not start this part of the assignment until we tell you to!

Existing speech corpora available to you

To save disk space (which is expensive and costs real money), you must use a shared copy of the existing data. Do not make your own copy of it! Take a look at the shared copy, using an Eddie login node:

# Shared dataset directory
DATA_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw

# Arctic dataset
# SLT speaker, American, female
ls ${DATA_DIR}/cmu_us_slt_arctic/wav
less ${DATA_DIR}/cmu_us_slt_arctic/etc/txt.done.data
# There are other speakers: replace slt with bdl, rms, or clb

# LJ Speech dataset
ls ${DATA_DIR}/LJSpeech-1.1/wavs
less ${DATA_DIR}/LJSpeech-1.1/metadata.csv

Set up your environment

Do the following once, to set up your environment:

# Shared dataset directory
DATA_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw

# Load anaconda
module load anaconda

# Set up the environment path - You only need to run the following commands once.
conda config --add envs_dirs ${DATA_DIR}/anaconda/envs
conda config --add pkgs_dirs ${DATA_DIR}/anaconda/pkgs

# Activate the environment
conda activate py312torch27cuda118

Then, next time you login, all you need to do is:

module load anaconda && conda activate py312torch27cuda118

You should see (py312torch27cuda118) at the start of your shell prompt, to remind you that you have activated this environment. You need to have this environment activated throughout this assignment. (If for some reason you want to deactivate it, just conda deactivate)

Create a new EveryVoice project

EveryVoice is a toolkit for training a model similar to FastSpeech 2. It is freely available on GitHub and has documentation. Do not install your own copy from GitHub! Use the shared copy that is already on Eddie.

# if you didn't already do this in the current session, activate your environment:
module load anaconda && conda activate py312torch27cuda118

# Make sure you are in the right directory
EXP_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/users/s1234567/tts_cw
mkdir ${EXP_DIR}
cd ${EXP_DIR}

# Start the configuration wizard
everyvoice new-project

The wizard will guide you through the process by asking you several questions. The first time you train a model, you will use data from a single speaker (pick one from the available ARCTIC corpora) Here are the answers to some of the questions (the rest should be obvious). To abort or go back one step, press Ctrl-C.

• Where should the Configuration Wizard save your files?
  • Current directory, specified using “.”
• Where is your data filelist?
  • Type “/” and then use the menus to browse to: /exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw/cmu_us_slt_arctic/etc/txt.done.data (or whichever ARCTIC speaker you have chosen to use)
• Select which format your filelist is in: festival
• Which representation is your text in? characters
• Which of the following text transformations would like to apply to your dataset’s characters?
  • press “space” to select Lowercase and NFC Normalization
• Would you like to specify an alternative speaker ID for this dataset instead? no
• language: eng
• Keep the current g2p settings and continue
• Where are your audio files?
  • Browse to: /exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw/cmu_us_slt_arctic/wav/
• Which of the following audio preprocessing options would you like to apply?
  • select only Normalization (-3.0dB)
• Which format would you like to output the configuration to?  yaml

Using multiple datasets to create a multi-speaker (or multi-style) model

Don’t do this for your first model, but you will need this information when training a model on your own recordings.

Training a model on multiple datasets is very easy – just follow the Wizard! For example, to train a 2-speaker model on data from the ARCTIC speaker ‘slt’ and your own recordings:

• Would you like to specify an alternative speaker ID for this dataset instead? yes
• Please enter the desired speaker ID: slt
• …
• Do you have more datasets to process? yes
• …
• Please enter the desired speaker ID: Simon
• …
• Do you have more datasets to process? no

The train a multi-style model, simply treat each speaker+style combination as a unique speaker (e.g., ‘slt’, ‘Simon_reading’, ‘Simon_conversational’).

There are other ways to achieve the same result, which involve creating a filelist that has a speaker ID column. Ask for help in the lab if you want to learn how to do it that way. You will need to use this approach if you only want to use part of an existing corpus (e.g., only the ‘B’ utterances from slt ARCTIC).

Preprocess the data

# Start an interative CPU session with 4 CPU cores for 2 hours.
qlogin -pe interactivemem 4 -l h_rt=2:00:00

# In the unlikely event that you experience a long waiting time, 
# trying adding -P ppls_ssgpu to use our priority job code 
# (this costs money, so only use when necessary)

# Activate your environment
module load anaconda && conda activate py312torch27cuda118

# Set the paths
EXP_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/users/s1234567/tts_cw
# Put the project name you have entered in the configuration wizard here.
TTS_PROJECT=your_project_name

# Go to the project directory
cd ${EXP_DIR}/${TTS_PROJECT}

# Start the preprocessing
everyvoice preprocess config/everyvoice-text-to-spec.yaml

# This should be quite fast - under 3 minutes for 'slt'
# (Ignore any warnings about 'pkg_resources')

# IMPORTANT!
# as soon as you have finished this step, quit the interactive session
# the cluster has a limited number of interactive nodes, 
# and other people might be waiting to use one

# Tip: Ctrl+D is the quickest way to log out
# Otherwise, type this:
logout

You should now be back on a login node. In a terminal, or using VS Code, browse through the contents of ${EXP_DIR}/${TTS_PROJECT}/preprocessed to see what files have been created.

Train the model

Edit line 60 of config/everyvoice-text-to-spec.yaml to specify the vocoder we will be using (note: you cannot use environment variables in this config file):

  vocoder_path: /exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw/hifigan_universal_v1_everyvoice.ckpt

Make your own copy of the training job submission script:

DATA_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw
EXP_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/users/s1234567/tts_cw
TTS_PROJECT=your_project_name

cd ${EXP_DIR}/${TTS_PROJECT}
cp ${DATA_DIR}/train.sh .

modify train.sh to specify

• a maximum run time
• the priority job code
• a meaningful job name (of your choice)
• your University email address

#!/bin/bash

# Grid Engine options (lines prefixed with #$)
# Runtime limit of 6 hours:
#$ -l h_rt=6:00:00
#
# Set working directory to the directory where the job is submitted from:
#$ -cwd
#
# Request one GPU
# IMPORTANT: do not use more than one GPU !
#  it will burn through your available GPU hours faster
#$ -q gpu 
#$ -l gpu=1
#
# Specify a priority project for shorter queue time:
#$ -P ppls_ssgpu
#
# Name your job so you can easily find it:
#$ -N my_meaningful_job_name
#
# Specify an email address to get job notifications:
#$ -M s1234567@ed.ac.uk

# Initialise the environment modules
. /etc/profile.d/modules.sh
module load anaconda
conda activate py312torch27cuda118

# Train the model
everyvoice train text-to-spec config/everyvoice-text-to-spec.yaml

Now we can submit this job to the scheduler

qsub train.sh

Use the skills you learned earlier to monitor this job. You should be able to:

• Examine the queue to see whether the job is still waiting or has started running
• Find the output files and inspect their contents

For an ARCTIC corpus comprising approx. 1100 sentences and approx. 1 hour of speech, training for 1000 epochs should take less than 3 hours.
Log in

What you will learn in this part:

• Performing inference to generate synthetic speech
• How to listen to it

Inference

“Inference” is a fancy word for using our trained model to make a prediction: in this case, it predicts synthetic speech conditioned on the input text.

# Start an interactive GPU session with 1 GPU for 30 minutes
#  (jobs of less than 1 hour should get scheduled much quicker than longer jobs)
qlogin -q gpu -l gpu=1 -l h_rt=0:30:00 -P ppls_ssgpu

# You may need to wait for a compute note to be allocated by the scheduler
# If you experience a long waiting time, try again later

# Once your job is scheduled, you will obtain an interactive session on a GPU node

# On the GPU node, activate your environment
module load anaconda && conda activate py312torch27cuda118

# Set the paths
DATA_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw
EXP_DIR=/exports/chss/eddie/ppls/groups/slpgpustorage/users/${USER}/tts_cw
TTS_PROJECT=your_project_name

# Go to the project directory
cd ${EXP_DIR}/${TTS_PROJECT}

# Perform inference for a single sentence
everyvoice synthesize from-text \
logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt \
-v /exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw/hifigan_universal_v1_everyvoice.ckpt \
-t "Hello World, is my first utterance." \
-a gpu -d 1 --output-type wav

# To save GPU resources, logout as soon as you are finished by type Ctrl+D, or:
logout

To perform inference for a list of sentences stored in a file called, for example test_sentences.txt:

everyvoice synthesize from-text \
logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt \
-v /exports/chss/eddie/ppls/groups/slpgpustorage/tts_cw/hifigan_universal_v1_everyvoice.ckpt \
-f test_sentences.txt \
-a gpu -d 1 --output-type wav

# To save GPU resources, logout as soon as you are finished by typing Ctrl+D, or:
logout

Listening to the synthetic speech

VS Code is able to play audio files. Simply navigate to ${EXP_DIR}/${TTS_PROJECT}/synthesis_output. To copy the audio files to somewhere other than ECDF, you will need to use the filesystem skills you learned earlier.

Preparation

Revise the Module 5 class about evaluation to remind yourself of what was covered there.

You should already have in mind some hypotheses that you want to test, since those would have influenced the additional speech that you have recorded in the studio. Now, you need to formalise those hypotheses, then test them.

It is essential to commit to each hypothesis in writing before proceeding. Spend a little time getting the wording just right, since that will be your guide when designing an experiment that tests the hypothesis (and, for example, doesn’t accidentally test something else).

Remember that you will get a higher mark for demonstrating understanding in your report, than for simply doing more work. Likewise, there are more marks to be obtained from a good evaluation of whatever voice(s) you mange to build, than for making those voices better or training an excessive number of models.

Listening test

You should have at least one (and perhaps two, but not many more) hypotheses that need to be tested using a listening test with multiple listeners. You will be asking a group of listeners to confirm something that you already believe to be true (i.e., to confirm what you have found through your own listening). This is most likely to be that there is some perceptual difference between two experimental conditions, but could also be to confirm that there is no difference.

Don’t use an excessive number of listeners – you won’t get a higher mark for that. Something around 10 listeners should be sufficient for the purposes of this exercise, depending on how many responses you gather per listener.

Other ways to test a hypothesis

You should also have a small number of other hypotheses that do not need a formal listening test, but can be tested in some other way. For example: expert listening (you are the expert!), or an objective measure. You might use qualitative and/or quantitive evaluations here.

Don’t test an excessive number of hypotheses: you won’t be able to fit them all in your report. It’s better to keep the number smaller, so that you can test each one rigorously, and write it up well.

Implementation

Use the forum to find out more about tools for running an evaluation.

Advice

You should run all your ideas past us in a lab session, before actually executing any evaluation. We will help you sharpen your hypotheses, and check that your proposed evaluation method will actually test the hypothesis (and only the hypothesis).

Writing up your report is an exercise in following a specific style guide, as you would have to do for a published paper.

In this section:

• Formatting
• Content
• Scientific writing
• Submission and marking

Formatting

Your report must conform to the following specification:

• Use IEEE Transactions double-column style, with single line spacing
  • Templates for Latex and Microsoft Word can be found the IEEE template selector here :
    • To obtain a zip file of the, e.g. LaTeX, template files select:  Transactions, Journals Letters > IEEE Transactions on Audio, Speech, and Language Processing > Original research and Brief > LaTeX
  • If you are using LaTeX (very strongly recommended), use the bare_jrnl.tex template.
  • Follow the style as closely as you can, including the in-text citations and references.
  • Replace the author name on the first page with your exam number + wordcount, and omit the author profile and photograph on the last page.
  • Include the abstract and index terms.
  • Your report should have the same overall format (but not necessarily the same structure) as the PDF versions of published papers: Example 1; Example 2; Example 3.

You should ensure that figures and graphs are large enough to read easily and are of high-quality (with a very strong preference for vector graphics, and failing that high-resolution images). It is recommended to use single-column figures as far as possible. However, a small number of large figures or table may be included at full page width, spanning both columns, if they are at the top or bottom of a page.

Length

• Word limit: 5000 words
  1. including: headings, footnotes, words within figures & tables; captions
  2. excluding: numerical data within figures and tables; references; statement on use of AI
• Page limit: no limit enforced, but typical papers will be 7 or 8 pages long
• Figures, graphs, & tables: no limit on number (but excessive or unnecessary ones may impact your mark in the “Scientific writing” category)

The word limit is a hard limit. The markers will simply not read anything beyond these limits, and your mark will only be based on what they read.

Content

Originality

The rules on originality for this assignment are strict, so that you have to explain everything your own way. That will help you learn better. The rules are stricter than for most academic journals, and that is intentional.

These rules are not designed to trick you or catch you out. They are designed to make you focus on your own understanding of the material.

All material in your submitted report (including text, figures, graphs, plots, tables, etc) must be your own original work. You must not include (even with proper attribution) any material from other sources.

These are not acceptable:

• Using a figure from another paper, or one of your own from a previous assignment
• Quotes, even if properly marked as such and with attribution
• Citing something without reading it (e.g., because you found it referenced in some other source)
• All of the unacceptable uses of generative AI for assessment listed in the university’s AI policy
• Any form of academic misconduct including, amongst other things, falsifying experimental results

whilst the following are acceptable:

• Your own explanations, in your own words, of concepts and ideas from other sources (with appropriate attribution of whose ideas those are, typically by citation)
• You are allowed, if you wish, to use AI to generate your recording script and/or test material for evaluating your system, in which case you must acknowledge this and provide details of your method, in your submitted report.

Use of AI

In addition to the above, this course requires you to comply with everything in the university’s AI policy. Your attention is drawn in particular to item 2 in the list of unacceptable uses of generative AI: “English is the language of teaching and assessment at Edinburgh – machine translation is treated as false authorship and is not acceptable.”. You are therefore very strongly advised to work in English only, from your initial notes and early drafts through to the final version for submission. This has the added benefit of being the best way to improve your English! For the same reason, you are also advised to follow the original, English language instructions for the assignment.

You probably know this by now, but it’s worth restating. Generative AI based on Large Language Models generally produces fluent, confident-sounding, and grammatically-correct writing. It can also give the illusion of ‘scholarly’ writing. However, this writing style is often verbose and over-complex. Perhaps it would score highly in an English language examination, but real scholars simply don’t write like that: we are succinct, precise, and direct.

Submitting writing produced by Generative AI isn’t just against University policy. It also means that any feedback from the marker will not be about your writing or your understanding, so it won’t help you keep learning and improving.

Statement on your use of AI

At the end of your report, you should briefly describe any use of AI tools in doing this assignment: for example, grammar checking, or using a generative AI chat app to investigate the topic. If you used tools based on Large Language Models (e.g., ChatGPT), describe the prompts that you used.   If you did not use any AI in your work, you must still include this section but can simply write “none”. Text in this section does not count towards the word limit.

Scientific writing

You should take advantage of all of the following sources of help to improve your scientific writing:

1. Advice from the Speech Processing course

You should check out the writing tips provided for the Festival exercise from Speech Processing – they apply equally well here. Students who did not take Speech Processing will be offered additional help and guidance on writing the report. They can ask for this at any point during the course, but not later than 1 week before the submission date.

Tips on writing

2. PPLS skills centre

PPLS Skills Centre

3. Scientific writing forum

4. Forum for this assignment (login required)

Submission and marking

Submission

• do not include your name or student number anywhere in your report
• submit a single document in PDF format via Learn; the filename must be in the format examnumber_wordcount.pdf (e.g., “B012345_4672.pdf”)
• state your word count on the first page of the report (e.g., “wordcount: 4672”)

Your work may be marked electronically or we may print hardcopies on A4 paper, so it must be legible in both formats. In particular, do not assume the markers can “zoom in” to make the figures larger.

If, after marking your report, there is reasonable uncertainty whether you hold the knowledge presented in your report – including that you did all the practical work described yourself (e.g., speech recorded, models trained, experimental results), we have the option of inviting you to an Affirmation Meeting with the Course Organiser.

Marking scheme

You must read the 2025-26 structured marking scheme because it will help you focus your effort and decide how much to write in each section of your report.

Then design a structure for your report that is consistent with the marking scheme, but do not simply copy the marking scheme sections as your headings because that is not the best structure.

 A well-structured report will be awarded marks under “Scientific writing”. Making the relationship between your report structure and the marking scheme clear will help the marker find all the places they can give you marks.