• A working Python installation (exercise tested with versions 2.7 and 3.6 but should be fine with other versions)
• Festival (use the latest version), plus
  1. Edinburgh Speech Tools (from the same place as Festival), compiled with the Python wrappers
  2. Various libraries (see Festival / Edinburgh Speech Tools documentation for dependencies) including lib32ncurses5-dev and libX11-dev
• HTK (if you use 3.4.1 then fix this bug related to ERROR +8522)
• sox

WARNING: setting this exercise up on your own machine is non-trivial! It requires advanced skills in compiling code from source. This page is only intended to help experts set up machines for summer school courses.

In this practical exercise, you’re going to build a unit selection voice for a text-to-speech synthesiser from your own voice. You will create a working voice that can be loaded into Festival and used to generate intelligible speech.

This exercise focuses mainly on the waveform generator stage of the synthesis pipeline, although we may make some minor changes to the front end, by adding a few words to the pronunciation dictionary.

Before starting, be a proper engineer and

• keep a logbook to record every single step

You’ll find this invaluable if you need to repeat any steps, and your notes will also be useful for writing up a lab report at the end.

To build your synthetic voice, you will follow step-by-step instructions and use a variety of existing tools. If you’re doing this exercise on your own, you now need to download and install Festival and HTK. This exercise was originally developed for Mac OS X but also works on Linux. Don’t attempt to do this exercise on Windows, unless you are a masochist.

Here are the main stages in this exercise:

1. Select a recording script.
2. Make the recordings in the studio.
3. Prepare the workspace.
4. Prepare the recordings in the required format, and sanity check.
5. Label the speech.
6. Pitchmark the speech.
7. Build the voice.
8. Evaluate the voice.
9. Write up.

Each stage will be described, and I’ll also provide links to relevant material on this website, such as blog posts describing some part of the theory, or discussion forums where I answer students’ questions.

Related posts

CUI 2024 slides available

Pipeline architecture for TTS

Related forums

I’m going to assume that you’ll put everything in your Documents directory. Download this zip file, unzip it (you might find that it is unzipped automatically after download) and move the resulting “ss” folder to your Documents folder. Open a Terminal and check that everything looks OK. bash$ is the bash prompt (yours may look different).

bash$ cd ~/Documents
bash$ cd ss
bash$ ls

and you should see three folders (pm, recordings, wav) and three files (setup.sh, utts.data, utts.pauses).

If you installed all the tools needed for this exercise yourself, you now need to edit the two paths in setup.sh. The variable SSROOTDIR should point at the root directory of your installed tools, and FROOTDIR should point at the Festival installation within there.

Every time you open a new Terminal to work on this exercise, you need to run the setup.sh script. Let’s do it now:

bash$ source setup.sh

You won’t see any output, but various environment variables have been configured for you. By the end of the exercise, you’ll have more folders than this; here’s the complete list:

We need to select a script for recording. The standard method for this involves greedily selecting sentences, one by one, from a large text corpus (e.g., novels or newspapers) in order to maximise phonetic (and possibly prosodic) coverage. In the first part of this exercise, we will simply use the existing CMU ARCTIC script.

This would result in about one hour of speech if you record the whole thing, but I suggest you start by recording only the ‘A’ set of 593 prompts and build a voice from those. You can go back and record the ‘B’ set later if you wish, or you could spend that time recording more of your own material, or on other improvements or evaluation.

Because recording will take a little time (around 5 hours in the studio per hour of speech material obtained), you should get started on recording the ARCTIC sentences immediately, and design your additional sentences in parallel.

Tip

You can get started before your recordings are ready, by downloading one of the ARCTIC corpora (I recommend ‘slt‘). These are actually complete Festival voices, so you should simply copy only the waveforms into the ‘wav’ directory of your ‘ss’ directory (and discard everything else), then proceed with this exercise. You should work in separate copies of the ‘ss’ directory for each voice you build.

If the download is slow, then you can copy the waveforms from our local copy of ARCTIC like this:

bash$ rsync -avu /Volumes/Network/courses/ss/corpora/ARCTIC/cmu_us_slt_arctic/wav/ ~/Documents/ss/wav/

Log in
The utts.data file
This file is the main index of the unit selection database. Festival uses it to discover which files it can select units from.
You will find a file called utts.data in your ss folder. This is the file that Festival uses to define the unit selection database. The initial version of this file contains the CMU ARCTIC sentences. 
The utts.data file looks like this:
( arctic_a0001 "Author of the danger trail, Philip Steels, etc." )
( arctic_a0002 "Not at this particular case, Tom, apologized Whittemore." )
( arctic_a0003 "For the twentieth time that evening the two men shook hands." )

Each line of the file represents one utterance in your database, in the following format:

An opening parenthesis ‘(‘
A unique identifier which will be used as the filename for any files which relate to this utterance. The ARCTIC sentences use filename of the format arctic_lnnnn; your own supplemental material should used identifiers of the format yourname_nnnn (where nnnn starts again from 0001)
The text, inside double quotes
A closing parenthesis ‘)’.

Adding your own material
Whilst the ARTIC script gives general diphone coverage, it's not ideal for synthesising all types of sentence. You can try to improve your voice's naturalness for one particular domain, by adding more material to the database.
The ARCTIC prompts come from old novels, and were selected under constraints described in the technical report, which means that they don’t give very good coverage of common short phrases, such as greetings. You can therefore supplement the ARCTIC script with some of your own material. 
If you are able to code, then you can skip this section and move on to the next section which covers automatic text selection in which you will write your own algorithm to select the text to record.
If you can’t code, then you can add some prompts to cover the following:

5 sentences which include your name in different contexts (e.g., phrase-initial and phrase final);
10 short frequently-used set-phrases such as “Hello.”, “Hi.”, “How are you?”, “Goodbye.” and so on;
about 50 sentences aimed to cover a very small limited domain.

Choose a limited domain where you can include all the words in the vocabulary items several times each (in different contexts and positions) within just 50 sentences, but from which you are able to synthesise a much larger range of new sentences. For example:

Times and dates.
Street addresses, with a very small set of street names (e.g., North Street, North Avenue, West Street, …) and a few numbers.

Invent your own domain, and don’t use one of those examples! It’s fine to artificially restrict the vocabulary and types of sentences: it’s just an exercise, after all.
After you design your additional material, prepare a new file in the same format as utts.data, containing your sentences – this file will be used by the SpeechRecorder tool to record those sentences. Number the sentences in a meaningful way, with your name and a unique number, in a similar way to the main scripts. You do not need to include the ARCTIC script: that is already loaded into SpeechRecorder. In addition, you need to add your new sentences to the end of the utts.data file.
Important: use a plain text editor (such as Aquamacs) to make those files – you must avoid non-ASCII characters.
Automatic text selection
Not all students will attempt this, but how about implementing your own greedy text-selection algorithm?
Note: this part of the exercise will require some basic coding skills (e.g., in Python).
Instead of, or as well as, adding a small set of handcrafted sentences from a limited domain, you can implement a simple text-selection algorithm and use that to select your additional material.
The most advanced version of this is to select an additional set that is the same size (in terms of recorded speech) as the ARCTIC ‘A’ set. You can then build voices from ARTIC ‘A’ or from your material, and compare them in a listening test.
Remember that you’ll also need to record this material in the studio! Make sure to record under exactly the same conditions (same studio, same microphone, etc) in all of your recording sessions, so that you have the option of combining the data to build a single voice.
Some ideas to get you started:

You could try to exactly replicate the ARCTIC text selection algorithm, or try your own ideas.
You could use much more up-to-date source text, instead of old novels.
If you select from limited domain source text, then you should be able to build a really good domain-specific voice. One challenge will be finding a suitably large body of such text (e.g., scrape the web for sports reports).

Practice makes perfect, so you need to allow a little time for learning how to make good recordings. If you are using a recording studio, you will work with a partner, one of you acting as recording engineer whilst the other is the voice talent. If you are recording at home, you will do both jobs yourself.

For 2023-24, students should use the University recording studios: instructions are below.

Microphone technique

Good technique is important for high quality recordings, and always remember that consistency is crucial, so take a few photos of the setup so you can reproduce it in subsequent sessions.

With a headset microphone, it’s important to place it to one side of the mouth to avoid breath noises

don’t place it below the mouth because you will still get breath noises from the nose

and don’t touch it whilst recording!

With a stand-mounted microphone, again you need the microphone placed to avoid breath noises from the mouth or nose, and kept at a constant distance (20-30cm). Make several test recordings to find a position that sounds good. During the recording sessions, the engineer should keep an eye on your voice talent: don’t let them move around in the chair.

Getting the recording level correct

With digital recording, it’s essential that you never ‘hit the red’ when recording because you will get hard clipping and that will sound very bad (as well as potentially interfering with the signal processing we need to do later).

But on the other hand, you do want to record at the highest level possible (what a recording engineer would call ‘hot’) so that you make the most of the available bit depth. Recording at too low a level is equivalent to using fewer bits per sample, and can also make any imperfections in the audio signal chain (such as electrical noise within the microphone amplifier) more obvious.

Recording software

If you’re recording on a Mac, then you can use CSTR’s SpeechRecorder software that presents each prompt to the voice talent, and saves the recordings in individual files. Here’s the manual. To load your own sentences into this tool, they need to be in Festival’s standard ‘utts.data’ format.

If you don’t have a Mac, then try this Python alternative to SpeechRecorder created by previous student Tim Loderhose, and now updated and maintained by Dan Wells.

Making good, consistent recordings

You will find that you can probably record for a maximum of 2 hours at a time, with short breaks every 30 minutes or so. After that your voice will start to become creaky. Stop when this happens: you need your voice to stay consistent (it may also be damaging to your voice to speak for excessively long periods). Some recording tips:

• Switch your phone, and that of anyone else in the studio, off or place it in ‘airplane’ mode (not just silent mode) to avoid interference.
• Take a bottle of water with you and take frequent sips during recording.
• Write down (or take a photos of) the recording levels you are using and set the same levels in every session.
• Ensure chair, microphone, etc. are positioned the same way in every session (again, photos are helpful here).
• Make sure any ventilation fans are switched off during recording.
• When you are speaking, ensure that you are not fidgeting, playing with any of the cables, your hair, etc…

Of course, you should make plenty of test recordings at the outset, and listen back to them carefully over headphones to spot any problems. Once you have perfected your technique, go ahead and record the ARCTIC ‘A’ set. You should build a voice from this, to confirm that you have made sufficiently-good quality recordings, before returning to the studio to record your own material.

During the actual recording, try to get each sentence correct in a single attempt. Don’t waste time on multiple takes, except in those few cases where you made a major error. The engineer should keep notes about any sentences that need checking after the recording session.

At the end of each session, make back up a copy of your recordings on a memory stick (if using a recording studio), and/or back them up somewhere secure.

Studio recordings use a high sampling rate (48 kHz or 96 kHz); this is unnecessarily high for the purposes of this exercise. 16 kHz will suffice and will make the files smaller and more manageable. Keep backups of the original recordings somewhere safe, in case you make a mistake.

Start by copying the original waveforms from the studio into the ‘recordings’ folder.

Choose amongst the multiple takes

In the studio, you probably made multiple attempts at a few of the more difficult sentences. It’s likely that the last take is the one you want (your engineer will have kept notes to help you), so you can move the other takes to somewhere other than the ‘recordings’ folder.

Sanity check

Before proceeding, make sure that you then have exactly one wav file per line in your utts.data file. Listen to all the files (Mac tip: use the Finder, navigate with the cursor keys, and use the spacebar to play each file). If you find any mismatches with the text (e.g., a substituted word), then an expedient solution is to edit the text (and not to re-record the speech). Also make sure that the file naming exactly matches utts.data.

The SpeechRecorder tool adds a suffix to the file basenames to indicate the ‘take’. You need to remove this so that the basenames exactly match the utterance identifiers in utts.data. Write a script to remove these suffixes (noting that the suffix might vary: “_1”, “_2”, etc. depending on which take you selected for each prompt).

In general, you should not re-record any utterances. A few missing utterances is not a major problem, for the purposes of this exercise.

Downsample

Here’s how to downsample a single file, and save it in the required RIFF format:

bash$ ch_wave -otype riff -F 16000 -o wav/arctic_a0001.wav recordings/arctic_a0001.wav

and you need to write a little shell script that does this to all the files in your ‘recordings’ folder. If you happened to record your data at 24bits instead of 16bits, you’ll need to use sox to change the bit depth, and you can downsample at the same time using sox instead of ch_wave:

bash$ sox recordings/arctic_a0001.wav -b16 -r 16k wav/arctic_a0001.wav

Now listen to a few of the files after downsampling, to check everything worked correctly.

Related posts

CUI 2024 slides available

Aliasing

Sampling and quantisation

Related forums

Before you continue, make sure you have completed the following:

1. you have finished recording at least the ARCTIC ‘A’ set;
2. you have a single utts.data file with a single line entry for each utterance;
3. you have checked your recorded data, and have a wav folder containing an individual .wav file for every utterance in utts.data;
4. you have checked to ensure the file naming and numbering is correct.

If you haven’t recorded the full ARCTIC script, then edit utts.data (obviously you should make a backup copy first) so that it only includes prompts for which you have a corresponding wav file.

The next stage is to create time-aligned phonetic labels for the speech, using forced alignment and the HTK speech recognition toolkit. First you must set up a directory structure for HTK:

bash$ setup_alignment

This creates a directory called alignment containing various HTK-related files. The script will also tell you that you need to make a couple of files: you will do that in the next step.

Related forums

make_pm_wave generates pitch marks from your waveforms:

bash$ make_pm_wave -[mf] pm wav/*.wav 
bash$ make_pm_fix pm/*.pm

Note that -[mf] means that you should specify either -m or -f, for male or female.

make_pm_fix adjusts the pitch marks so they each align with a peak in the waveform.

Optional: view the pitch marks

To view pitch marks (so that you can check they are accurate) you need to convert them into label files. These can then be viewed in wavesurfer along with the corresponding waveform.

bash$ mkdir pm_lab
bash$ make_pmlab_pm pm/*.pm

Optional: adjust the pitch marker settings

You might need to adjust some of the parameters of the pitch marker to make it work well on your voice. If you need to make changes, copy the script $MBDIR/bin/make_pm_wave to your ss directory and make changes to your own copy. Then run the script as ./make_pm_wave so that your version is run rather than the default version.

Look for the DEFAULT_PM_ARGS lines in the script. The min and max values are minimum and maximum times between pitch marks (i.e., the pitch period, which is 1/F0). The lx_lf and lx_hf values are frequency values in Hertz and control a filter which is used to remove both high and low frequencies before the actual pitchmarking is performed.

If you decide to change the default values, you need to find out what the range of F0 is for your voice. You can use Praat’s edit command to examine a few waveforms along with pitch contours, and make a note of the maximum and minimum values you find (excluding any obvious errors made by Praat). Set the high and low frequency filters to just above and just below (maybe 40Hz either direction) this pitch range, as well as setting the appropriate min and max values for your pitch range. Re-run the pitch marker and check the output again. You will need to convert the pitchmarks to label files in order to view them.

It is helpful to remove the -fill option when tuning the pitchmarker settings, and to re-instate it just before pitchmarking all your data at the end. For more tips try this short guide.

Related forums

We’re nearly there, and the remaining steps are mostly fully automatic.

To run the voice, start Festival and load the voice as follows (changing rpx to edi or gam if appropriate):

bash$ festival
festival>(voice_localdir_multisyn-rpx)
festival>(SayText "Hello world.")

So far, we have just built a single voice. It would be good to have something to compare that against.

The methodology to use in this part is to create multiple versions of your voice, and then to compare them informally or in a listening test.

The way to have multiple voices is to make a complete copy of your ss folder for each variant (you should move the original recordings elsewhere first (to save space). You should also use symbolic links so that all the variants share the same wav, mfcc, and lpc folders. Not only does this save even more space, it is also good engineering practice to avoid unnecessary copies of data.

The first thing to try is to revisit each of the stages in building the voice, and see whether there is anything you can improve. For example, you could adjust the pitchmarking parameters to more closely fit your voice. Then, try some or all of the following variations:

How to save a synthesised waveform

festival> (set! myutt (SayText "Hello world."))
festival> (utt.save.wave myutt "myutt.wav" 'riff)

and of course you can change myutt.wav to any other name you like (but always keep the .wav suffix).

Use the forum to find out more about tools for running an evaluation.

Writing up your report is an exercise in following a specific style guide, as you would have to do for a published paper.

Formatting

Your report must conform to the following specification:

• Use IEEE Transactions double-column style, with single line spacing
  • Templates for Latex and Microsoft Word can be found the IEEE template selector here :
    • To obtain a zip file of the, e.g. LaTeX, template files select:  Transactions, Journals Letters > IEEE/ACM Transactions on Audio, Speech, and Language Processing > Original Research > LaTeX
  • If you are using LaTeX (very strongly recommended), use the bare_jrnl.tex template.
  • Follow the style as closely as you can, including the in-text citations and references.
  • Replace the author name on the first page with your exam number + wordcount, and omit the author profile and photograph on the last page.
  • Include the abstract and index terms.
  • Your report should have the same format as the PDF versions of published papers: Example 1; Example 2; Example 3.

You should ensure that figures and graphs are large enough to read easily and are of high-quality (with a very strong preference for vector graphics, and failing that high-resolution images). It is recommended to use single-column figures as far as possible. However, a small number of large figures or table may be included at full page width, spanning both columns, if they are at the top or bottom of a page.

Length

• Word limit: 5000 words
  1. including: headings, footnotes, words within figures & tables; captions
  2. excluding: numerical data within figures and tables; references
• Page limit: no limit enforced, but typical papers will be 7 or 8 pages long
• Figures, graphs, & tables: no limit on number (but excessive or unnecessary ones may impact your mark in the “Scientific writing” category)

The word limit is a hard limit. The markers will simply not read anything beyond these limits, and your mark will only be based on what they read.

Originality

The rules on originality for this assignment are strict, so that you have to explain everything your own way. That will help you learn better. The rules are stricter than for most academic journals, and that is intentional.

These rules are not designed to trick you or catch you out. They are designed to make you focus on your own understanding of the material.

All material (including text, figures, graphs, plots, tables, etc) must be your own original work. You must not include (even with proper attribution) any material from other sources.

These are not acceptable:

• Using a figure from another paper, or one of your own from a previous assignment
• Quotes, even if properly marked as such and with attribution
• Citing something without reading it (e.g., because you found it referenced in some other source)

whilst the following are acceptable:

• Your own explanations, in your own words, of concepts and ideas from other sources (with appropriate attribution of whose ideas those are, typically by citation)

I encourage you to ask for clarification on the private forums, if you need it.

Marking scheme

You must read the structured marking scheme (this document has been updated for 2023-24) because it will help you focus your effort and decide how much to write in each section of your report.

Then design a structure for your report that is consistent with the marking scheme, but do not simply copy the marking scheme sections as your headings because that is not the best structure.

 A well-structured report will be awarded marks under “Scientific writing”. Making the relationship between your report structure and the marking scheme clear will help the marker find all the places they can give you marks.

Submission

• do not include your name or student number anywhere in your report
• submit a single document in PDF format via Learn; the file name must be in the format examnumber_wordcount.pdf (e.g., “B012345_4672.pdf”)
• state your word count on the first page of the report (e.g., “wordcount: 4672”)

Your work may be marked electronically or we may print hardcopies on A4 paper, so it must be legible in both formats. In particular, do not assume the markers can “zoom in” to make the figures larger.

Recommendations

You should also check out the writing tips provided for the Festival exercise from Speech Processing – they apply equally well here – and please use the forums to obtain general advice on scientific writing.

Tips on writing

Related forums