Training a DNN involves reading the training data many times (the complete set is read every epoch), and then saving the DNN to a file which will be 100s of MB. Network speed can be a problem. To make things go faster, you need the training data and the saved models to be on your computer’s local disk.
Working on local disk
bash$ mkdir /tmp/mylocalspace
Then change the value of work in feed_forward_dnn_WORLD.conf to point to the new location.
Staging the data to local disk
It’s smarter to use rsync, than cp:
bash$ rsync -avul ./data/ /tmp/mylocalspace/data/
(note: the trailing slashes make a difference). In fact, you don’t need to copy everything in data. You could save time and only copy the normalised input and output features and the file list: nn_no_silence_lab_norm_340 nn_norm_mgc_lf0_vuv_bap_199 file_id_list.scp
Copy files back to your own filespace
After you’ve finished training, you will want to copy the trained model (and optionally the log files and synthesised speech) back to your own file space. You can use rsync to do that too.
There is no need to copy the data back, of course; that will not have changed.
