Malloc errors with run_lstm.py

[Etienne D]

Hello,

Running the run_lstm.py script for training with different neural network architectures, I regularly get error such as this one:

2016-06-03 16:40:58,589 DEBUG main.train_DNN: calculating validation loss
python(96444,0xa08151d4) malloc: *** mach_vm_map(size=181813248) failed (error code=3)
*** error: can’t allocate region
*** set a breakpoint in malloc_error_break to debug
2016-06-03 16:41:01,141 CRITICAL main : train_DNN threw an exception
… Python traceback …
MemoryError:
Apply node that caused the error: Dot22(Subtensor{int64:int64:int8}.0, W_xf)
Toposort index: 49
… More info …

I was originally calling python /Volumes/Network/…/run_lstm.py from a wrapper shell script, which systematically threw this error, so I tried without, which usually works but still ocasionally fails. I attempted running it using both Network account and local disk, if that matters.
However, this never happens when synthesising utterances with the same program, even when called from my wrapper shell script.

Do you have any ideas as to why that may happen, and any suggestions as to how to avoid it, and also maybe allow for the program to be called from a shell script?

Thank you,
Étienne

[Simon]

A malloc (“memory allocation”) error of “can’t allocate region” suggests that you are running out of memory (RAM). Try reducing the minibatch size.

In sequence training, the minibatches need to be constructed from entire utterances, rather than randomised frames. So, the minibatch size will vary slightly, and not be constant. This may be why you only get this error seemingly randomly.

[Srikanth R]

Also, please use the latest configuration file:
https://svn.ecdf.ed.ac.uk/repo/inf/dnn_tts/configuration/configuration.py

There was a small bug (incorrect data type initialization) which prevented training in sequential mode even when set to True. That was corrected and updated later (from Revision no. 252).

[Simon]

I’ve updated …/dnn_tts/configuration/configuration.py in the centrally installed version.

[Etienne D]

Thank you, reducing the batch size worked and I’ve been able to wrap the whole thing in my script.

[Author]

Posts