Normalisation

[Shubha G]

The first main video this week is titled “Tokenisation and normalisation.” My question may be trivial, but I feel like I may have missed something important: which part of the process is normalisation?

[Simon]

Normalisation means converting the sequence of tokens into a sequence of words. A word is something that we can attempt to look up in a dictionary, or pass to the Letter To Sound module. That is, a word is something we can pronounce(i.e., say out loud).

[Author]

Posts