Logs in Entropy Equation

[Loralynn M]

In the equation for entropy, I am wondering about the use of logs and why we use them here. Logs tend to pop up in lots of equations regarding NLP, and in some situations (all?) it is to avoid underflow, but I’m wondering if that’s true in this situation as well.

[Marcel D]

To answer your question, I think that apart from the underflow problem that you mention, logarithms are also used in this case to compute how many bits are necessary to exchange information to another person, computer etc. What you’d want is the information to be sent as efficiently as possible. Computers exchange this information via bits, where each bit can have 2 values, 1 or 0. To sent the information efficiently, you want to give the option with the highest probability the shortest possible bitstring, as this will save us on the number of bits needed to transmit information. Using the 2log in entropy, we can compute what this minimum average number of bits is that is needed to transfer information.
In the example that Simon gave, you will probably assign the shortest bitlength to the item with the highest possibility (eh, 3/7) and longer bitstrings to the other possibilities. If you would assign eh the bitstring 0, ao the bitstring 10 and aa the bitstring 11, you can send any bitstring to another computer and still distinguish what was meant. For example, the bitstring 01011100 can only be split in 0-10-11-10-0, meaning eh-ao-aa-ao-eh. This is a toy example, but you can probably imagine that if you have a large collection of words/sounds, it is quite useful to limit the length of bitstrings as much as possible.

[Simon]

Yes – the log (base 2) is converting to bits.

Log (base 2) comes up quite often in this context, and related ones. For example, the depth of a binary decision tree is of the order of log (base 2) of the number of leaves.

[Author]

Posts