information gain

[Ruiduan L]

In the worked example1, the concept(information gain) has been mentioned. It is the reduction of bits after splitting mathematically, while what’s the exact meaning of it?

[Simon]

When we partition some data points using a binary question, we hope to make the distribution of values of the predictee less uniform and more predictable. In other words, we try to reduce the entropy of the probability distribution of the predictee.

If we manage to do that, we have gained some information about the value of the predictee. We know more about it (= we are more certain of its value) after the split than before it.

The reduction in entropy from before to after the split is the information gain, measured in bits.

[other part of question answered separately – please include only one question per post]

[Author]

Posts