Information Gain Calculator

Information gain is the drop in entropy that a split produces: the entropy of the parent node minus the size-weighted average entropy of its child nodes, measured in bits. This information gain calculator takes the class counts in each child and returns the gain, the split information, the gain ratio and the matching Gini impurity figures, with a tree diagram and the arithmetic.

What is information gain in a decision tree?

A decision tree is grown one question at a time. At every node the algorithm lists the candidate splits, scores each one, and keeps the best. The score answers a single question: how much purer are the groups after the split than the group before it?

Purity needs a number, and the two usual choices are entropy and Gini impurity. Both are 0 when a node holds one class only and largest when the classes are evenly mixed. The recipe is the same for either measure: compute the impurity of the parent, compute the impurity of each child, average the children with weights equal to their share of the samples, and subtract. With entropy the difference is called information gain; with Gini it is the Gini gain, or impurity decrease.

What are the formulas for entropy, information gain and Gini impurity?

• \(S\) is the parent node and \(S_1, \dots, S_m\) are the \(m\) child nodes the split creates.
• \(p_i\) is the fraction of a node’s samples that belong to class \(i\), out of \(c\) classes.
• \(n\) is the number of samples in the parent and \(n_k\) the number that go to child \(k\), so \(n_k/n\) is that child’s weight.
• \(H\) is entropy and \(G\) is Gini impurity. A term with \(p_i = 0\) contributes nothing: \(0 \log_2 0\) is taken as 0.

The logarithm is base 2, so entropy and information gain are in bits. With \(c\) classes entropy runs from 0 to \(\log_2 c\) (1 bit for two classes, 2 bits for four) and Gini impurity from 0 to \(1 – 1/c\). The weighted child entropy is an average over the branches, the same kind of sum the expected value calculator works out, and each child’s class mix is a conditional distribution, P(class | attribute value), of the kind you can check with the conditional probability calculator.

Worked example: the Outlook split in the play-tennis data

The classic 14-row “play tennis” table has 9 days labelled yes and 5 labelled no. Splitting on Outlook gives three children: sunny (2 yes, 3 no), overcast (4 yes, 0 no) and rain (3 yes, 2 no). These are the calculator’s default inputs.

1. Parent entropy. \(-\tfrac{9}{14}\log_2\tfrac{9}{14} – \tfrac{5}{14}\log_2\tfrac{5}{14} = 0.409776 + 0.530510 = 0.940286\) bits.
2. Child entropies. Sunny: \(-\tfrac{2}{5}\log_2\tfrac{2}{5} – \tfrac{3}{5}\log_2\tfrac{3}{5} = 0.970951\). Overcast is pure, so 0. Rain mirrors sunny: 0.970951.
3. Weighted child entropy. \(\tfrac{5}{14}(0.970951) + \tfrac{4}{14}(0) + \tfrac{5}{14}(0.970951) = 0.693536\).
4. Information gain. \(0.940286 – 0.693536 = 0.246750\) bits.
5. Gain ratio. The child sizes 5, 4 and 5 have split information 1.577406, so the ratio is \(0.246750 / 1.577406 = 0.156428\).
6. Gini. The parent is \(1 – (9/14)^2 – (5/14)^2 = 0.459184\). The children are 0.48, 0 and 0.48, which weight to 0.342857, a Gini gain of 0.116327.

How does a decision tree choose which attribute to split on?

It repeats that calculation for every candidate and takes the largest drop. Here are all four attributes of the tennis table, each scored against the same parent (9 yes, 5 no).

Outlook wins on every criterion, so it becomes the root. The overcast branch is already pure and turns into a leaf; the sunny and rain branches are split again by the same procedure, each using only its own five rows. Notice how close Outlook and Humidity are on gain ratio (0.156428 against 0.151836) even though their raw gains are far apart: the three-way split pays a penalty that the two-way split does not.

Entropy or Gini impurity: which should you use?

In the table above the two measures rank the four attributes in the same order, and that is the usual outcome. They are both concave functions that peak at an even class mix, so they rarely disagree about the best split, and when they do the resulting trees tend to perform about the same.

Gini avoids the logarithm, which makes it slightly cheaper to evaluate over many candidate thresholds, and that is one reason it is the default in scikit-learn. Entropy has the advantage of a unit: a gain of 0.246750 bits has a direct information-theoretic reading. It is also the quantity behind the training loss of most classifiers, which the cross-entropy loss calculator evaluates for predicted probabilities.

Why does information gain favour attributes with many values?

Because chopping the data into many small groups makes each group purer almost by accident. Take the extreme case: split the 14 tennis days on a day-ID column, giving 14 children with one row each. Every child is pure, the weighted child entropy is 0, and the information gain is the full 0.940286 bits, nearly four times the gain of Outlook. Yet the split is useless for prediction, since a new day has an ID the tree has never seen.

Gain ratio, introduced with C4.5, divides the gain by the split information, which grows with the number and evenness of the branches. For the day-ID split it is \(\log_2 14 = 3.807355\), so the ratio falls to \(0.940286 / 3.807355 = 0.246966\). The penalty is real but on a table this small it still leaves the ID column ahead of Outlook, which is why identifier-like columns should be removed before training rather than left for the criterion to reject. Paste fourteen lines of 1 0 and 0 1 into the tool to see it.

Where information gain is used in machine learning

• Growing trees. ID3 picks the attribute with the highest information gain, C4.5 uses gain ratio, and CART uses Gini with binary splits. Random forests and gradient-boosted trees repeat the same impurity-decrease search at every node of every tree.
• Feature importance. Impurity-based importances in tree ensembles add up the weighted impurity decrease each feature achieved across all its splits.
• Feature selection. The information gain of a feature about the label is the mutual information between them, a common filter score for ranking features before modelling.

A split criterion only says how the tree was built, not how well it predicts. After training, measure the result on held-out data with the confusion matrix calculator, or summarise precision and recall in one number with the F1 score calculator.

Common mistakes

• Averaging child entropies without weights. The plain mean of 0.970951, 0 and 0.970951 is 0.647301; the correct size-weighted value is 0.693536. The two only agree when every child is the same size.
• Mixing logarithm bases. Natural logs give nats, not bits. The parent entropy above is 0.940286 bits but 0.651757 nats. Rankings do not change, but numbers from two sources will not match.
• Treating 0 log 0 as an error. A pure child has a zero count. Its term is defined as 0, not NaN.
• Comparing entropy numbers with Gini numbers. A gain of 0.246750 bits and a Gini gain of 0.116327 describe the same split on different scales. Compare splits within one criterion.
• Expecting scikit-learn to make a three-way split. Its trees are binary. It would test Outlook as a series of yes/no questions on encoded columns, so the per-node numbers differ from the multiway table above.

How do you calculate information gain in Python?

scipy.stats.entropy accepts raw counts, normalises them, and handles zero counts. Pass base=2 to get bits.

import numpy as np
from scipy.stats import entropy

# rows = sunny, overcast, rain; columns = yes, no
children = np.array([[2, 3], [4, 0], [3, 2]])
parent = children.sum(axis=0)        # [9 5]
sizes = children.sum(axis=1)         # [5 4 5]
w = sizes / sizes.sum()

h_parent = entropy(parent, base=2)               # 0.940286
h_children = entropy(children, base=2, axis=1)   # [0.970951 0. 0.970951]
gain = h_parent - w @ h_children                 # 0.246750
split_info = entropy(sizes, base=2)              # 1.577406
print(gain, gain / split_info)                   # 0.246750 0.156428

def gini(c):
    p = c / c.sum()
    return 1 - np.sum(p ** 2)

gini_gain = gini(parent) - sum(wk * gini(c) for wk, c in zip(w, children))
print(gini_gain)                                 # 0.116327

To have a library grow the whole tree with this criterion, use scikit-learn:

from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier(criterion="entropy")   # default is criterion="gini"
# clf.fit(X, y) then picks, at every node, the binary split with the largest impurity decrease

The function is documented on the scipy.stats.entropy reference page, and the criteria, with their exact formulas, in the scikit-learn decision trees guide.

Frequently asked questions

What is a good information gain value?

There is no fixed threshold. Information gain lies between 0 and the entropy of the parent node, so it is only meaningful relative to the other candidate splits at the same node. A gain of 0 means the split leaves the class mix unchanged, and a gain equal to the parent entropy means every child is pure.

Can information gain be negative?

No. The size-weighted entropy of the children can never exceed the entropy of the parent, so information gain is always 0 or more. It is exactly 0 when every child has the same class proportions as the parent.

What is the difference between information gain and gain ratio?

Gain ratio is information gain divided by split information, the entropy of the child sizes. The division penalises splits that create many branches, which plain information gain tends to favour. Gain ratio is undefined when all samples fall into a single child, because the split information is then 0.

Do entropy and Gini impurity give the same tree?

Usually very similar ones. Both measures are 0 for a pure node and largest for an even class mix, so they rank most candidate splits in the same order. Gini impurity is slightly cheaper to compute because it needs no logarithm, and it is the default criterion in scikit-learn.

Related calculators

See every metric tool in the model evaluation calculators hub, or continue with one of these:

• Cross-entropy loss for predicted probabilities
• Accuracy, precision and recall from a confusion matrix
• F1 score from precision and recall
• Conditional probability and Bayes’ rule