Reading 2: Building a Tree from Data

Two Ways to Get a Decision Tree

The reading support tree in Reading 1 was constructed by hand — someone with domain knowledge decided which questions to ask and in what order. That is a valid approach for simple, well-understood problems where experts can articulate the decision logic clearly.

But consider a harder problem: a bank wants to decide which loan applications to approve. It has six attributes for each applicant — credit score, annual income, existing debt, employment status, requested loan amount, and loan purpose — and a history of thousands of past loans, each labeled "repaid" or "defaulted." No single loan officer could look at all of that data and distill it into a clean set of rules. The patterns are too complex, too numerous, and too intertwined.

This is where machine learning enters. Instead of having a human write the tree, a learning algorithm examines the labeled examples and builds the tree automatically — discovering which attributes matter most, which questions to ask first, and how deep the tree needs to go. The result is a decision tree that encodes patterns from the data rather than from expert intuition.

Training Data: The Raw Material

To build a decision tree, the learning algorithm needs training data: a collection of past examples where both the attributes and the correct outcome are known. Each example is called a training instance.

For the loan problem, a training instance might look like this:

The outcome column — "Repaid" or "Defaulted" — is called the label. It is what the algorithm is trying to learn to predict. The other columns are the attributes (also called features) — the information available at the time a decision needs to be made.

The learning algorithm reads all of these labeled examples and uses them to construct a tree that, when given a new application with no known outcome, makes the best possible prediction.

What Makes a Good Split?

The central challenge in building a decision tree is deciding which question to ask at each node. With six attributes available for the loan problem, there are six candidates for the root node. How do we choose?

The guiding principle is this: a good split separates the training examples into groups that are as pure as possible. A group is pure if it contains mostly one outcome — mostly "Repaid," or mostly "Defaulted." A mixed group, with roughly equal numbers of each outcome, is not useful for making a decision.

Think about it concretely. Suppose we split first on loan purpose:

• Car loans: 55% repaid, 45% defaulted
• Home loans: 58% repaid, 42% defaulted
• Personal loans: 51% repaid, 49% defaulted

These splits are nearly useless. Knowing whether the loan is for a car or a house does not tell us much about whether it will be repaid. Both groups are heavily mixed.

Now suppose we split first on credit score (above 650 vs. at or below 650):

• Credit score above 650: 84% repaid, 16% defaulted
• Credit score at or below 650: 31% repaid, 69% defaulted

This is a much better split. The two groups are now quite different from each other and much more homogeneous internally. An applicant with a high credit score is probably going to repay; an applicant with a low credit score is more likely to default. The split is doing useful work.

The learning algorithm evaluates every possible split at every node and chooses the one that produces the purest groups. This process repeats recursively at each branch — for the "credit score above 650" group, which attribute best separates the remaining uncertainty? Perhaps income. For the "credit score below 650" group, perhaps employment status. The tree grows downward, one carefully chosen split at a time, until each leaf node is pure enough to make a confident prediction.

A Simplified Loan Decision Tree

Here is what a decision tree for the loan problem might look like after being learned from training data. This is deliberately simplified — a real bank's model would be larger — but it illustrates the structure that emerges when good splits are chosen.

Notice how the tree's logic mirrors what a careful loan officer would do: start with the strongest predictor (credit score), then refine based on income or stability depending on which side of the credit split the applicant falls on. The tree did not have this logic written into it — it discovered the logic from patterns in thousands of past loans.

Tracing an Applicant

Let us trace an applicant: credit score 690, annual income $42,000, stable employment, medium existing debt.

1. Root: Credit score above 650? Yes (690 > 650). Go left.
2. Internal node: Annual income above $50,000? No ($42,000). Go right.
3. Internal node: Existing debt low? No (medium). Go right.
4. Leaf node: Review manually.

The tree cannot make a confident call here: the applicant has a decent credit score but lower income and medium debt. Rather than forcing an approve/deny decision, this tree sends borderline cases to a human reviewer — a sensible design for high-stakes decisions.

The Overfitting Trap

Here is a tempting idea: if a tree that gets most training examples right is good, then a tree that gets all training examples right should be better. Build the tree deep enough, add enough nodes, and eventually you can classify every single training instance correctly.

Unfortunately, that tree is almost certainly worse, not better, for new applications. This problem is called overfitting.

Think about what happens when a tree grows very deep. It stops learning general patterns and starts memorizing specific examples. It might create a branch just for the three applicants named "Johnson" who happened to default, or a special leaf for loans taken out on a Tuesday in March. Those patterns exist in the training data by coincidence. They are not real signals — they are noise. But a sufficiently deep tree will learn them anyway.

When that tree sees a new applicant, the branches built on coincidental patterns fire in unpredictable ways. The tree has memorized its training data rather than learning from it.

Overfitting is one of the most important concepts in all of machine learning. A model that is too perfectly tuned to the data it was trained on will often perform poorly on new data it has never seen. The goal is not to memorize the past — it is to learn patterns that generalize to the future.

The standard remedy for overfitting in decision trees is pruning: deliberately stopping the tree from growing too deep, or going back after construction and removing branches that do not meaningfully improve accuracy on held-out data. A shallower, simpler tree that captures the main patterns will almost always outperform a deep, complicated tree that has memorized the noise.

There is a lovely parallel here to teaching. A student who has memorized every worked example in the textbook can reproduce those examples perfectly. But put a novel problem in front of them — one that requires applying the underlying principle to a new context — and the memorizer often fails while the student who understood the concept succeeds. Overfitting is the machine learning version of teaching to the test.

What Comes Next

We now know what decision trees are, how to trace them, and how they are built. Reading 3 looks at where they appear in real systems, what their limits are, and how they connect to the broader category of supervised learning — the foundation for everything we will cover in Topic 6e on artificial neural networks.