3,000 wages, 1,250 days, 64 cells.

Inputs and output. The columns we predict from are the inputs, also called predictors or features. The column we predict is the output, also called the response. Later chapters write the number of rows as n and the number of inputs as p.

Wages are in thousands of dollars a year: 75.0 means $75,000.

A scatter plot shows how two variables move together. Two things show at once: young men tend to earn less, and at any one age the dots spread a long way up and down.

The line estimates the average wage at each age. It climbs steeply until about 40, stays roughly flat until about 60, then falls.

Try the slider. A narrow window follows every few men, so the line jitters. A wide one smooths away the real bend. How much to smooth is a question this book keeps coming back to.

Only 29 of the 3,000 men are over 70, so the right end of the line rests on very little.

The line predicts an average, not a man. The spread around it is everything age can't explain: education, job, region, luck, and all that isn't in the table. Age alone can't predict one man's wage with much accuracy.

Notice the band at the top. 79 men earn over $250k and sit apart from everyone else. Real data has quirks like this, and methods have to cope with them.

Real, but small. The average rose about $10,000 over six years, while wages within any single year differ by tens of thousands. A trend can be clear in the averages and nearly invisible in the dots.

The typical distance from the average, $41.7k, is the standard deviation: roughly, how far a wage usually sits from the mean. The dashed line is the straight line that fits all 3,000 dots best; Chapter 3 shows how it's found.

Why not just (116.0 − 106.2) ÷ 6? That gives $1.63k a year, but it only uses the first and last years. The fitted line uses every wage in all seven years, and the middle years pull it flatter: 2005 dipped below 2004. Both numbers are honest; they answer slightly different questions.

The box always holds the middle half. Whatever the data, about a quarter of it sits below the box, about a quarter above, and about half inside. It’s “about” because a real data set rarely splits into exactly equal whole numbers: 650 men make piles of 162 or 163. That makes box plots easy to compare side by side, as the next scene does.

Outliers aren’t errors. They’re just wages far from the middle by this rule. Of the 18 high ones here, 7 belong to the separate band of top earners above $250k from scene 4.

Unlike age, this climb never turns down. The median rises at every level, and the biggest jumps come at the top.

Associated, not caused. The data shows that education and wage go together. It can't show that more education raises wages: people who stay in school may differ in other ways too.

Each prediction here is the average of similar men: the same age, or the same age band (18–25, 26–35, … 66–80) and education level. Age alone barely helps. Adding education helps more. Year adds little.

Most of the spread remains: wage depends on far more than this table records. Predicting a number like wage is a regression problem. Chapter 3 combines inputs with linear regression, and Chapter 7 handles the bend in age.

Why the average? Of all single guesses, the average is the one with the smallest RMSE. Try it in the playground: wherever you drag the line, the RMSE only gets worse once you leave the average.

These misses are measured on the same 3,000 men the averages were built from, which flatters the predictions a little. Chapter 2 explains why that matters.

The lags are the inputs. Each row carries the last five days with it, so we can ask whether the recent past says anything about today.

Orange bars are days the index rose; blue bars, days it fell. Calm stretches and stormy stretches come in runs: 2001 and 2002 swing much harder than 2004.

Regression predicts a number: a wage, a price, a temperature. Classification predicts a label: Up or Down, spam or not, which of several diseases. A category can only be right or wrong, so the methods and the ways of scoring them differ.

The two boxes almost coincide, however far back you look. Their medians never differ by more than 0.15 percentage points, tiny next to a typical day's move of 1.14%. The past few days say almost nothing about today.

Early on the scores swing wildly. After a few hundred days they settle near 50%. A rule needs many days before its score means anything.

Why no easy pattern? If a simple rule worked well, traders would use it, and their buying and selling would erase the pattern. The opposite rule's 53.9% hints at a weak tendency to reverse, but weak is the word.

Train on the past, test on the future. Scoring a model on the days it learned from would flatter it. Scoring it on days it never saw is the honest test.

Always compare with the simplest strategy. The market rose on 56% of 2005's days, so “always Up” already scores 56%. Next to that, 59.9% is a real but modest gain.

Every prediction sits close to a coin flip. The model leans the right way a little more often than not, which adds up to 60% right, yet it is never confident about any single day.

The dots in the middle of each box mark the group's average. The two clouds overlap almost entirely.

This is unsupervised learning: there are inputs but no output to guide (supervise) the learning. Nothing can be predicted right or wrong, so instead we look for structure: which rows resemble each other.

Colours: orange means a gene is more active than usual for that gene, blue less. Some rows share a pattern even in this sliver of the table.

This is principal components analysis (Chapter 12). With all 6,830 genes it does the same thing: Z1 is the direction of most spread, and Z2 the direction of most spread at right angles to Z1. Each cell line becomes two numbers: where it sits along each.

Squashing loses information. These two numbers keep 18% of the variation among the 64 cell lines. In exchange, we get a picture.

We chose four groups because that's what the map suggests; k-means has to be told how many. Choosing that number well is a hard problem in general (Chapter 12).

The groups come from Z1 and Z2 alone. Nothing about cancer types went in.

The map partly recovers the cancer types without ever seeing them: 44% of nearest neighbours match, against 9% by chance. That's evidence the structure is real, though not proof that four groups is the right number.

Squashing cost something. With all 6,830 genes, 35 of 64 nearest neighbours match. Two numbers keep most of that, but not all.

Supervised learning has an output to predict: regression when it's a number, classification when it's a category. Unsupervised learning has no output; clustering is one way to find structure, and Chapter 12 shows others.