Reading 1: What Data Mining Is (and Isn't)

A Different Kind of Question

Think back to the SQL queries we worked with in Topic 5c. Every one of them started with a human being who had a specific question: Which students are enrolled in American Literature? What professional development has Carmen Rivera completed? The database answered those questions efficiently and precisely — but only because a person had already formulated the question clearly enough to express it as a query.

Now consider a different situation. A retail chain has ten years of purchase records for 40 million customers — billions of individual transactions. No single person can look at that data and formulate useful questions about it, because no single person can hold enough of it in mind to even know what questions are worth asking. The patterns that matter are buried too deep, across too many records, for human intuition to find them.

Or consider a hospital system with a decade of patient records, lab results, medications, and outcomes for hundreds of thousands of patients. A physician treating an individual patient brings clinical expertise and asks specific questions about that patient. But the question "are there patterns in this entire decade of records that predict which patients will be readmitted within 30 days of discharge?" is not a question any individual physician could answer by examining records one at a time. The pattern, if it exists, only becomes visible when the entire dataset is analyzed computationally.

This is the territory of data mining: the use of computational techniques to discover patterns in large data collections that would be invisible to human examination alone.

How Data Mining Differs from What Came Before

Data mining occupies a distinct position in the landscape of data analysis. It is worth being precise about how it differs from two other things it is often confused with: traditional database querying and basic statistics.

Data Mining vs. Database Querying

A SQL query retrieves stored facts in response to a question. The question comes first; the query operationalizes it. Data mining, by contrast, generates questions — or rather, it generates findings that raise questions. You do not go into a data mining process knowing what you are looking for. You go in looking for whatever is there to be found.

Put another way: database querying is like looking up a word in a dictionary. You know the word; you want its definition. Data mining is like reading the entire dictionary and noticing that an unexpected number of words beginning with the letter Q relate to silence or stillness — a pattern you never would have found by looking up any particular word.

Data Mining vs. Basic Statistics

Calculating averages, medians, and percentages is valuable — and it is something your students will do throughout the Data and Analysis standards from elementary school onward. But these calculations describe data you already have organized and understand. The average attendance rate across a school district is a useful number, but it does not reveal hidden structure. It does not tell you that the attendance patterns of students in one zip code cluster into three distinct groups with very different characteristics, or that students who are chronically absent in October are statistically likely to have transferred out by March.

Data mining goes beyond description to discovery. It uses algorithms — computational procedures — to find structure, groupings, associations, and anomalies that are not visible in summary statistics alone.

Why Data Mining Uses Data Warehouses

Data mining is typically not performed on the live operational databases that organizations use day-to-day. Instead, it is performed on what are called data warehouses — large, static collections of data assembled specifically for analysis.

There are two main reasons for this. First, finding patterns in data requires the data to hold still. A live school database is constantly changing — students enroll and transfer, grades are entered and revised, attendance is updated daily. Running a pattern-discovery algorithm on data that is changing underneath it is technically complicated and can produce unreliable results. A data warehouse is a snapshot: it captures the state of the data at a point in time, allowing analysis to proceed on a stable foundation.

Second, data mining often draws from multiple source systems that were never designed to work together. A district might want to mine patterns across attendance records, assessment results, demographic data, and special services records — data that lives in four different operational systems. A data warehouse brings that data together into a single unified collection, cleaned and organized for analysis.

Data warehouses are the reason large-scale educational research is possible. When researchers study the relationship between early literacy indicators and long-term academic outcomes across hundreds of thousands of students, they are working with data warehouses, not live school databases.

The Hard Part: Patterns vs. Coincidences

Here is something that surprises many people when they first encounter data mining: the algorithms are very good at finding patterns. Almost too good. Given a large enough dataset, a data mining algorithm will almost always find something — some grouping, some association, some sequence of events that appears more often than chance would predict.

The hard part is not finding patterns. The hard part is determining whether a found pattern is meaningful or merely a coincidence produced by the size of the dataset.

A famous example: a data mining analysis of lottery ticket sales found that a particular convenience store in a mid-sized city had sold a disproportionately high number of winning tickets over a five-year period. Is the store lucky? Does it attract lucky customers? Should you buy your tickets there? Almost certainly not — with thousands of lottery retailers across a state over many years, statistical fluctuations guarantee that some stores will have unusually good streaks purely by chance. The pattern is real; the significance is illusory.

Contrast that with the discovery, through health record data mining, that a particular combination of medications significantly increased the risk of a specific cardiac event — a pattern that would have been invisible in any individual physician's practice but became clear across millions of patient records. That pattern is both real and meaningful, and acting on it saves lives.

Distinguishing meaningful patterns from coincidences requires domain expertise, statistical judgment, and intellectual honesty. It is the human judgment layer that data mining algorithms cannot replace — and it is one reason why data mining is most powerful when the people running it deeply understand the domain they are investigating.

What Comes Next

Data mining encompasses a family of distinct techniques, each suited to a different kind of pattern discovery. In Reading 2, we will explore six of the most important techniques, with concrete examples from education, business, and medicine that illustrate what each one is designed to find.

In Reading 3, we will look at where data mining is actually being used today — including in K-12 education in ways that directly affect your students — and we will take an honest look at the ethical questions that large-scale pattern discovery raises.