Reading 3: Data Mining in the Real World

Where Data Mining Is Already Happening

Data mining has moved from research laboratories into everyday life with remarkable speed. It operates in the background of systems your students interact with constantly, making decisions and recommendations that affect their experiences without most of them — or most adults — being aware of it.

In Business and Commerce

The recommendation systems on Netflix, Spotify, Amazon, and YouTube are driven by association analysis and cluster analysis running continuously on enormous datasets of user behavior. When Netflix suggests a show you end up loving, that is not a human editor's intuition — it is a pattern discovered by mining the viewing histories of millions of subscribers who behaved similarly to you.

Retailers use sequential pattern analysis to predict what customers will need next based on what they have recently purchased. A well-known example: a major retailer's data mining system identified that customers who buy certain combinations of products — unscented lotion, large bags of cotton balls, supplements — are frequently pregnant, and began sending targeted pregnancy-related advertising to those customers before they had publicly announced their pregnancies. The system was accurate. It was also deeply unsettling to the people who received the advertising, and it sparked a significant public debate about what companies should be allowed to infer and act on from purchase data.

In Medicine and Public Health

Data mining has become one of the most powerful tools in medical research. Outlier analysis applied to electronic health records has identified previously unknown drug interactions that individual physicians would never have detected in their own patient panels. Cluster analysis of genomic data has revealed that conditions once thought to be single diseases are actually families of related but distinct conditions that respond differently to treatment.

During the COVID-19 pandemic, data mining was used to track the spread of variants, identify which patient characteristics predicted severe outcomes, and monitor social media for signals of emerging outbreaks — all examples of the technique operating at a scale and speed that would have been impossible a generation ago.

In Education

Educational data mining is a growing field with significant implications for K-12 schools. Districts and states are increasingly using data mining techniques on longitudinal student records to identify early warning indicators of dropout risk, predict which students are likely to struggle in specific courses, and evaluate the effectiveness of interventions across large populations.

Adaptive learning platforms — software that adjusts the difficulty and type of practice a student receives based on their responses — use real-time data mining to personalize instruction. Every time a student answers a question, the system updates its model of that student and adjusts what comes next.

As a teacher, you are likely already affected by educational data mining even if you have not been told so explicitly. Early warning systems, risk scores, and intervention recommendations in your school's data systems are frequently powered by data mining techniques applied to historical student records.

The Ethical Landscape

Data mining's power to find patterns in large datasets creates serious ethical questions that do not have easy answers. These questions are the subject of the Social and Ethical Considerations page for this week, so we will not explore them exhaustively here — but it is important to name them before closing this topic.

Privacy

Data mining can reveal deeply personal information from data that seems innocuous on its own. The pregnancy example from the previous section illustrates this. Purchase records, browsing histories, location data, and social media behavior can be combined and mined to infer health conditions, political beliefs, financial stress, relationship status, and much more. People who shared that underlying data did not necessarily consent to those inferences being made.

In education, student data is protected by federal law (FERPA), but the boundaries of what is permissible are actively contested. What can a third-party vendor do with behavioral data collected through an adaptive learning platform? Who owns the patterns discovered by mining student records? These are live legal and ethical questions without settled answers.

Bias and Fairness

Data mining discovers patterns in historical data. If that historical data reflects past inequities — and most real-world data does — the patterns the algorithm discovers will reflect those inequities too. A model trained to predict dropout risk based on historical student records may encode the biases of past resource allocation decisions rather than identifying genuine risk factors. A student flagged as high-risk by such a system may then receive fewer resources or lower expectations, creating a self-fulfilling prophecy.

The pattern was real. The data supported it. But acting on it without understanding its origins can perpetuate exactly the inequities the system was meant to address.

The Limits of Correlation

We raised the correlation vs. causation issue in Topic 5a, and it is worth revisiting here. Data mining finds correlations. It does not establish causes. A system that predicts which students are at risk of not graduating does not explain why those students are at risk, and acting on the prediction without understanding the cause can lead to misguided interventions. A student flagged because they share characteristics with previous non-graduates needs support and opportunity — not a label that limits their options.

Closing the Week: Everything Connects

Step back and consider where we started this week and where we have arrived.

In Topic 5a, we established that data becomes useful only when driven by questions, that the investigation cycle is the framework for moving from question to answer, and that different types of data call for different tools. In Topics 5B and 5C, we explored the most important tool for organized, structured data at scale: the relational database, with its carefully designed relations, schemas, and query language. And now in Topic 5d, we have explored data mining — what happens when the dataset is too large, too complex, or too uncharted for any individual to know what questions to ask.

Data mining sits at a fascinating and somewhat unsettling edge of the investigation cycle. In most of the cycle, humans are in control: they pose the question, design the collection, make cleaning decisions, and interpret the findings. In data mining, the algorithm is discovering things that humans did not anticipate — which is its power, and also its risk. The human judgment required to evaluate what the algorithm found, determine whether it is meaningful, and decide whether and how to act on it is not replaced by the technology. It is made more consequential by it.

Iowa's Data and Analysis standards, from kindergarten through high school, are building exactly the foundation students will need to navigate this landscape. Students who learn to pose precise data questions in kindergarten, evaluate data quality in third grade, sort and filter structured data in middle school, and create data visualizations in high school are developing the conceptual framework that makes data mining legible rather than magical. The techniques are complex; the underlying logic is accessible to anyone who understands the investigation cycle.

That is what this week has been about.