Week 6 — Social & Ethical Considerations

Background

This week you studied the foundations of how AI systems reason and learn. You saw how agents perceive their environment and act on it, how search reduces complex problems to path-finding, how supervised learning encodes human judgments into predictions, and how decision trees and neural networks represent two very different approaches to the same basic task: classifying inputs correctly.

Every one of those technical choices has a social and ethical shadow. Training a classifier on labeled data means someone labeled the data — and their judgments are now embedded in every prediction the model makes. A decision tree that is transparent about its logic may still be opaque about where its logic came from. A neural network that is highly accurate may be impossible to audit or explain. Building a system that finds patterns automatically means trusting that the patterns it finds are real, stable, and fair to act on.

The scenarios below are grounded in the Week 6 material: reasoning systems, decision trees, supervised learning, and the interpretability tradeoff between decision trees and neural networks. Week 7 will take up different questions — reinforcement learning, unsupervised systems, and large language models — but the issues here are not preliminary. They are foundational, and they run through every AI application that follows.

How to Use These Scenarios

Use the same approach established in earlier weeks: read each scenario carefully, write down your initial thoughts using the five guiding questions, and come prepared to discuss in your small group. A full description of the process is available on the SEC scenario discussion guide.

Scenarios

Scenario 1 — The Transparent System That Was Not Transparent Enough

A county juvenile justice system adopts a decision tree-based risk assessment tool to assist judges in making pretrial detention decisions for juveniles. The tool was marketed specifically on the basis of its transparency: unlike "black box" AI models, the decision tree could be printed on a single page and shown to any judge, attorney, or defendant. The vendor described this as a significant ethical advantage over opaque systems.

Two years after deployment, a public defender files a challenge on behalf of several clients. She argues that while the tree itself is visible, the training data it was built from is not. The historical case data used to train the tool reflects decades of racially disparate policing and charging decisions. The tree has learned to weight certain neighborhood zip codes and prior contact with police heavily — attributes that are themselves products of the historical inequities in the data. The decisions the tree makes are legible. The bias embedded in those decisions is not.

• The decision tree is transparent in the sense that its logic can be read and followed. What does this scenario reveal about the limits of that kind of transparency?
• The vendor's claim that the system was more ethical than black-box alternatives was probably made in good faith. Is it accurate?
• What would "meaningful transparency" look like for a system like this — transparency that is actually useful to a defendant challenging a decision?
• This tool assists judges rather than replacing them. The judge still makes the final decision. How does that affect your thinking about responsibility?

Scenario 2 — The Pattern That Was Real but Wrong to Use

This scenario is based on real research. The findings described here were published by Obermeyer et al. in Science in 2019. You can search for "Obermeyer 2019 Science healthcare algorithm racial bias" to find the original paper and related coverage.

A health insurance company trains a machine learning model to predict which patients are likely to have high future medical costs, with the goal of identifying patients who would benefit most from care coordination programs. The model is trained on historical claims data and performs well in validation: it correctly identifies high-cost patients with considerably higher accuracy than previous methods.

A health equity researcher reviewing the model notices a problem. The model uses prior healthcare utilization as one of its strongest predictive features. This is a sensible choice from a purely statistical standpoint — patients who have used a lot of healthcare in the past tend to use a lot in the future. But prior healthcare utilization is systematically lower for Black patients than for white patients with equivalent health needs, because of historical barriers to healthcare access. The model has learned a real pattern. The pattern, however, reflects access disparities rather than health needs. As a result, the model systematically underestimates the care needs of Black patients and overestimates those of white patients. The system directs resources away from the patients who need them most.

• The model learned a statistically valid pattern from real data. In what sense was something still wrong?
• What is the difference between a pattern that reflects reality and a pattern that is appropriate to act on? Are those always the same?
• Supervised learning requires choosing which attributes to include in the training data. What responsibility does the model builder have for the attributes they choose — and those they choose not to include?
• This type of feedback loop — where a model trained on historically unequal data produces decisions that perpetuate that inequality — has been documented in multiple domains. What structural changes would be needed to interrupt it?

Scenario 3 — The Hiring Algorithm That Learned the Wrong Lesson

A large technology company builds a neural network to screen job applicants. The system is trained on ten years of historical hiring data: resumes of people who were hired, and resumes of people who were not. The model learns to identify patterns that predict hiring decisions. It performs well in validation — its recommendations agree with historical hiring decisions at high rates.

An internal audit eventually reveals a problem. The model has learned to downweight resumes that include the word "women's" — as in "women's chess club" or "women's leadership program." It has also learned to prefer certain university names and certain extracurricular patterns that correlate strongly with gender and socioeconomic background in the training data. The system is not discriminating intentionally. It is doing exactly what it was trained to do: reproduce the patterns in historical hiring decisions. Those historical decisions reflected the biases of the humans who made them.

The audit also reveals something that cannot be fixed by retraining: even a version of the model with the most obviously biased signals removed cannot explain, for any individual resume, why it was ranked the way it was. A rejected candidate cannot be told what specifically disqualified them, because the disqualification emerged from the interaction of hundreds of weighted features across dozens of hidden layers. The decision was made. The reasoning cannot be produced.

• The original engineers did not train the system to discriminate. Does that absolve them of responsibility for the outcome?
• This system reproduces historical hiring decisions faithfully. If those decisions were biased, what does "accuracy" actually mean for a model like this?
• A decision tree-based hiring system could at least print the rules it used. A neural network cannot. Given what you learned about the interpretability tradeoff this week, how should organizations weigh accuracy against accountability in high-stakes classification tasks?
• Legal frameworks in many countries require employers to be able to justify hiring decisions. How does this interact with the fundamental opacity of deep learning systems?
• As a teacher, you make consequential evaluations of students every day. What parallels — or important differences — do you see between human judgment in evaluation and algorithmic judgment in hiring?