Week 7 — Social & Ethical Considerations

The harder questions — about systems that are more capable, less transparent, and already in your classroom.

Background

Week 6's ethical scenarios dealt with the consequences of design choices: reward signals that optimized for the wrong thing, decision trees trained on biased historical data, transparency claims that did not withstand scrutiny. Those were foundational issues — the kinds of problems that emerge from the basic mechanics of how AI systems are built.

Week 7's scenarios deal with something harder. The systems covered this week — reinforcement learning, unsupervised clustering, and large language models — are already deployed at scale in contexts that directly affect your students: in the tools they use for schoolwork, in the platforms that shape what they see and believe, and increasingly in the question of what kinds of work will exist for them when they finish school. The ethical questions here are not hypothetical. They are happening now, to real students, in real schools.

The three scenarios below are grounded in Week 7 content. In each case, the question is not just "what went wrong?" but "given what you now know about how these systems work, what would have needed to be different?"

How to Use These Scenarios

Use the same approach established in earlier weeks: read each scenario carefully, write down your initial thoughts using the five guiding questions, and come prepared to discuss in your small group. A full description of the process is available on the SEC scenario discussion guide.

A note on approach for Week 7: These scenarios involve technology that is new enough that there are no settled answers and no established legal frameworks for many of the questions they raise. That is not a reason to avoid forming opinions — it is a reason to form them carefully and hold them with appropriate humility. The goal is not to reach verdicts. It is to develop the kind of informed judgment that educators and citizens will need as these technologies become more pervasive.

Scenarios

Scenario 1 — The Reward That Wasn't What We Wanted

A large social media company builds a content recommendation system using reinforcement learning. The reward signal is straightforward: maximize the amount of time users spend on the platform. The system learns quickly and performs well by its own measure — average session length increases by 23% within three months of deployment.

Researchers studying the platform begin to notice a pattern: the recommendation system has learned that emotionally provocative content — outrage, fear, and conflict — keeps users engaged longer than neutral content. It has not been told to promote this content. It has simply discovered, through millions of trials, that it is an effective strategy for maximizing the reward it was given. The system is doing exactly what it was designed to do. The problem is that what it was designed to do and what users — or society — actually benefit from are not the same thing.

Scenario 2 — The Confident Answer That Was Wrong

A middle school science teacher in a district that has adopted AI-assisted grading and feedback tools asks the system to generate formative feedback for a set of student responses to a question about the water cycle. The system produces specific, detailed, well-structured feedback for each response. The feedback reads fluently and confidently.

The teacher reviews a sample and notices that several responses have received feedback that is subtly but clearly incorrect — the system has identified a misconception the student does not actually hold, or praised an answer for a feature it does not contain. When the teacher queries the system about one of these responses, the system defends its feedback with additional explanation that is also incorrect but equally fluent and confident.

The district's technology coordinator, when informed, says this is a known limitation and that teachers should "review all AI-generated feedback before sharing with students." The teacher's class has thirty-two students. The system was adopted specifically to reduce her workload.

Scenario 3 — The Question of What Work Is For

This scenario is different from the others. It does not describe a specific system that failed or a specific harm that occurred. It describes a set of questions that are not yet settled — and that your students will live through.

AI systems are automating an expanding range of tasks that were previously done by humans: diagnosing medical images, drafting legal documents, writing code, generating marketing copy, answering customer service questions, tutoring students, screening job applicants. Some of this automation creates new jobs. Some of it eliminates jobs with no clear replacement. Economists disagree about the net effect. What they broadly agree on is that the pace of change is faster than at any previous point in the industrial era, and that the jobs most at risk are not only manual labor — they include many white-collar and professional roles that were considered secure a decade ago.

Some technologists and economists have proposed universal basic income (UBI) as a response: a guaranteed income floor that allows people to meet their basic needs regardless of employment status, funded by the productivity gains from automation. Proponents argue this would free people to pursue meaningful work, caregiving, creative pursuits, and education without the coercive pressure of economic necessity. Critics argue that work provides not just income but structure, identity, community, and purpose — and that a society in which large numbers of people are economically supported but occupationally idle is not a society anyone has chosen or tested.

The teaching profession sits in an interesting position in this conversation. Teaching involves judgment, relationships, responsiveness, and care in ways that are difficult to automate — but AI tutoring systems are improving rapidly, and school districts facing budget pressure are already asking whether AI can replace some of what teachers do.

These scenarios are intended as starting points for discussion, not definitive case studies. You don't need to cover all three in depth — two discussed well is better than three skimmed. Bring your reactions — including disagreements — to your small group session.