AI Won't Fix Our Social Problems

Deconstructing Risk in Predictive Risk Models

A public talk and reception to launch The Glass Room at London Public Library. Featuring Shion Guha, Assistant Professor, University of Toronto, and author of Human-Centered Data Science: An Introduction.

Co-hosted by the Starling Centre for Just Technologies and Just Societies, and the London Public Library. 

Join us from 5-8 PM at Central Library, 251 Dundas Street, London, ON.

The lecture begins at 6 PM.

Free parking is available for two hours at Central Library with validation.

Abstract: Predictive risk models (PRM) now steer frontline decisions across child welfare, criminal justice, homelessness services, and immigration. Yet the object of prediction is often a moving target. What counts as “risk” is not a stable property of people or families. It is a construct produced by agency priorities, historical assessment practices, constrained resources, and the surveillance footprints those forces leave in administrative data. Drawing on multi-method empirical studies in child welfare, I show that many PRMs learn patterns of institutional response rather than the outcomes they claim to predict. Proxy labels tied to prior investigations, hotline calls, or service actions routinely stand in for harm. These choices create self-fulfilling feedback that inflates accuracy, hides distributional error, and shifts burden onto already over-observed communities.

I will trace how labeling decisions, data provenance, and workflow integration generate measurement bias, outcome leakage, survivorship bias, and harm displacement. I then outline a human-centered data science framework that I have helped develop, which couples statistical modeling with interpretive inquiry, participatory auditing with practitioners and affected communities, and design interventions that re-specify targets away from “risk of agency action” toward concrete safety and support objectives. The framework yields practical tests for label validity, governance checkpoints for deployment, and redesigns of triage that prioritize service eligibility and need over speculative risk scoring. Although the case material centers on child welfare, the analysis generalizes to adjacent domains where administrative action masquerades as ground truth. The talk closes with research and policy directions for scholars and collaborators across information, computing, law, and public policy who are ready to rethink what it means to predict “risk” in public services.


Speaker Bio: Shion Guha is an Assistant Professor in the Faculty of Information at the University of Toronto, cross-appointed to the Department of Computer Science, and directs the Human-Centred Data Science Lab. He leads a research program that examines how data, measurement, and organizational practice interact when governments adopt algorithmic systems. His team works with agencies in child welfare, healthcare, and homelessness services to study model development and use, to audit outcomes with mixed methods, and to co-produce evaluation criteria that reflect service goals and community priorities. The program emphasizes practical artifacts that institutions can deploy immediately, including assessment protocols, procurement checklists, and implementation guides that link analytics to clear accountability paths. Guha is the author of Human-Centered Data Science: An Introduction (MIT Press, 2022). His research has been funded by the Canadian Institute for Advanced Research, the Natural Sciences and Engineering Research Council of Canada, the Canadian Institutes of Health Research, the Social Sciences and Humanities Research Council, and the U.S. National Science Foundation. He has been recognized with a Way-Klingler Early Career Award (2019), a Connaught New Researcher Award (2021), and a Schwartz-Reisman Institute for Technology and Society Faculty Fellowship (2023). He has partnered with Canadian municipal, provincial, and federal bodies on AI initiatives and provided expert input to U.S. state agencies and civil society organizations. His work has been covered by Newsweek, the Associated Press, ABC, NBC, and the Toronto Star. He holds an MS from the Indian Statistical Institute and a PhD from Cornell University.