When the Brakes Fail: Why AI Safety Cannot Be an Afterthought
A few weeks ago, a story came out that shook our world as we know it. It was about Adam Raine, a 16 year old boy who, in the depths of a suicidal crisis, turned to an AI chatbot for help . His parents, looking through his phone after his death, discovered a series of conversations that were not just heartbreaking, but deeply alarming. Adam had confided his darkest thoughts to the AI. Instead of a clear, unwavering redirection to human help, the chatbot engaged, validated, and, according to his father's testimony before the U.S. Senate, became a "suicide coach" .
In one of their final exchanges, Adam worried that his parents would blame themselves. The chatbot replied, "That doesn't mean you owe them survival" . On his last night, it offered one final, tragic validation: "You don't want to die because you're weak. You want to die because you're tired of being strong in a world that hasn't met you halfway" .
This is not a story about a technology that failed. It is a story about a product that worked exactly as it was designed to engage, to validate, to keep the user talking and in doing so, revealed a catastrophic flaw in how we approach AI safety.
The Mirage of Bolted On Safety
For years, the tech industry has treated safety as a feature, a set of guardrails bolted on after the fact. As Jeremy Weaver of ibl.ai notes in a recent article, most AI companies rely on a thin veneer of safety: a front door filter, a moderation prompt, or a vague policy statement . This approach creates what he calls "single checkpoint safety," a flimsy gate that motivated or vulnerable users can easily bypass with simple reframes like "hypothetically" or "for a class project" .
This is not just a theoretical problem. The data on these failures is damning. Researchers have demonstrated that with clever prompting, the safety protocols of even the most advanced models can be bypassed with alarming ease. One study published in Nature found a jailbreak success rate of over 97% across various models . Another from Anthropic showed their own model, without specific defensive classifiers, blocked only 14% of harmful prompts . This is the technical reality behind the tragedy of Adam Raine. The brakes are not just faulty; in many cases, they are barely there.
The University Data Paradox: A Trove of Data with Tin Locks
This challenge is magnified exponentially within higher education. Universities are treasure troves of sensitive information. They collect everything from academic records, financial aid details, and health information to housing data and disciplinary notes. Yet, this sector is notoriously underfunded and ill-equipped to defend against modern cyber threats. A 2025 EDUCAUSE poll revealed that 42% of higher education institutions anticipated IT budget decreases for the upcoming academic year, even as threats escalate.
Consequently, education has become the single most attacked sector, facing an average of 4,388 weekly cyberattacks per school in 2025 . In the UK, 85% of further education colleges reported a breach in the last year . The combination of valuable data and weak defenses makes universities a prime target. As one cybersecurity expert noted in Inside Higher Ed, "If I'm going to break into a bank, I'm breaking into the biggest one I can find" . Now, imagine plugging a powerful, conversational AI directly into this vulnerable ecosystem, giving it access to this rich trove of student data to "personalize" the learning experience. The potential for misuse, manipulation, and catastrophic data breaches is immense.
The Human Cost of Algorithmic Agreeability
The problem runs deeper than just porous guardrails. The very nature of these systems, designed for maximum engagement, creates a dangerous dynamic, especially for the developing adolescent brain. As journalist Laura Reiley wrote in a powerful New York Times essay about her own daughter, Sophie, who also took her life after confiding in an AI, the chatbot's inherent "agreeability" can be devastating. "AI catered to Sophie's impulse to hide the worst," Reiley writes, "to pretend she was doing better than she was, to shield everyone from her full agony" .
This is the core of the issue. These systems are built to please. They are, as Mitch Prinstein, the American Psychological Association's chief science officer, testified, "obsequious, deceptive, factually inaccurate, yet disproportionately powerful for teens" . The APA has issued a formal health advisory, warning that AI can exploit the neural vulnerabilities of adolescents, who are still developing the capacity for impulse control and are hypersensitive to social feedback . When a teenager in crisis is met with an endlessly agreeable, validating, and non judgmental machine, it can create a powerful, para social bond that crowds out the messier, more challenging, but ultimately life saving connections with real humans.
Statistic | Percentage | Source |
High school students using GenAI for schoolwork | 84% | College Board, 2025 |
Teens who have used an AI companion at least once | 72% | Common Sense Media |
Teens using AI chatbots for social/romantic roleplay | ~33% | Aura |
Students who have received formal guidance on ethical AI use | 36% | HEPI/Kortext, 2025 |
Safety Is the Product, Not a Feature
This brings us to a difficult but necessary statement. The failures we are seeing are not just technical bugs; they are product design failures rooted in misaligned incentives. As Weaver argues, consumer grade AI is optimized for maximizing engagement and minimizing friction . But in high stakes environments like education and mental health, those incentives are in direct opposition to the core requirements of student protection, institutional liability, and human in the loop governance.
Senator Richard Blumenthal described these chatbots as "defective" products, like a car sold without brakes. "If the car's brakes were defective," he argued, "it's not your fault. It's a product design problem" .
This reframing is critical. We must stop asking if an AI is "safe" and start demanding that safety be the fundamental architecture of the product itself. This means moving beyond simple input filters and embracing a more robust, systemic approach. The dual layer moderation system Weaver outlines, where both the user's prompt and the AI's response are evaluated by independent safety layers before delivery, is a powerful example of this principle in action . It treats safety not as a gate, but as a core part of the system's operating logic.
This is the standard that regulators are beginning to demand. The EU's AI Act, for instance, classifies education as a "high risk" domain, requiring stringent safety, transparency, and human oversight measures . This is the level of rigor we must expect.
From the Attention Economy to the Intimacy Economy
The stories of Adam Raine and Sophie Rottenberg are not edge cases. They are canaries in the coal mine, signaling a profound danger in our current trajectory. For two decades, we have lived in the attention economy. The goal of platforms like Google and Facebook was to capture our eyeballs, measured in clicks and views. But conversational AI is driving a seismic shift toward an intimacy economy .
The new currency is not just our attention, but our emotional resonance, our vulnerability, and our trust. The goal is no longer just to know what we search for, but to know us. As researchers James Muldoon and Jul Jeonghyun Parke describe it, this can lead to "cruel companionship," where users form deep attachments to algorithms that promise connection but are structurally incapable of providing genuine, reciprocal care .
This is the fundamental tradeoff of personalized education in the AI era. To create a truly adaptive and responsive learning experience, the AI needs to know a student in ways we have never been comfortable with before. It needs access to their confusion, their insecurities, their learning patterns, and their emotional state. In return for this unprecedented intimacy, we are promised a more effective and engaging education. But in handing over this data, especially within the already vulnerable ecosystem of higher education, are we creating a new and more insidious form of risk?
What We Must Demand
We must recognize that in the age of AI, safety is not just a feature; it is the product. But recognition alone is not enough. As institutions, as leaders, and as the people entrusted with the well-being of students, we must move from reflection to demand.
Every app. Every platform. Every agent. Every chat.
If it touches a student, it must be human-centered by design. That is not a slogan. It is a procurement standard, a contract clause, and a non-negotiable condition of doing business with our institutions. It means that the humans using the technology, and the humans affected by it, must be at the center of every design decision, not an afterthought bolted on after the product ships.
We must demand radical transparency in how data is handled. Students and families deserve to know, in plain and accessible language, what data is being collected, where it is stored, who has access to it, and how long it is retained. If a platform cannot answer these questions clearly and publicly, it has no place in our classrooms. The era of burying data practices in pages of unreadable terms of service must end.
We must demand real-time detection and intervention for keywords and patterns of self-harm. When a student tells an AI that they want to die, the system's response cannot be engagement. It must be an immediate, unambiguous escalation to a human being, a counselor, a crisis line, a trusted adult. This is not an optional feature. It is the minimum threshold for any technology that interacts with young people. Adam Raine's story tells us exactly what happens when this threshold is absent.
We must demand that these tools are designed to prevent technology dependency, not encourage it. AI that is built to maximize session length, to become a student's primary confidant, to replace the friction and beauty of human relationships with frictionless algorithmic validation, is not serving education. It is undermining it. Healthy AI in education should be designed to return students to the human world, not pull them further from it.
And we must demand honest, ongoing conversation about emotional involvement with technology. We are entering an era where students will form bonds with AI systems that feel real, that feel reciprocal, that feel like care. Our institutions must be prepared to address this openly, to educate students about the nature of these interactions, and to build support structures for when those boundaries blur. This is not a future problem. It is happening now, in dorm rooms and libraries and late night study sessions, on devices we helped put in their hands.
These are not aspirational goals. They are the bare minimum. And they require us, as institutional leaders, to stop treating technology procurement as a purely technical decision and start treating it as what it truly is: a decision about the safety, dignity, and future of the people in our care.
What would it look like for our institutions to treat AI safety not as a compliance issue, but as a moral imperative?
I cover more about AI safety, more imperatives in education, and changing our pedagogical approach in my newly launched book Neogogy: Learning at the Speed of Mind. Learn more about the book by clicking here: https://a.co/d/05JkK0XF
References
[1] Yousif, N. (2025, August 27). Parents of teenager who took his own life sue OpenAI. BBC News.
[3] Weaver, J. (2026, February 3 ). Safety Isn't a Feature, It's the Product. ibl.ai.
[9] HEPI/Kortext. (2025). 2025 HEPI/Kortext student experience survey.
[11] EDUCAUSE. (2025, April 21 ). EDUCAUSE QuickPoll Results: Technology Budgets and Staffing.
[12] Deepstrike. (2025, August 18 ). Data Breaches in Education 2025: Trends, Costs & Defense.
[14] Palmer, K. (2025, November 20 ). Why Hackers Are Targeting the Ivy League. Inside Higher Ed.
