Case Study: The Hidden Ethics of Mental Health AI—Lessons from Three Startups

The 2 AM Test

Sarah, 16, opens her phone at 2 AM. She’s been struggling with anxiety for months, but therapy feels intimidating and expensive. Instead, she turns to an AI chatbot that promises 24/7 emotional support. The conversation feels surprisingly natural—empathetic, even. For the first time in weeks, she feels heard.

But here’s the catch: Sarah thinks she’s talking to something that understands therapy. The AI thinks it’s providing coaching. Her parents have no idea this is happening. And nobody—not Sarah, not her parents, not even the company that built the tool—is entirely sure what happens when AI meets mental health crisis at 2 AM.

This scenario isn’t hypothetical. It’s playing out thousands of times daily as AI-powered mental health tools proliferate among young people desperate for support. And it’s exactly the kind of situation that prompted one forward-thinking impact investor to make a crucial decision: hire Compass Ethics to systematically examine the ethical landscape three of their portfolio companies were working through. Rather than wait for scandals or regulatory crackdowns, this youth mental health oriented investor decided to get ahead of the curve. They brought in Compass Ethics with a clear but challenging mission: identify the ethical strengths and ethical risks across three AI-powered mental health platforms.

We weren’t looking just to find problems. We were looking to understand the ethical terrain these companies operate in, which inevitably will include both remarkable opportunities and the genuine risks. The three companies participated voluntarily, opening their operations to detailed ethical analysis. This collaborative approach proved essential to uncovering insights that a traditional audit would have missed.

Compass then engaged in the following process: initial discovery interviews that informed a series of ethical exercises—with a phase for critical thinking and analysis between. What we discovered through these engaging cases reveals the complex moral landscape of AI-powered mental health support. Note that the investor and portfolio companies have been anonymized to protect both our clients and their users.

Here’s what that process looked like in detail.

(1) The Discovery Phase

Compass designed the engagement as a two-phase process: discovery followed by collaborative exercises. The discovery phase involved what we call “ethical landscaping”: examining everything from the way the AI actually communicates with a user, to parental oversight mechanisms, to the company’s stated use cases. All the while, our consultants uncovered the values tensions and moral assumptions embedded in seemingly neutral technical systems.

• KEY DEFINITION: A values-tension is a place where two good things we might want come into conflict with each other. For example, an app developer might want to maximize user freedom, but doing so would risk causing more harm—that’s a values tension.

The real insights came from the exercises phase. Rather than delivering a top-down audit, our team worked directly with each company’s team to map their ethical landscape together. This included scenario planning sessions, stakeholder impact assessments, and “ethical stress testing”—examining how systems might behave in situations like Sarah’s 2 AM crisis, or edge cases where the AI might not work as expected.

The portfolio companies were incredibly engaged. They wanted to understand these issues as much as we did. That collaborative approach was essential to getting beneath the surface.

Once we had collected enough information about each company, the Compass team came together internally to unpack what we’d learned. What made this investigation particularly revealing was its scope across multiple companies. While each platform had unique features, we started to notice patterns—recurring ethical tensions that seemed intrinsic to AI-powered mental health technology rather than specific to any one company’s choices.

There were, broadly, two classes of values tension: implementation problems that could be fixed with better policies, and deeper structural tensions built into the technology itself. Once we had identified these patterns and come up with particular examples, we met back up with the client teams.

(2) The Exercises

The next phase of the consultation involved working through a set of ethical exercises with each small company. Compass has a set of proprietary exercises designed to highlight the most important moral questions that might arise for a given organization. The teams we worked with engaged well, diving deep into several case studies and structured activities to directly face the implications of their products, both good and bad.

Through systematic analysis across all three companies, five critical themes emerged—what Compass calls the critical values tensions that anyone building, investing in, or using AI mental health tools ought to have concrete answers to.

The Five Critical Tensions:

1. The Promise of Mental Health Technology emerged as both the most optimistic and complex theme. AI tools genuinely can expand access to mental health support in unprecedented ways, but this promise comes with responsibility, ensuring accessibility doesn’t compromise ethical standards. Precisely because mental health technology has the promise to help people, it must be executed well so that it can realize its potential. With great power comes great responsibility.
2. The Therapeutic Gray Zone proved most immediately concerning. Regardless of how platforms positioned themselves, users often engaged with them as primary emotional care, creating what researchers termed “implicit duty of care” that companies weren’t always prepared to handle. The fact of the matter is that AI will never be a therapist, and knowing where coaching and encouragement end and actual therapy begins is critical.
3. AI Neutrality revealed itself as perhaps the most subtle issue. Despite appearing neutral, AI systems consistently reflect biases and cultural assumptions from their training data. “Neutrality is never actually neutral,” the Compass team noted. “It’s always somebody’s version of normal.” This means that companies designing AI mental health tools should strive to incorporate diverse perspectives at every stage of the design process.
4. The Data Stewardship Imperative focused on the extraordinary sensitivity of mental health information, especially from young people. Compass found significant variation in how companies approached these responsibilities. However, all had a clear objective: ensure that they use state of the art data security protocols to protect vulnerable users’ information.
5. Parental Oversight and Transparency emerged as the area where companies struggled most with competing values—balancing youth autonomy with parental involvement and safety requirements. This will vary significantly depending on the company’s user demographic, but is always something that a company developing mental health tech should have a clear set of guidelines for.

These tensions were critical to understand. But in addition to these categories, we also came upon what we called the “harm reduction insight.” Our research revealed that young people are already turning to general-purpose AI tools like ChatGPT for mental health support—often without any safeguards. This completely reframed our analysis. The question isn’t whether AI should be involved in youth mental health care. It already is. The question is how to make that involvement as safe and beneficial as possible.

This, then, led to Compass’ core recommendation: purpose-built mental health AI tools, when designed ethically, provide essential safety infrastructure for a phenomenon that’s already happening. Rather than replacing human care, the best tools create pathways to it. And they do this much better than an all-purpose AI like ChatGPT can.

Beyond Identification to Broader Impact

The Compass team’s engagement didn’t end with identifying problems. Working with each company, we developed “ethical implementation strategies”: practical approaches for addressing the tensions while maintaining innovation potential. These included everything from redesigning user interfaces to developing sophisticated escalation protocols connecting users with human support when needed.

Ultimately, the aim of any innovation is and should be to help people. By including the right safeguards and policies, and making technological design choices, companies can innovate well, rather than simply for the sake of making something novel alone.

Throughout the course of this contract, we were able to help three companies in concrete, morally significant ways. But what makes this case study valuable is its systematic approach to examining an entire ecosystem rather than individual companies in isolation. By analyzing patterns across multiple .organizations, Compass identified structural ethical challenges that anyone entering this space will encounter.

Our methodology offers a model for proactive ethical analysis in other sensitive AI domains. Most importantly, this work shows that the future of AI in mental health care isn’t predetermined. The choices being made now will shape how an entire generation relates to mental health support.

As Sarah and millions like her turn to AI for support, the question isn’t whether we need ethical oversight—it’s whether we’ll implement it thoughtfully enough to preserve both innovation and human welfare.