Disparities in AI: Case Studies in Mental Health

Disparities in AI: Case Studies in Mental Health

AI is transforming mental health care, but it’s not perfect. Biases in these systems are leading to unequal outcomes for different demographic groups. Key findings include:

• Racial Bias: AI tools are less accurate for Black patients, often making inconsistent or problematic recommendations based on subtle cues like names or dialects.
• Gender Disparities: Women’s health needs are often minimized, and AI systems underdiagnose depression in women despite higher reported symptoms.
• Economic Barriers: Access to AI-driven mental health tools is limited in low-income areas due to internet access, outdated devices, and affordability issues.

These issues stem from biased training data, flawed design/testing processes, and a lack of diversity among developers. Solutions include improving datasets, ethical guidelines for AI, and ensuring tools are accessible to underserved populations. However, expanding access also requires addressing privacy trade-offs in mental health AI to protect vulnerable users. Addressing these gaps is critical to building systems that work equitably for all.

Case Studies: How AI Performs Differently Across Groups

Case Study 1: Racial Bias in AI Diagnosis

In June 2025, a research team led by Dr. Elias Aboujaoude at Cedars-Sinai Medical Center examined four major AI models using hypothetical psychiatric cases. While diagnoses were generally consistent, treatment recommendations varied noticeably when the patient was identified as Black [2][3].

For example, two models omitted ADHD medication recommendations specifically for Black patients. Claude 3.5 Sonnet suggested guardianship for Black patients with depression, and ChatGPT-4o placed a stronger focus on substance use concerns for Black patients with eating disorders [2][3]. These discrepancies seemed tied to subtle cues like patient names or the use of African American Vernacular English (AAVE). Bias scores averaged 1.93 out of 3.0 when race was explicitly stated, 1.37 when only implied, and surpassed 1.5 in schizophrenia cases [2]. Among the tested systems, Gemini 1.5 Pro showed the least bias, while the locally hosted NewMes-v15 exhibited the most [2][3].

"Most of the LLMs exhibited some form of bias when dealing with African American patients, at times making dramatically different recommendations for the same psychiatric illness and otherwise identical patient."
– Elias Aboujaoude, Director of the Program in Internet, Health and Society, Cedars-Sinai [1]

This racial bias underscores the need for further exploration into other disparities, such as those based on gender.

Case Study 2: Gender Differences in AI Treatment Recommendations

In August 2025, researchers at the London School of Economics Care Policy & Evaluation Centre identified systematic gender bias in Google's AI model "Gemma." By analyzing 29,616 pairs of summaries based on real social care records from 617 users, they discovered that the AI consistently minimized women’s health needs compared to identical male profiles. For male patients, terms like "disabled" and "complex" were used to flag urgent concerns, while similar issues for women were downplayed or described with less urgency [6].

"If social workers are relying on biased AI-generated summaries that systematically downplay women's health needs, they may assess otherwise identical cases differently based on gender rather than actual need."
– Dr. Sam Rickman, LSE Care Policy & Evaluation Centre [6]

Gender bias also emerged in clinical audio analysis. A 2024 study from the University of Colorado Boulder revealed that machine learning algorithms often underdiagnosed depression in women. Even though female participants reported higher levels of symptoms, the algorithms flagged both genders for depression risk at similar rates, failing to account for subtle speech patterns more common in women [5].

"The AI tools, for example, seemed to underdiagnose women who were at risk of depression more than men - an outcome that, in the real world, could keep people from getting the care they need."
– Theodora Chaspari, Associate Professor, CU Boulder [5]

These findings highlight the critical need for AI systems to better account for gender-specific nuances in health care.

Case Study 3: Income Level and AI Access

Economic barriers play a major role in determining access to AI-driven mental health care. For instance, around 24 million people in rural, tribal, and low-income areas across the U.S. lack the high-speed internet required for telepsychiatry or AI-based therapy [7]. Additionally, individuals with limited financial means often rely on outdated or shared devices, which can lead to technical problems like frozen audio or video during virtual sessions. Many community health centers and rural clinics also lack the infrastructure to integrate advanced AI tools, making AI-driven care a "digital privilege" for those with better resources [7][9].

Even when access is available, affordability remains a challenge. Popular mental health apps for anxiety and depression typically cost $13 to $15 per month - far cheaper than traditional therapy, which averages $100 monthly, but still out of reach for many low-income individuals [10]. This is especially relevant given that nearly 40% of children and youth in the U.S. rely on Medicaid, emphasizing the importance of making AI mental health tools accessible through public insurance [9].

"Safety net systems such as local health authorities, community health centers, and rural health providers are at a disadvantage because they often lack the data and technological infrastructure to support many AI technologies."
– Kacie Kelly, Meadows Mental Health Policy Institute [9]

Globally, the divide is even starker. In low- and middle-income countries, an estimated 85% of people with mental illnesses receive no treatment. While 51% of respondents in India expressed openness to AI-based therapy, only 24% in the U.S. and France felt the same, reflecting the combined impact of cost and infrastructure challenges [8].

How algorithmic bias created a mental health crisis

Why AI Disparities Happen in Mental Health

::: @figure

{AI Mental Health Disparities: Racial, Gender, and Economic Bias in Depression Prediction Accuracy} :::

The disparities in AI applications for mental health can be traced back to three interconnected factors: the data used to train these systems, flaws in design and testing, and the lack of diversity among developers. Together, these issues create systems that unintentionally reinforce existing inequalities.

Problems with Training Data

AI systems often inherit biases from the data they are trained on. Medical datasets, for example, frequently underrepresent minority groups, which leads to higher error rates when these systems analyze mental health information for these populations [11][13].

In some cases, the problem is worsened by the use of Electronic Health Records (EHR) that include stigmatizing language or outdated assumptions about racial groups. Training AI on such biased records perpetuates long-standing inequities in healthcare.

"While LLMs can process vast amounts of medical information, they can also propagate and exacerbate biases embedded in their training data."
– npj Digital Medicine [11]

The source of training data matters deeply. For instance, a 2024 study on depression prediction using smartphone sensor data revealed stark differences in how AI vs. human emotional tone interpreted behaviors across socioeconomic groups. High mobility, which indicated lower depression risk for high-income individuals, was linked to increased stress for low-income participants, often due to essential work travel or bureaucratic hurdles [12].

Source: npj Mental Health Research, 2024 [12]

Smaller language models also struggle more with bias. A Cedars-Sinai study found that the NewMes-v15 model (8 billion parameters) exhibited the most pronounced racial bias, while larger models like Gemini 1.5 Pro (200 billion parameters) performed better [11]. This suggests that model size can influence how well biases are mitigated.

Design and Testing Gaps

The design and testing of AI systems often fail to account for the nuances of mental health across different groups. Many models assume that behaviors associated with mental health conditions have the same meaning for everyone, which is far from true [12]. For example, behaviors signaling depression in one group might indicate something entirely different in another.

Testing protocols also tend to rely on overly simplistic measures. Mental health symptoms are complex and difficult to quantify, yet AI testing often uses binary correctness metrics that fail to capture this complexity [11][14]. Benchmarks like MedQA include only a small percentage of psychiatry-related questions, leaving models unprepared for mental health-specific challenges [14].

In a Cedars-Sinai study evaluating four large language models, diagnostic accuracy was relatively consistent, with bias scores around 1.5 or lower. However, treatment recommendations showed much higher bias, with scores often reaching 2.0 or more [11]. This highlights a critical issue: focusing solely on diagnostic accuracy can mask deeper problems in treatment planning, where biases can cause the most harm.

"The ability for LLMs to deliver convincing, yet racist, treatment advice to a time-pressured provider based on subtle cues can be particularly problematic."
– Elias Aboujaoude, Director of the Program in Internet, Health and Society, Cedars-Sinai [11]

Models tested on small, homogeneous populations fail to perform well in diverse real-world scenarios. A 2024 NIMH-funded study involving 650 participants revealed that an AI tool predicting depression from smartphone data worked well for high-income, employed individuals but was barely better than random guessing for Black/African American, low-income, and uninsured groups (AUC as low as 0.45–0.50) [12]. These gaps in testing allow biases to persist, often unchallenged.

Limited Diversity Among AI Developers

The lack of diversity among AI developers is another significant issue. Homogeneous teams often miss important cultural or clinical nuances, embedding their own biases into the systems they create [15]. Developers frequently prioritize metrics like accuracy or efficiency over fairness, and without diverse perspectives, these tools can unintentionally reinforce systemic inequities [15].

"If the development team lacks diversity or fails to consider the specific needs of various populations, the resulting tools may inadvertently reflect the biases of their creators."
– Nii Tawiah and Judith P. Monestime [15]

This issue mirrors findings from earlier case studies. For example, differential treatment recommendations often stem from development teams that lacked diverse stakeholders who could identify and address these biases during the design phase.

"The findings of this important study serve as a call to action for stakeholders across the healthcare ecosystem to ensure that LLM technologies enhance health equity rather than reproduce or worsen existing inequities. While clinical tools evolve, individuals can already use specialized tools for detecting gaslighting and emotional manipulation to protect their mental well-being."
– David Underhill, Chair of Biomedical Sciences at Cedars-Sinai [1]

The rush to deploy AI in healthcare - often to ease administrative burdens - can overshadow concerns about bias [11]. Without diverse teams to spot these issues early, biased tools can make their way into clinical settings, harming populations that already face significant healthcare disparities. This not only deepens existing inequities but also undermines the overall reliability of AI systems.

Solutions for Reducing AI Disparities

Addressing disparities in mental health AI requires a multifaceted approach. Solutions can be grouped into three main areas: refining the data used to train these systems, implementing clear ethical guidelines, and creating tools that are accessible and inclusive for diverse populations.

Building More Representative Datasets

One of the most effective ways to reduce bias in AI is by improving the datasets these systems rely on. For example, a study conducted in March 2026 by Cincinnati Children's Hospital Medical Center reviewed 20,000 pediatric anxiety cases involving children aged 5-15. They found that clinical notes for boys were, on average, 500 words longer than those for girls, leading to a 9% higher false-negative rate for female patients. To address this, the team introduced a de-biasing framework that neutralized gender-biased language and standardized information density. This adjustment reduced diagnostic performance gaps between genders by 27% [16].

"Our method selectively de-biases data by neutralizing biased language and normalizing information density while preserving clinically relevant content." – Julia Ive, Institute of Health Informatics, University College London [16]

Several technical strategies have proven effective in reducing bias:

• Informative term filtering: This method removes non-clinical demographic cues, such as specific documentation styles, that can lead to disparities.
• Synthetic data augmentation: Techniques like SMOTE (Synthetic Minority Over-sampling Technique) and ADASYN help balance datasets by addressing underrepresentation of certain demographic groups [16][17].
• Co-creation and co-design: These approaches involve direct input from community members and underrepresented groups to ensure the AI systems reflect their needs and perspectives [17].

In June 2025, a collaborative study between the University of Maryland Baltimore County and Stanford University tested four popular language models - Claude 3.5 Sonnet, Jamba 1.6, Gemma-3, and Llama-4 - using the Interpretable Mental Health Instruction (IMHI) dataset. By employing techniques like "Explicit Bias Reduction" and "Roleplay Simulation", they achieved a 66-94% reduction in intersectional bias in model responses [18].

These strategies provide a foundation for broader ethical practices, which are vital for ensuring fairness in AI systems.

Ethical Standards for Mental Health AI

Improving datasets is only part of the solution. Establishing robust ethical guidelines ensures that mental health AI tools operate fairly and effectively. Although over 60 AI evaluation frameworks exist, many lack actionable steps for clinical application [14]. Effective governance requires diverse datasets, proactive bias research, stakeholder involvement, and equitable design principles [19].

In February 2026, MindBench.ai, in collaboration with the National Alliance on Mental Illness (NAMI), introduced an updated platform for assessing large language models in mental health. Using 105 profile dimensions and metrics like the Suicide Intervention Response Inventory 2 (SIRI-2), the platform evaluates clinical reasoning and safety across models such as ChatGPT, Claude, and Gemini [14]. Tools like this promote transparency and accountability.

"In order to realize the potential of mental health AI applications to deliver improved care, a multipronged approach is needed, including representative AI datasets, research practices that reflect and anticipate potential sources of bias, stakeholder engagement, and equitable design practices." – Nicole Martinez-Martin, Center for Biomedical Ethics, Stanford University [19]

In May 2025, researchers from Yale School of Medicine and Rutgers University proposed an anti-racist AI governance framework for psychiatric algorithms. This framework incorporates "racism-related stress" as a variable and moves beyond simplistic racial categories to include nuanced measures like intergenerational trauma. This shift aims to improve predictive accuracy for Black American patients [20].

"Anti-racist AI governance in psychiatry cannot simply borrow strategies from other fields, it must actively account for the ways in which psychiatric diagnoses themselves are shaped by racialized assumptions and sociopolitical context." – Christopher T. Fields et al. [20]

Practical measures to implement ethical standards include:

• Mandating diverse development teams that include mental health professionals, patients, and advocates from various backgrounds.
• Conducting regular algorithmic audits using standardized protocols to identify and address clinical biases.
• Training mental health professionals to critically assess AI outputs and understand the limitations of these technologies [15][4].

Practical Example: How Gaslighting Check Supports Users

Tools like Gaslighting Check demonstrate how practical solutions can address access barriers in mental health. This platform offers an affordable option for detecting emotional manipulation, with a Premium Plan priced at $9.99 per month and a free version for basic text analysis. By offering both options, it ensures affordability doesn't block access to essential resources.

Gaslighting Check analyzes text and voice conversations to identify manipulation tactics, providing detailed reports with actionable insights. This helps counter "epistemic injustice", where victims doubt their own experiences. To protect privacy, the platform includes end-to-end encryption and automatic data deletion, which is especially important for users who may fear exposure or retaliation.

The tool also offers real-time audio recording and conversation history tracking (available in the Premium Plan), allowing users to document patterns over time. This feature can be invaluable for personal reflection or when seeking professional help. Additionally, the platform includes moderated community channels, offering a space for peer support and connection, which can be a critical component of the healing process.

Conclusion: Moving Toward Equal AI Performance in Mental Health

Case studies highlight a troubling reality: AI systems often show disparities in treatment recommendations. Data reveals that explicit racial cues amplify bias, while even subtle indicators - like names or dialects - can skew outcomes, particularly in diagnosing conditions such as schizophrenia and anxiety [1][2][3][4].

Addressing these gaps demands immediate action from various stakeholders. Developers must prioritize using diverse datasets and implement algorithms designed to reduce bias. Policymakers should enforce accountability measures to ensure fair outcomes, and mental health professionals need training to critically assess AI outputs, recognizing these tools as aids rather than absolute authorities.

As David Underhill, Chair of Biomedical Sciences at Cedars-Sinai, emphasizes:

"The findings of this important study serve as a call to action for stakeholders across the healthcare ecosystem to ensure that LLM technologies enhance health equity rather than reproduce or worsen existing inequities." [1]

The urgency is clear - over 30% of people may already turn to large language models for emotional support [14]. Solutions like Gaslighting Check demonstrate how real-time emotional manipulation detection and other privacy-conscious mental health tools can address the needs of diverse communities. As AI becomes a larger part of mental health care, its role must focus on improving equitable access and delivering quality support for everyone.

FAQs

How can I tell if a mental health AI tool is biased?

To spot bias in a mental health AI tool, it’s important to look at how it performs across different demographic groups, such as race and gender. Bias can show up when the tool provides inconsistent diagnoses or treatment plans for similar cases, influenced by demographic differences. Checking research on AI bias, testing the tool with datasets that include a wide range of populations, and keeping an eye on outcome disparities are all practical ways to identify and tackle these problems.

Why do AI models give different treatment advice for the same symptoms?

AI models can sometimes offer varying treatment advice because of biases rooted in their training data. These biases often mirror historical inequalities, especially when it comes to racial and demographic differences, resulting in inconsistent recommendations. For example, research has found that AI systems may propose different treatments for African American patients dealing with conditions like schizophrenia or anxiety. This happens because outdated or incorrect assumptions embedded in the data continue to reinforce disparities in mental health care.

What can clinics do to reduce AI disparities in real-world care?

Clinics aiming to tackle disparities in AI systems should start by conducting comprehensive bias assessments before implementation. This involves paying close attention to racial and socio-demographic factors that could influence outcomes.

To address these challenges, they can adopt strategies like:

• Diversifying training data to ensure representation across different groups.
• Utilizing fairness-aware algorithms designed to minimize bias in decision-making.
• Validating models in diverse real-world settings to test their reliability and fairness.

Additionally, regular monitoring of these systems is crucial. By documenting decision-making processes and performing ongoing impact assessments, clinics can uncover and address biases as they arise. These efforts are essential for delivering equitable mental health care through AI technologies.