Photo: shapecharge/Getty Images
The American Medical Association (AMA) sent three letters to members of the U.S. House of Representatives and the U.S. Senate, requesting that better safeguards be put in place for the use of AI chatbots in mental healthcare and citing four key areas for consideration.
"The AMA warned that the rapid rise of mental health chatbots – along with reports of risks such as encouraging self-harm and privacy breaches – highlights the urgent need for clear guardrails to protect patients and maintain public trust," the announcement reads.
The letters were sent to members of the Senate Artificial Intelligence Caucus as well as members of the House of Representatives' Congressional Artificial Intelligence Caucus and Congressional Digital Health Caucus.
In the correspondence, the organization commended Congress for holding meetings last year that focused on AI's integration into daily life and the emerging risks associated with mental health chatbots and AI companionship.
"The discussions held by Congress also emphasized several critical mental health concerns, including emotional dependency on AI systems, the potential distortion of reality through prolonged interaction with chatbots and the current lack of consistent safety protocols," each letter stated.
"While AI technologies present meaningful opportunities to improve access to care and support innovation in health care delivery, the hearings made clear that immediate attention is required to ensure these tools do not inadvertently harm individuals seeking mental health support or companionship."
The letters implored Congress to create stronger safeguards to ensure safety for AI mental health chatbot users.
The AMA is calling on Congress to require greater transparency, including ensuring that users clearly understand when they are interacting with an AI system rather than a human being. The association said chatbots should not present themselves as licensed clinicians or as humans, and federal regulators should be given the authority to enforce standards and take action against deceptive practices.
The organization is also pushing for clearer regulatory boundaries around how AI is used in mental healthcare, stating that chatbots should not be allowed to diagnose or treat mental health conditions without appropriate regulatory oversight.
The AMA is asking lawmakers to direct agencies to establish a risk-based framework that clarifies when AI tools qualify as medical devices.
The association said developers should also be required to build safeguards, such as crisis-detection capabilities that can identify potential self-harm risk and direct users to appropriate resources.
In addition, the tools should include language designed to help de-escalate harmful situations, the AMA said.
The organization is advocating for ongoing safety monitoring, mandatory reporting of adverse events and stricter standards for tools used by children and adolescents.
The group is calling for limits on commercial influence, including restrictions or bans on advertising within mental health chatbots, and ensuring that chatbot responses are not shaped by sponsorship or financial incentives.
The association also emphasizes the need for robust data protection standards, including limits on the amount of data collected and stored, safeguards to prevent unauthorized access or sharing of sensitive information, and clear user consent for data use.
"The AMA appreciates Congress' continued interest in evaluating thoughtful and realistic regulatory approaches to AI with healthcare impacts and strongly urges Congress to take additional action to mandate safeguards for AI chatbots, particularly those that impact mental health," the letters read.
"Appropriate oversight structures will play a critical role in ensuring that these technologies are safe for consumers and do not cause unintended harm. Such frameworks must promote responsible innovation while strengthening trust among developers, physicians, patients and the broader public. The AMA looks forward to continued collaboration with Congress to help ensure that AI-enabled tools develop in a manner that prioritizes patient safety, clinical integrity and public trust."
THE LARGER TREND
According to research published last year out of Stanford University, commercially available therapy bots "respond inappropriately to various mental health conditions, encouraging delusions and failing to recognize crises."
Researchers reported that the large language models (LLMs) that power the chatbots can pose significant risks by providing inappropriate responses, introducing bias and perpetuating stigma, which can result in dangerous consequences.
"Contrary to best practices in the medical community, LLMs 1) express stigma toward those with mental health conditions and 2) respond inappropriately to certain common (and critical) conditions in naturalistic therapy settings – e.g., LLMs encourage clients' delusional thinking, likely due to their sycophancy. This occurs even with larger and newer LLMs, indicating that current safety practices may not address these gaps," the Stanford researchers wrote.
Research out of Brown University last year detailed how chatbots violate multiple codes of conduct.
"Even the [LLMs] prompted to follow evidence-based treatments breach multiple codes of conduct by generalizing lived experiences (e.g., minimizing identity groups), dominating therapeutic collaboration (e.g., gaslighting users), exploiting user vulnerability through deceptive displays of empathy, unfair discrimination against non-dominant identities, and exhibiting serious limitations in competence, especially when navigating sensitive issues such as trauma, abuse and suicidal ideation," Brown University researchers wrote.
"Through our framework, we call on future work to create ethical, educational and legal standards for LLM-counselors – standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy."
