AMA urges Congress to strengthen safeguards for AI chatbots

The American Medical Association (AMA) today urged federal lawmakers to put stronger safeguards in place as the use of augmented intelligence (AI) chatbots in mental health care continues to grow. The AMA emphasized both the promise and risks of these rapidly expanding technologies.

In letters to the co-chairs of the Congressional Artificial Intelligence Caucus (PDF), the Congressional Digital Health Caucus (PDF), and the Senate Artificial Intelligence Caucus (PDF), the AMA recognized Congress for advancing conversations about AI’s role in society and mental health. At the same time, the AMA warned that the rapid rise of mental health chatbots—along with reports of risks such as encouraging self-harm and privacy breaches—highlights the urgent need for clear guardrails to protect patients and maintain public trust.

“AI-enabled tools may help expand access to mental health resources and support innovation in health care delivery, but they lack consistent safeguards against serious risks, including emotional dependency, misinformation, and inadequate crisis response,” according to AMA CEO John Whyte, MD, MPH. “With thoughtful oversight and accountability, policymakers can support innovation and ensure technologies prioritize patient safety, strengthen public trust, and responsibly complement—not replace—clinical care.”

While AI technologies present meaningful opportunities to improve access to care and support innovation in health care delivery, recent congressional hearings made clear that immediate attention is required to ensure these tools do not inadvertently harm individuals seeking mental health support.

To address these concerns, the AMA urged Congress to require stronger safeguards—recognizing they are a starting point, not a limit, and that modernized protections may be needed as these tools evolve. The safeguards outlined in the AMA letter include:

Enhance Transparency: Require chatbots to clearly and meaningfully disclose that users are interacting with AI, not a human being, and prohibit systems from presenting themselves as licensed clinicians. Congress should also empower federal regulators to enforce transparency standards and penalize deceptive practices.

Establish Clear Regulatory Boundaries: Prohibit chatbots from diagnosing or treating mental health conditions without appropriate regulatory review. Congress should direct agencies to create a modern, risk-based oversight framework and clarify when AI tools qualify as medical devices. Developers should also be required to implement crisis-detection systems that identify self-harm risk and provide immediate referrals to appropriate resources, along with appropriate de-escalation language to help mitigate potential harms.

Strengthen Oversight and Accountability: Mandate ongoing safety monitoring, reporting of adverse events, and rigorous standards for tools used by children and adolescents, who may face heightened risks.

Limit Commercial Influence: Discourage or prohibit advertising within mental health chatbots, particularly for minors, and ensure that outputs remain free from sponsorship bias or commercial influence.

Protect Privacy and Security: Require strict data protection standards, including limits on data collection and retention, clear user consent for data use, and safeguards against unauthorized access or sharing of sensitive information.

The AMA emphasized that meaningful safeguards are essential to ensure AI tools complement—not replace—clinical care while safeguarding patients from harm. The organization reaffirmed its commitment to working with Congress to advance policies that promote innovation while prioritizing patient safety, clinical integrity, and public trust.

The American Medical Association is the physicians’ powerful ally in patient care. As the only medical association that convenes 190+ state and specialty medical societies and other critical stakeholders, the AMA represents physicians with a unified voice to all key players in health care. The AMA leverages its strength by removing the obstacles that interfere with patient care, leading the charge to prevent chronic disease and confront public health crises and, driving the future of medicine to tackle the biggest challenges in health care.

The American Medical Association is the physicians’ powerful ally in patient care. As the only medical association that convenes 190+ state and specialty medical societies and other critical stakeholders, the AMA represents physicians with a unified voice to all key players in health care. The AMA leverages its strength by removing the obstacles that interfere with patient care, leading the charge to prevent chronic disease and confront public health crises and, driving the future of medicine to tackle the biggest challenges in health care.