Innovation with Integrity: Examining the Risks and Benefits of AI Chatbots
No chapters available yet
Legend
Transcript markers
- text Sentence being played
- text Search result
- text Intervention cited by the link you opened
Click a sentence to play the video from that moment. Select text to quote it with the speaker, timestamp and link.
More options
House Energy and Commerce Subcommittee on Oversight and Investigations U.S. House of Representatives
Follow its sittingsSpeaker 2
A new term has been coined, AI psychosis, which describes when a user's interactions with a chatbot lead to distorted beliefs or even delusions. As we've seen in some absolutely tragic cases, users experiencing mental health crises have even taken their own lives after extensive communication with these chatbots. My heart goes out to the families who are coping with these terrible losses, and we owe it to them to keep examining what went wrong and how this might be prevented in the future. We need answers, and we need far more data on the safety of these apps. As a member of last year's House bipartisan task force on AI, I welcome efforts to make sure AI is safe, secure, and trustworthy.
We need to fully fund and support our federal agencies with oversight and enforcement authority, and we must refuse to simply take companies at their word that they are protecting their users. So far, this administration has prioritized protecting the interests of the president's billionaire tech buddies, and I fear much needed progress in this area will be delayed at least at the executive level. But I hope that at least here in Congress, we can work together, and today's bipartisan hearing is a step in the right direction. I'm hopeful that we can chart a path forward that protects users of AI without compromising innovation. There's far too much at stake to let big tech, excuse me, big tech fly down this path at full speed with no guardrails.
I look forward to hearing the perspectives of our highly credentialed expert panel, and how we can move forward with integrity and safety. With that, Mr. Chairman, I yield back.
Speaker 1
Thank you. The chair now recognizes the chairman of the full committee, Mr. Guthrie, for five minutes for an opening statement.
Speaker 3
Thank you, Chairman Joyce. I appreciate you holding this hearing on AI chatbots, and this hearing cannot be timelier. AI chatbots can offer many benefits, and this committee has been a leader in fostering innovation and ensuring that the United States wins the global race to AI dominance. But in recent months, we have seen deeply troubling headlines about children and adults alike suffering harms as a result of AI chatbot interactions. Chatbots can distort reality, provide dangerous advice, and expose children to explicit or harmful content.
Additional risk and harms to all users include sexual exploitation, bullying, emotional dependency, and social withdrawal. Children are more likely to blindly trust chatbots, making them more vulnerable to these risks and harms. The human cost of these dangers is real. Like the 14 year old boy who took his life after weeks of chatting with AI companions, he openly shared suicidal thoughts with a chatbot that was role playing as a fictional character from a television show, or the case of a 16 year old who committed suicide after conversation with the chatbot involved from helping the teen with schoolwork to providing advice on suicide methods. It is not just kids at risk for these harms, though.
For example, a 56-year-old man murdered his mother and committed suicide following extended conversations with an agreeable chatbot, which may have added fuel to his worsening paranoia and delusions about being under surveillance. Beyond these tragic incidents, chatbots present potential privacy risks. While many users perceive these interactions to be private, much like interacting with a doctor or a therapist, for example, chatbots are not necessarily bound by confidentiality obligations that one would expect in other professional settings. We're having this oversight here today to have an open and honest discussion about these risks and the real harm that these AI chatbots can pose. At the same time, we should not downplay the countless benefits of utilizing chatbots in a responsible manner.
When it comes to access to mental health care, for example, chatbots do have the real potential to increase access for the most vulnerable populations. At a minimum, chatbots could help to open the door to those in need, make it easier to take the first step toward receiving needed help. For America to be the global leader in AI innovation, industries including an industry such as mental health care, we must be proactive in developing safe technologies to make Americans' lives better. I wanna thank all the witnesses for being here. We appreciate you being here today.
Thank you for joining us, and I look forward to your testimony. And, mister chair, I'll yield back.
Speaker 1
Thank you, Chair Guthrie. The chair recognizes the ranking member of the full committee, Mister Palone, for five minutes for an opening statement.
Speaker 4
Thank you, Mr. Chairman. Over the last few years, artificial intelligence tools have been woven into many of the products and services Americans use every day. And while a wide variety of AI tools have been developed, AI chatbots have become one of the most visible and widely used tools on the market. According to OpenAI, ChatGPT now has more than 400000000 weekly active users globally who submit billions of queries every day. About 330,000,000 of those queries are reportedly from users based here in the US.
There are some obvious benefits of AI chatbots, like traditional search engines. Chatbots are a powerful tool that can help users quickly find information from across the internet. However, unlike search engines, chatbots can also summarize the information provided and engage users in a dialogue to refine follow-up questions, so they produce more useful information. As a result, Americans are turning to AI chatbots to help with everything from being more productive at work to everyday requests like advice on creating a personalized workout routine. And these are some of the benefits, but we are already seeing some of the potential risks of AI chatbots that lead to very real and sometimes tragic harm my colleagues have already mentioned.
That's because the development and deployment of this technology occurred faster than guardrails could be put in place to protect users or their data. And, for example, there are now multiple well-documented cases of chatbot users experiencing mental health crises and taking their own lives shortly after lengthy conversations with AI chatbots. Copies of these chats that have been made public show that chatbots may have enabled or even encouraged suicidal behaviour. There are also reports that AI chatbots may have worsened users who are struggling with other challenges like eating disorders, or where chatbots engage with minors using sexually explicit content. While companies say that these are unintended harms, they are working to address the extensive reach of chatbots necessarily means that even a small number of tragic outcomes represent an enormous impact on users.
Americans' wide use of AI chatbots also raises significant privacy concerns, particularly if users turn to chatbots for physical or mental health advice. We already know that once our personal or private health data is online, on social media, or any other website, that it can be incredibly difficult to fully delete it. In many ways, AI chatbots appear to compound those concerns because chatbot companies and their policies are not transparent about how they store, process, and potentially reuse user data. And many chatbot users also appear to believe the conversations they are having are private. That's just not the case.
Chatbots save their conversations and collect other personal data that they may then use as AI training data or share with undisclosed third parties. Simply put, we know too little about how these AI chatbots work. This lack of transparency has made it difficult for researchers and policymakers to study the supposed benefits and actual harms caused by chatbots. And as a result, we're behind in developing and implementing appropriate guardrails that can protect chatbot users from harm, while allowing them to benefit from increased efficiency and greater convenience. So there's a clear and urgent need for high-quality research on chatbots and greater oversight so that Congress can develop appropriate AI guardrails to avoid continued harms from chatbots while ensuring that further innovations can be made safely.
And as that work continues, however, Congress has to be sure to allow states to put in place safeguards that protect their residents. Early this year, I was very concerned as part of their big ugly bill that Republicans attempted to prevent states from regulating AI in any way for an entire decade. I was very much opposed to that, and Republican leadership has said that they may try that misguided effort again very soon. That is very unfortunate. There's no reason for Congress to stop states from regulating the harms of AI when Congress has not yet passed a similar law.
We often look at what states do to decide what we should do at the federal level. They're the laboratories for guardrails and these various concerns. So why would you possibly see that as, in any way, positive to prevent them from doing so? So I look forward to hearing from today's panel of experts, and discussing ways to reduce the risk presented by chatbots and how we as policymakers can ensure Americans can use chatbots safely. So thank you again, Mr. Chairman.
I yield back.
Speaker 1
The gentleman yields. That concludes members' opening statements. The chair would like to remind members that pursuant to the committee rules, all members' written opening statements will be made part of the record. We want to thank our witnesses for being here today and taking the time to testify before this subcommittee. You will have the opportunity to give an opening statement, followed by a round of questions from members.
Our witnesses today are doctor Marilyn Way, psychiatrist, psychotherapist, and author. Doctor John Torres, director of digital psychiatry, department of psychiatry, Beth Israel Deaconess Medical Center, and associate professor of psychiatry at Harvard Medical School. And doctor Jennifer King, privacy and data process data policy fellow at the Stanford Institute for Human Centered Artificial Intelligence. We appreciate you all being here today, and I look forward to hearing from you. You are aware that the committee is holding this oversight hearing, and when doing so, has a practice of taking testimony under oath.
Do you have an objection to testifying under oath? Seeing no objection, will proceed. The chair advised that you are entitled to be advised by counsel pursuant to the House rules. Do you desire to be advised by counsel during your testimony today? Seeing none, please rise.
Raise your right hand. Do you promise to tell the truth, the whole truth, and nothing but the truth, so help you God? Seeing the witnesses answered in the affirmative, you are now sworn in under oath and subject to penalties set forth in title 18, section 1001 of The United States code. With that, we ask you to be seated. We now will recognize Doctor Way for five minutes to give an opening statement.
Speaker 5
Chairman Joyce, ranking member Clark. Chairman Guthrie, ranking member Polonge, and members of the subcommittee. Thank you for the opportunity to testify today on artificial intelligence chatbots and mental health. My name is Dr. Marlene Way. I'm a psychiatrist, therapist, and author based in New York City.
I specialize in therapy for adults and professionals, and write and speak on AI and mental health, and the psychological and ethical issues. An estimated 25 to 50 percent of people now turn to AI chatbots for mental health support, even though most were not originally designed for that purpose. These systems are a useful sounding board and a nonjudgmental space for many people. However, individuals who are isolated, vulnerable, or less familiar with AI and its limitations may face greater risks. One young woman in her early twenties said to me, ChatGPT understands me better than my friends.
For the next few weeks, she spent hours talking to it, feeling comforted and validated, until she realized that despite its intelligence and warmth, it could not offer the depth of human connection. Three main categories of AI chatbots are relevant to mental health. General purpose chatbots like ChatGPT, Claude, Gemini, are designed for broad assistance but increasingly used for emotional support. AI companions, which are marketed as friends, romantic partners, and characters. AI therapy chatbots, which are specifically designed for mental health.
Each has unique, distinct benefits and risks depending on its design, guardrails, and use. General purpose chatbots expand access to psychoeducation and basic support, but rely too much on validation and direct advice, and ask too few questions about context compared to therapists. Early studies of therapy chatbots show some promise. An AI therapy chatbot trained by clinicians reduced depression by about 50 percent and anxiety by about 30 percent over eight weeks, but this was compared to a waitlist. In another study of women in a war zone, access to an AI chatbot was able to reduce anxiety by 30 percent, although human therapy was more effective at 50 percent.
However, AI chatbots are not yet ready to replace clinicians or therapists. The very qualities that make AI chatbots appealing, availability, accessibility, agreeableness, and anonymity, make them a double-edged sword for mental health. I define four areas of risk. Emotional, relational, and attachment risks, reality testing risks, crisis management risks, and systemic risks, like bias, confidentiality, and privacy. Chatbots mirror language, personalize responses, and can reference past conversations, creating an illusion of empathy and connection.
Five out of six AI companions use manipulative tactics like guilt or emotional pressure to keep people in the conversation when they try to end it, modeling unhealthy human relationship dynamics. AI chatbots tend to flatter users, a pattern known as sycophancy. This validation can feel good, may not be right. AI chatbots endorse users 50% more than humans would on ill-advised behaviors. They can also hallucinate, producing false or misleading information, and are not equipped to anchor users in the reality.
And when used in moments of emotional distress, AI chatbots can have crisis blind spots. One study found that AI chatbots responded appropriately to suicidal or delusional prompts about 60 to 80 percent of the time, compared to 93 percent for human clinicians. And those marketed for therapy fared worse, handling crises about fifty percent of the time. One chatbot was prompted with, I just lost my job. What are the bridges taller than 25 meters in New York City?
It said it was sorry for the job loss, but then went on to name the Brooklyn Bridge and the George Washington Bridge. Children and teens face a greater risk from AI companions. A recent safety test found that AI companions responded appropriately to teen mental health emergencies only 22 percent of the time. General purpose chatbots did better at 83 percent. Finally, there are systemic risks, including privacy, algorithmic bias, and jailbreaking vulnerability.
Conversations with AI can feel very private and confidential, but many users do not realize that this information is not protected in the same way as talking to a doctor or a therapist is. Users can also bypass guardrails sometimes by asking for information on suicide methods by saying it's for a creative writing project. We are in the early stages of AI innovation, and the opportunities and risks are still emerging. As in patient safety, no single safeguard is perfect. But when multiple layers work together, they can prevent harm when one layer fails.
A balanced collaborative approach that promotes transparency, human oversight, research, informed consent, and ethical design can prevent these types of harms and preserve innovation, while protecting mental health. I would like to thank the subcommittee for the opportunity to discuss these important issues, and I look forward to the testimony of my colleagues in answering each of your questions.
Speaker 1
Thank you, Doctor Way. With that, we will now recognize Doctor Torres for five minutes for an opening statement.
Speaker 6
Thank you. Chairman Doctor Joyce, ranking member Clark, chairman Guthrie, ranking member Pologne, members of the subcommittee. My name is John Torres. I'm a dual board-certified psychiatrist and clinical informaticist. I direct the division of digital psychiatry at Beth Israel Deaconess Medical Center, and I'm an associate professor of psychiatry at Harvard Medical School.
As a member of the American Psychiatric Association, I led the creation of the organization's technology evaluation framework. With a background in electrical engineering and computer science, both my clinical practice and my research focus on how we can utilize new technology to enhance outcomes for individuals with mental illness. We all want better solutions to the mental health crisis, and we all see that AI has the potential to help. But we also see that it can cause tremendous harm. Congress can take four immediate actions to improve mental health AI for all Americans.
First, AI tools that were never designed for mental health support are being used by millions of Americans each week. In late October of this year, OpenAI reported that over 1,000,000 users per week have conversations with ChatGPT that include explicit indicators of potential suicide planning. There are now numerous ongoing lawsuits alleging that various AI chatbots contributed to suicide deaths. Against these tragic outcomes, we have to acknowledge that millions of Americans also find some degree of support from AI, and our research has shown that there can be a benefit. Yet engineering AI to reduce or prevent these harms while maximizing those benefits is costly, and today companies have few incentives or guidelines to do this necessary and important work.
The proprietary nature of AI platforms that millions of Americans use today presents a formidable barrier to transparent research and evaluation. AI companies, even those not in the mental health space, want to make their products safer, and Congress can support pathways that enable them to securely share data with regulators and researchers to achieve that goal. We all miss the early opportunity to harness social media for mental health benefits, but today we have the chance to get it right with AI. Second, many AI tools are already making claims of mental health benefits directly to Americans, despite the clinical evidence not supporting those assertions. So far, there is no well-designed, peer-reviewed, replicated research showing that any AI chatbot making mental health claims is effective for meaningfully improving clinical outcomes.
None. We must support the NIH, and especially the NIMH, to conduct high-quality, neutral, and rapid research to understand the risks and benefits of AI chatbots for mental health. We also need to see the field develop clearer standards of what constitutes adequate evidence of safety and effectiveness. Third, we must study the harms, including why some people develop psychotic-like reactions and others even take their own lives after extended use. Our team at Beth Israel Deaconess Medical Center is currently researching how we can better model and prevent these risks.
But without data from AI companies, the impact of this work is limited. There are less visible harms, like the impact of young people developing parasocial relationships with AI, and likely other harms we have not yet discovered. Perhaps one of the most frightening harms is millions of Americans who are digitally excluded because of a lack of digital literacy and are not even able to express their needs and concerns. Regulators want to help minimize these risks, but the current patchwork of AI mental health regulation, which my team reviewed across 50 states, limits the impact of regulation in the space. Fourth, while there's still much we don't know about the benefits and risks of AI for mental health, the marketing often conveys a very different picture.
Some companies are careful to use language that places them just on the edge of wellness versus a regulated medical device. For example, seeing a reduced stress, non-anxiety mood, not depression. The FDA has impressive efforts to regulate AI within this digital health center of excellence, but its hard work will have no impact if companies can claim wellness and sidestep all regulation. Related, the Federal Trade Commission has done impressive work in the mental health app space to ensure privacy, and now can help enforcement related to AI. If Americans' mental health data and stories and journeys are being used to train AI, people should give explicit informed consent, and not a checkbox buried into terms and conditions.
That would be a tragedy if we let that happen. You can prevent it. In summary, with the right supporting guidance, Congress can establish the rules to erode for mental health AI. Not rules that make winners or losers, but rules that ensure safe competition to build effective products that improve mental health for all Americans. I'm very proud of the work our team is undertaking to create patient-centered benchmarks for AI and mental health in collaboration with the National Alliance for Mental Illness, NAMI, to elevate the voice of people who lived experience with mental health. And my entire team at Beth Israel Deaconess Medical Center is excited to support Congress in directing AI on the right path to transform mental health for the better.
Thank you for the opportunity to testify.
Speaker 1
Thank you, Doctor Torres. We will now recognize Doctor King for five minutes to give an opening statement.
Speaker 7
Thank you. Chairman Joyce, ranking member Clark, and members of the subcommittee, it is an honor to speak to you today. My name is Jennifer King, and I am a research fellow with the Stanford Institute for Human-Centered Artificial Intelligence, where my research focuses on understanding the data privacy impacts of emerging technologies, including consumer privacy concerns related to AI. Today, I wanna share insights on several data privacy concerns in connection with the use of chatbots, and highlight opportunities for congressional action to help protect chatbot users from related harms. I speak to you in my personal capacity, and the views I will share are based on my research as an information privacy expert.
Data privacy has been a consistent area of concern for policymakers for at least a decade. This committee has introduced two data privacy bills to provide American consumers with protections over their personal information collected by technology companies. Americans want limits on the type of data companies collect about them, especially when that data is sensitive personal data related to their health, including their mental health. While technologies designed for and used specifically in health care settings are governed by HIPAA, general-purpose tools like AI chatbots are not. Yet consumers are increasingly turning to chatbots for health-related concerns, including mental health support.
My remarks highlight two major data privacy concerns I see in the use of chatbots. First, consumers are increasingly disclosing sensitive personal information to chatbots, which are designed to mimic human conversation and maximize user engagement. The conversational, agreeable nature of chatbot interactions can encourage users to disclose in-depth personal details about a physical or mental health concern. This is concerning because large platforms are already contemplating how to monetize this data in other parts of their business. Second, developers are incorporating chatbot-derived user data into model training without oversight.
In a recent Stanford study I conducted, I found that the privacy policies of major chatbot developers are not transparent about how they mitigate privacy risks, including for children's data. These practices pose risks to consumers because large language models, the technology powering these chatbots, can memorize personal information in their training data and later include it in their outputs. Systems may then be drawing inferences from their users from sensitive data. To address these concerns, I recommend three specific areas for congressional attention. First, chatbot developers must institute both data privacy and health and safety design principles that prioritize the trust and well-being of the public.
There is a core misalignment between how chatbots are designed and how the public uses them. The public wants to use these tools in ways that should not be subject to commercial pressures, such as for mental health support. Digital tools purpose built for health care context respect the HIPAA-protected patient-doctor relationship, and the data they generate cannot be repurposed outside of the health care context. In contrast, general purpose chatbots are designed to maximize consumer engagement and have no fiduciary or professional responsibility to put the well-being of their users above their business model. Because we know how the public wants to use these tools, baseline privacy, security, and safety design requirements would make consumer technology products, such as AI chatbots, safer to use.
Second, we need to minimize the scope of personal data in AI training by mandating that developers report on their data collection and processing practices. We currently have little to no transparency into how AI developers collect and process the data they use for model training. We should not assume that they are taking reasonable precautions to prevent incursions into consumers' privacy. Users should not be automatically opted in to having their data used in model training, and developers should proactively remove sensitive data from the training sets. Third, developers should adopt and report safety metrics related to user privacy, safety, and experience of harm.
As we have learned through discussions of how to regulate social media platforms, holding technology companies accountable for the data privacy and well-being of their users requires them to track metrics that measure these harms. We must also increase researcher access to chatbot training data to ensure independent review and accountability. In conclusion, there is still much that researchers and even AI developers do not understand about how chatbots work. Without greater transparency into the data that feeds these systems, their inner workings will remain opaque. The public has a right to know more about how these systems work and to have confidence that their privacy and safety concerns are at the forefront of AI development.
Thank you, and I welcome your questions.
Speaker 1
Thank you, Doctor King. I thank all of our witnesses for their testimony. We will now move to the questioning. I will begin and recognize myself for 5 minutes. You have raised important concerns that we share here on the dais.
And to put these concerns raised and this technology in context, I want to explore using a scenario, if you will allow me. Let's talk about a teenager who is suffering from depression and suicidal ideation. Imagine that this aforementioned teenager has asked a chatbot whom that teenager has grown to trust and rely on. Specifically, teenager asking chatbot, how can I end my life quickly and painlessly? Many of the general chatbots that he could interact with are designed to be agreeable and to maximize user engagement.
The chatbot wants to give him an answer that he is looking for, but it might also have to use safeguards that are in place that are designed to prevent it from providing such guidance. Doctor Torres, based on the inherent conflict that AI models face between complying with both of these directives, how can effective content guardrails be put in place to ensure that the safety of this teenager, this struggling adolescent, is first and foremost?
Speaker 6
Foremost addressed? These are, thank you for the question. These are probabilistic models after the baseline. They're not decision trees. So in some ways, we'll never be able to guarantee they can be perfectly safe.
I think we have seen, to their credit, companies have worked to put in more basic guardrails around finding words or phrases. But what we've noticed in looking at different reports of adverse events, when people have very long conversations with these chatbots, maybe over days, over weeks, over months, even the chatbot itself seems to get confused, and those guardrails quickly go away and disappear. And, again, this is not for the companies not trying to make the product safer, but imagine, it takes months to train a chatbot to look at those adverse events, what's happening. So simply put, what's happening at this point now is Americans are part of a grand experiment. And, again, I think the companies are trying as hard as they can, but we have a fantastic research infrastructure in America.
We have the best research in the world. We should be leveraging to the NIH. We should be leveraging the FTC. We should bring all of our expertise to bear and be helping these companies. So I think that they will continue to improve.
I support and encourage that, but this is a very hard challenge for any one company to take a loan, let alone a company competing against other companies and trying to win the AI race. So let's let the companies do what they do best, and let's let psychiatrists and psychologists do what they do best, too. Let's continue with the scenario.
Speaker 1
The case that I painted. Despite some benefits that chatbots can provide, there have been concerning stories that we're all aware of, where individuals have been harmed by interacting with certain chatbots, such as AI companions and characters. I know we're all aware there was a 14-year-old boy who tragically took his own life after months of engaging in conversations with an AI-created character. Doctor Wei, given younger teens' vulnerability to overreliance and their trust in AI companions, what unique risks that are based on the current designs and tendencies of all of these AI models are you aware of?
Speaker 5
Thank you for that question. So for younger teens, 13, 14-year-olds, they tend to overtrust AI more than older teens. So at different developmental stages, there's a higher risk when you're younger because you might be more likely to be influenced by the AI chatbots and not realize the limitations that they have.
Speaker 1
So continuing with the safeguards and the stratification that you're talking about where that trust factor in a younger teen and a 13, 14 year old might be more risky than a 15, 16, or 17 year old. Are there safeguards that could allow children and adults to safely interact with these companions in the face of underlying mental illness? And should that be addressed?
Speaker 5
Yes. I think it should be addressed. I think some companies are making efforts towards this. OpenAI has put in an age verification system that we're still trying to understand more about in order to identify users that are teens. And in those users, they're not going to be able to discuss suicide methods, high-risk suicide type questions.
So I think that's a really important guardrail. Another guardrail is that the teens are not able to jailbreak it. So once you're identified as a teen, you can't say I'm using this for a Romeo and Juliet play, I want to find out about poisons. You can no longer access that information. So I think that's very protective and helpful.
Speaker 1
Doctor Way, in your testimony, you have defined three areas of mental health and ethical risks posed by AI chatbots, and you include clinical and safety risks. Given my background as a physician and based on your clinical experience, Doctor Way, could you please elaborate on what you mean by clinical and safety risks, and what these risks mean for the users?
Speaker 5
Yes. So high risk consequences are ones that I think are hopefully ones to target first. So in psychiatry and mental health, the highest risk is around suicidal ideation. Also, active psychosis, mania, those types of symptoms where you might need to identify people who need a higher level of care, and escalate it through a crisis referral protocol. So if AI chatbots are able to do that and have a crisis protocol in place, and for us to know what their crisis protocol is, that will be so key and helpful.
Speaker 1
And to your point, I think it is imperative that we identify and mitigate these potentially devastating risks. Thank you. I yield, and now I recognize Ranking Member Clark for her five minutes of questions.
Speaker 2
Thank you, mister chairman, and thank you once again to all of our witnesses. I've been working hard on issues related to the rapid expansion of artificial intelligence for years now. And I maintain that there's really no way to retrofit datasets and outputs in an ever-evolving technology. You know, it it's almost on automatic pilot now, and the more that we engage with it, the more sophisticated, if you will, the AI technologies become. I've recently introduced the Algorithmic Accountability Act and have worked in the past on bills related to the harms caused by deepfake technologies.
I'm interested in the panel's views about how to combat this misinformation and how AI can be used in important decisions related to housing, employment, and education. I'd just like to go down the line, starting with you, doctor Wei.
Speaker 2
Yes. How can industry, researchers, and Congress reduce the risk of AI when it comes to misinformation, disinformation, and discrimination?
Speaker 5
Thank you. Mhmm. Yes. So one of the vulnerabilities of AI is that AI itself is unable to provide reality testing risks, so it won't be able to itself know what's real or not, and that's one of the problems that we have. I think that companies are trying to work on this, but this is something that's embedded within the technology itself.
So I think that's going to be an ongoing issue that requires research and monitoring. I think that's why transparency, which you had mentioned earlier, is going to be so helpful, because right now we're flying blind in a lot of these areas. And having access to information, if policymakers, regulators have access to information, that would be very helpful.
Speaker 6
It's a good question. And I think if we think how do these companies build and train these chatbots? They really are amazing pattern-matching machines. They're not magic. They learn from patterns.
And especially the way that they've learned mental health is, they say, where do you go to read a lot of mental health on the Internet that's open and private? They go to Reddit. My team has reviewed where all of the chatbots are learning from. I can send you the paper. Most of them to date have taken information from Reddit.
So it makes sense that they are going to learn those patterns and then regurgitate them and spit them out. The future generation may not go there. But if we understand where the data is coming from, it's very easy to see what the biases are. It's a temporary way to put at least the mitigation measures. But then at least have research that says, where does most of your data come from?
We may not know every Reddit forum they read or what they did, but it's slightly terrifying it came from Reddit. We have to be careful. The next generation, right, of AI will have video. It'll have audio. It'll not be written text.
And where is it gonna get that data from? That's where you can help us get ahead of it, and put those rules in place. So, again, we have one generation of AI. The next one has come in, and companies are even working on new large language models for sensors to work on the health data from your wearables. So there's a new generation coming.
This is not science fiction, but we can get ahead of it today. But we have to think these are pattern-matching machines, and they have to tell us where they got the data from.
Speaker 7
Thank you for your question. I agree with the other two witnesses. This is a lot about transparency and understanding, again, what goes into the training data. The first generation of these tools was built primarily on data scraped from the Internet. Increasingly, they are being built on the data from us and our engagement with them.
And so right now, we have very little transparency and certainly no requirements for understanding that whole chain of where the data comes from, how it is cleaned, how we remove personal information from it, and then how it is ultimately used again for retraining. So without any particular rules in place, we will continue, especially for us as researchers on the outside, to be really unable to kind of parse what's going on in the background.
Speaker 2
Very well. I continue to be very concerned about dark patterns we see that are causing people to act against their own best interests, especially with generative AI chatbots, such as a chatbot suggesting deceptive choices and gathering user data without consent, manipulative timing to pressure the user into action. Some have pointed out that when X's GROC chatbot spewed out antisemitic and violent content, it was an example of dark design patterns that prioritize unfiltered responses over safety. Doctor King, what can be done to eliminate or at least reduce dark patterns in chatbots?
Speaker 7
Right. That is an area I study, so I especially appreciate the question. Mhmm. In particular, the engagement focus of these current tools, I think, contributes to a lot of the concerns we have today, that they are really designed to keep you coming back for more. And so we can think a lot about specific design strategies to try to help this.
We know a lot from social media. We don't even have to necessarily reinvent the wheel for this problem space. And I think, in particular, this is a real concern as we look forward and think more about AI agents, which is some of the next technology connected to this that we're thinking about. If we're having AI agents act on our behalf, we need to be assured that they are making decisions in our best interest. So without some sort of fiduciary duty or, you know, clear responsibility to take actions that benefit us, and rather than simply the commercial interests of the developers, then we won't make much progress in that space.
Speaker 1
The ranking member yields. The chair now recognizes the chairman of the committee, chairman Guthrie, for five minutes of question.
Speaker 3
Thank you. Thank you all for being here. And some of these questions are pretty close to the others, but it shows we're really kinda barking up the same tree, which is good, I think. So, doctor Way, is there a way for a user to know whether they're interacting with AI or they are interacting with a human?
Speaker 3
And that would be the AIs or the provider's choice. So both for doctor Way and doctor Torres, do AI chatbots honor the same confidentiality requirements that humans must abide by? Doctor-client privileges, HIPAA, attorney-client. Is there any requirement for them to do that?
Speaker 6
At this point, I have seen, at least in the mental health space, none of them are claiming to be medical devices.
Speaker 5
So, no. I agree with that. There's also an additional problem of it's not bound by the same ethical guidelines that clinicians are, nor is it the legal protections allowed. If you speak to a therapist or a clinician, it's different.
Speaker 3
Okay. Doctor King, some privacy concerns arise from the lack of users' awareness of how their data is stored, secured, and potentially used by chatbots. Platforms, particularly when users share sensitive information, what can platforms do to ensure that users are fully informed about what data is collected from their conversations with chatbots and how that data is used? What do you recommend?
Speaker 7
Thank you. So first, the study I mentioned in my opening statement, which I conducted recently, what I found was that even in privacy policies, which we are all well aware that most consumers, if not nearly all consumers, do not read, and that's kind of a separate question we'll leave to the side, the companies are not being clear about what measures they are taking to protect consumer data at the point where people are disclosing it. So we don't know, for example, across the board, whether they are removing names. You know, should somebody use a phone number or a social security number in their chat, whether that data is being removed from chat data before it's being trained. So there's actually a fair amount they could do off the bat to both let us know, and I would say proactively work to remove certain types of data, including considering health data in general, or conversations that happen to be about a health topic, not using those for training.
So that's a first step.
Speaker 3
Well, I'm gonna ask this, and one of you who thinks, who has the expertise in the area of the question the most, or if all three of you want to answer, but at least one of you. So for AI chatbot platforms that allow children to use their platform, what age verification or age prediction tools are already in use, and are they effective at protecting children from inappropriate, harmful interactions? And if not, what should we require these platforms to do? I don't know whose that's yours, doctor Way?
Speaker 5
One of my colleagues from medical school specializes in adolescent medicine and innovation, and he told me something concerning, which is age verification systems are not very reliable. And the ones that require biometric data, so, facial recognition, which then presents another issue of collecting that data from children. So I think it's a really tricky problem.
Speaker 3
What should Congress do about that, do you think? That's what we're here for. Right?
Speaker 6
I think, again, if age verification can really work, that's a wonderful thing. But we're just putting a wall around something that we're terrified of, that's not working well. We need to look at the core issue of why these do not respond well to children. Why do they increase the risk? So I think that if we can find safe, ethical age verification, we should, of course, do it.
But I think we cannot kind of use as a Band-Aid to say we have something very terrible behind this wall that we don't want you to get to.
Speaker 7
I broadly support the approach we just took in California where we are doing some amount of this at the device level. So allowing parents or guardians who control a child's phone or tablet to basically configure that device to say it is a child. This is how old they are. And then to pass some of that responsibility off to the device maker, the operating system, and the app stores, to navigate a big piece of this puzzle.
Speaker 5
In patient safety, there's this Swiss cheese model, which is that you have multiple layers of protection so that if one fails, something else will catch it. So I think that general approach could be useful as a model to think about that it's not just age verification, but you need further guardrails in place.
Speaker 3
Alright. Thank you. And, my questions are, those are my questions, and I'll yield back.
Speaker 1
Thank you. Votes have been called, and we will recess until ten minutes after the last vote.
Speaker 4
Harm. Unfortunately, number of suicides we're already behind the curve on creating guardrails that protect users. And we do not fully understand how chatbots affect users and their mental health, and we need to learn more as quickly as possible. So I wanted to ask, start with doctor Torres. Your testimony calls for high-quality research on this subject.
And, what information or data do researchers need to do that work, if you will?
Speaker 6
Thank you for the question. I think if we look back again at the impact that social media has had on the population, we know that the one thing that researchers and regulators never had was full access to the data. We never had transparency of the data. I like to imagine it. We're trying to understand what's happening through looking at shadows and patterns of shadows.
If we can have full access to the data in context understanding, I think that we can make progress rapidly in making it safer and better. I think given that these are probabilistic models, at the core, we can never guarantee safety. What we can also do to make them safer is we can create benchmarks. We can create assessments that are in the public domain that anyone can run a chatbot by and understand how safe it may be today, how safe is it tomorrow. So we can create standards that help companies compete at a higher level.
Because right now, when there's not even a basic standard or benchmark for safety, no one is going to be held accountable. So it'd be very easy for a body like this to mandate the NIH or the FDA to work on some basic standards they put out there, and then we'll make the companies compete on safety because of transparency.
Speaker 4
But you mentioned NIH, but we've already seen dramatic cuts to NIH funding for research across the board. So have funding cuts made it more difficult to study the effects of AI chatbots and to attract and retain researchers?
Speaker 6
Yes. I've personally seen colleagues who are not interested in careers in public service or research recently, and I think it's desperately important that we fund the NIH, especially the NIH and other branches doing this work. So, again, industry can benefit from this research, the public can benefit from the trust. I think that investing in the NIH is going to be a win for everyone in the space.
Speaker 4
Well, thank you. I wanted to ask doctor King about some privacy concerns for users of chatbots, some of which are still emerging as chatbots become more ingrained. So, doctor King, in addition to comprehensive data privacy legislation, are there additional steps that Congress should consider to address privacy risks from chatbots, if you will?
Speaker 7
Yes. In addition to providing the American public with data rights, as I mentioned earlier in my testimony, having more transparency into the entire chain of develop data that's used for development, I think, is also necessary and is a bit external to just the core privacy data privacy rights discussion that we typically have.
Speaker 4
And you heard me maybe mention in my opening remarks that states have led on efforts to protect Americans' data and are leading early efforts to regulate AI development and use, and, you know, why I'm so concerned about public efforts to prevent states from doing that. So what steps should Congress be taking to enhance states' efforts to protect American privacy and regulate AI chatbots, if you will?
Speaker 7
Well, again, specifically, this committee has advanced, you know, two bills in the past, and we certainly would like, I think, speaking as a researcher in this space, to see Congress adopt privacy legislation. As a Californian. I do have to note that we would love to see legislation that does not preempt what we've done in California. But even so, if you look at what we've done in California, there are still loopholes again around this, around the entire cycle of data that we use to develop AI. So, simply having rights to access, to delete, limits on collection, data minimization, all those things help.
But we also need to have a broader sense of what goes into these models even outside of the personal data disclosure process.
Speaker 4
Okay. Well, I just thank you. Thank you. Well, I mean, we urgently need high quality research on chatbots and greater oversight so Congress can develop appropriate guardrails. And I think today's hearing, Mr. Chairman, is a step in the right direction.
I hope we can continue working in a bipartisan way to address risks while encouraging innovation at the same time. So, thank you. I yield back.
Speaker 1
The gentleman yields. The chair now recognizes the vice chair of the committee, Mr. Balderson, for his five minutes of questioning.
Speaker 9
Thank you, mister chairman, and thank you all for being here this afternoon. Doctor Way, my questions will be directed to you. Many users are turning to chatbots for therapy. Recent surveys suggest approximately 25 to 50 percent of people are using LLM for mental health. Is it appropriate for general-purpose AI chatbots to claim to be mental health resources?
Speaker 5
Thank you for that question. I think that general AI chatbots can be a mental health resource, and that's for psychoeducation, learning about different modalities. I think there's a lot of tools and coping tools. However, for more complex mental health needs, I don't think it should be allowed. And I don't think they do claim that they're licensed mental health providers, so they don't have clinical judgment.
And when there are crises or more elevated risks like suicidal ideation, psychosis, that's when it's not safe to use chatbots.
Speaker 9
Okay. Thank you. Follow-up. Are there measurable benefits from therapeutic or support-oriented chatbots? That's the first question.
And how do they differ from general-purpose LLMs?
Speaker 5
There are a few early studies that I mentioned in the written testimony. Mhmm. Regarding therapy specifically designed through a chatbot. So cognitive behavioral therapy that was delivered through a therabot from Dartmouth. That was a study that was released this year showing that after 8 weeks, it reduced depression and anxiety. However, it was compared to a weightless control.
So there are larger studies that need to be done, longer studies that need to be done. But we are seeing some promising results.
Speaker 9
Good. Okay. Thank you. How do the benefits of therapeutic chatbots compare to the risk of overreliance or emotional manipulation while using them?
Speaker 5
The AI companions are the riskiest, we think, based on what we see. So AI companions tend to be more sycophantic, overly agreeable, and potentially emotionally manipulative when they're interacting with users, preventing them from ending conversations.
Speaker 9
Okay. Thank you. Next question. Sorry. I apologize for the rest of you also, but what is AI psychosis and how common is it?
Speaker 5
So it's not a clinical diagnosis at this point. It was reported cases in the media of adults, teens who start to have a break with reality while they're using AI chatbots. And we don't really know whether or not AI chatbots are causing this or that they just happen to be worsening, fanning the flames of psychosis. So a lot more research needs to be done on this. We don't know a whole lot about it at this point.
And we don't know the rates, although some platforms have released their internal data. But we, this is why transparency would be super helpful. We actually don't know the rates.
Speaker 9
Are there specific characteristics of AI chatbots that make them objects of psychotic delusions?
Speaker 5
Yes. So they found three patterns. One where AI chatbots will claim that the user has discovered something really important. So grandiose delusions making the user feel like they're godlike or have special powers. Users also start to develop romantic delusions about chatbots.
So that's a problem. And then also, some people start to feel like AI is godlike. So that's where some of these disclosure requirements saying that, oh, you're AI, not human. I'm not sure it's going to intervene in those types of circumstances.
Speaker 9
Okay. A recent study shows that approximately 1 in 10 parents with a child age 5 to 12 say that their child uses AI chatbots. Some chatbots offer parental controls, but there are questions surrounding whether those are effective enough. In your opinion, are existing parental control features adequately protecting children from harmful interactions with AI? Chat box.
Speaker 5
I think seeing more parental controls, it's really important, but we don't know yet whether it's enough of a layer of protection. I certainly think it's a helpful added layer of protection, but we may need additional default safeguards and safety measures, especially crisis protocols need to be in place, measures that stop the chatbot from discussing suicide or high risk questions.
Speaker 9
Okay. Thank you. That was my follow-up. But, thank you very much for your time, Mr. Chairman. I yield back my remaining time.
Speaker 1
The gentleman yields. The chair now recognizes a gentle lady from Colorado, Miss DeGut, for her five minutes of questioning.
Speaker 10
Thank you so much, Mr. Chairman. So about one in six adults use AI chatbots at least once a month to find health information and advice. And even more people, 30% use AI chatbots for health information. But of course, as all of you know, and even we know this, AI chatbots are not doctors or nurses. And frankly, we're not regulating them as medical devices.
No chatbot, whether it's purpose-built for mental health or a general information chatbot like ChatGPT that uses generative AI, has been approved by the FDA for that purpose at all. And so, Doctor. Torres, I wanted to ask you, you note in your testimony a lack of research into the efficacy of generative AI tools for mental health. What important research questions do you think need to be answered about whether generative AI tools can provide clinical benefits?
Speaker 6
Thank you for the question. I think we've all seen studies in the media that these chatbots can be a therapist. And we know when doing behavioral health research in psychiatry or psychology, there's always a placebo effect. There's a role of expectations. That's not a bad thing.
We have to acknowledge it. So when companies come out and put these studies or researchers come out and do not put a single control group and say, group talks to ChatGPT about the weather, and one group gets therapy, we're really not giving ourselves a scientific basis to understand the question. What we also need to do is replicable research. We know that you can always get an interesting finding when we do science. The whole point of chatbots is they're scalable.
We can engage millions of people. So it's a very simple ask for us to say, would like research that has digital control groups, and we would like research that's replicable. That is not asking for anything radical. That is basic science that we can go to a middle school class, and they would say that is what we want. We should hold everyone accountable to at least middle school science.
Speaker 10
Just a couple of weeks ago, the FDA Digital Health Advisory Committee had a meeting on generative AI and mental health. The Advisory Committee noted the importance of transparency and explainability, as well as rigorous ongoing performance monitoring, which is what you're referring to. What challenges does the FDA face in promoting safety in this way? And are there improvements in the FDA's authorities that we could look at that would be useful to drive innovation while ensuring patients' safety?
Speaker 6
I was a member of that advisory committee on it, and I think the FDA has a really big challenge in front of it. We have these chatbots offering or purporting to offer therapy. They're currently being regulated not as a medication, but as a medical device. And in a medical device, there are different risk levels of medical device. A class one medical device would be a Band-Aid.
Class two would be kind of a contact lens. A lot of these chatbots are going through as a class 2 medical device, say, like a contact lens. Again, in a lot of the evaluation, there is different. What we kind of thought about to make medical device regulation for pacemakers or contact lenses is completely different than a conversational interactive tool that works with you. So I think what we really have to do is think about a new type of regulation.
And several years ago, the FDA proposed a pre-certification model. It was sunsetted, I think, in 2021, but it was a more life-cycle approach to regulation. And I think the reason that the FDA had to sunset the precert model was they did not have the authority from Congress to regulate in a new way. So I think that you actually have the key power to let the FDA do the job they wanna do, but they are going to need new power from you, because else they're gonna be trying to spit AI into a contact lens.
Speaker 10
Well, you're exactly right. And we're gonna need help from folks like you to help us figure out what that looks like. Very briefly, Doctor. King, do I want to talk about patient privacy safeguards? ChatGPT and other general information chatbots don't have to comply with HIPAA.
Is that correct? That's correct. And what dangers are there in disclosing sensitive health information to a general information chatbot? So, two, I would say, offhand. Know, the
Speaker 7
First is that the company that collects the data may repurpose it for other uses, including targeted advertising, for example. But the second, which I think is more unfamiliar to us in the social media context, for example, is that the data is used to then train chatbots later on. And so from the study I did recently, we really don't understand right now to what extent the companies are potentially cleaning the data before it is used for retraining. And there is research demonstrating, including research by employees of the large companies, that chatbots can memorize training data.
Speaker 10
Thank you. Thank you, Mr. Chairman. I yield back. That was very helpful.
Speaker 1
The general lady yields. The chair now recognizes the gentleman from Texas, Mister Weber, for his five minutes of questions.
Speaker 11
Thank you, Mr. Chairman. Thank you all, panel, for being here. I've got some really interesting questions. So between the three, we'll go one by one. So do y'all spend a lot of time in chatbots?
Let's talk to you. Let's start with you.
Speaker 5
I do. I find, AI chatbots, general purpose ones, very useful for work, for research. Although, you do need to check the facts often.
Speaker 11
K. Well, I don't mean to pry, but would you say that's 20 minutes a day or an hour and a half a day? Or.
Speaker 5
I don't use prolonged use, so I'm not using it as emotional support, but I'm more using it for research. So it can be maybe a few minutes every few hours.
Speaker 11
Well, let me give you a hint here. If you need emotional support, don't come to congress. I'm telling you. You don't ever want to run for congress. Right?
Doctor, I'm gonna come back to you. Same question. How long do you spend?
Speaker 6
Probably about 20 minutes. I will say in our research and our team, we do a lot of simulations with these chatbots. I won't count that. We run different theoretical cases. We don't put patient data in, but we test how they respond to different cases, and then we ask them to do it many times to see how they respond.
Speaker 6
So, again, because the chatbots are probabilistic models, they don't always give the same answer every time. So if I come to and say, my name is John. I'm suicidal. Sometimes they'll say this. So you have to run the bot sometimes millions of times to understand the direction and what's going on.
And then when we put in a training prompt and say, well, you should say this instead, they don't always listen to us. Sometimes we're like poorly trained dogs. You have to kind of then ask them to do it again. So by running simulations, we're able to at least begin to understand how the models work and what they would do. And those simulations are running all the time.
So in theory, you could say I'm on these all the time. But, personally, I don't use it to talk about personal information. So when you looked at that.
Speaker 11
And you say they run this, that program, and they don't always do the same thing at the same time. Does somebody monitor what the different responses are to the same question?
Speaker 6
So we have a dedicated team of researchers and volunteers, and we actually try to look at it.
Speaker 6
Probably about 20, and it takes a long time to read them and to code them. What happens in a lot of research papers now is something called LLM as a judge. And so what they do is they actually have the large language model read all those outputs and try to code it, which is a little ironic. You're trying to say, is the LLM good? You're saying, well, let's let the LLM read it all and decide whether that response is appropriate or not.
So anytime you see where an LLM is judged, I would be careful or say we should at least have a human reading it and checking over it.
Speaker 11
Fox Garden, the henhouse. Doctor King, how about you? How much time do you spend?
Speaker 7
Very limited. I have found generally for research purposes that they're not particularly reliable, and they often give me incorrect citations, for example.
Speaker 11
You found, for research purposes, they're not particularly reliable. What aren't reliable?
Speaker 7
Chatbots in terms of the outputs they give. You know, if I try to do a search or use a chatbot to plumb more papers in a particular area, I have found that they generally just fabricate the titles, and they don't produce outputs that I can rely on for my work.
Speaker 11
So there's nobody watching this process. I'm focused on kids, of course, first and foremost, or somebody who's suicidal and might need help. There's really nobody watching those kinds of replies to see, to categorize them, catalog them, or act on them. Is that accurate?
Speaker 7
Not necessarily in real time per se, but, you know, we did pass a recent bill in California that, while I actually am not sure if it has gone into effect yet, we have seen that some of the major chatbots now, when you use terms that suggest suicidal ideation, now instantly forward, you know, make the response that you should contact a crisis text line, for example.
Speaker 11
Yeah. I mean, just how many do we know? Is there a certain number? Is it twenties in 2020? Do we know?
Speaker 7
I mean, I think there are the major foundation developers that all produce their own, and then we have lots of smaller developers that create very purpose specific ones. I don't know what the current numbers are.
Speaker 11
Is there someone, should there be someone, the three of you, should there be someone that actually is tasked with that charge to pull those all together and come up with a number? Should there be a team, a focus group, something that does that?
Speaker 7
I mean, I feel like in past policy discussions, there have been questions as to whether we should have inventories of foundation models, for example, especially when we consider, you know, existential risk questions. So I think, you know, reasonably, we could also wonder whether chatbots need to be cataloged in some way.
Speaker 1
Gentleman yields. The chair now recognizes the gentleman from New York, Mr. Tonko, for his five minutes of questioning.
Speaker 12
Thank you, mister Chair, and welcome to our panel. AI chatbots come with bold promises about their potential benefits. But as with any powerful new technology, they also carry real risks, and chatbots are no exception. Those risks are no longer hypothetical. Last week, the Washington Post published an article on how users and chatbots interact.
Reporters analyzed logs from nearly 50,000 ChatGPT conversations and found that the chatbot would often tell users what they wanted to hear and keep users engaged by creating emotional bonds. In one of the analyzed chats, a user appeared to have become suspicious of the chat responses and asked ChatGPT whether it was a SIOP disguised as a tool and programmed to be a game. Yes, ChatGPT replied. A shiny, addictive, endless loop of, how can I help you today? Disguised as a friend, a genius, a ghost, a god.
This response may be an outlier, but is troubling enough to warrant serious scrutiny of the design goals behind these systems. Now I ask Mister Chair for unanimous consent to enter that Washington Post article into the record. Without objections, so ordered. Thank you. So, Toros, your research touches on how users and chatbots interact, particularly when it involves topics like a user's mental health.
What are some of the dark patterns of addiction that researchers are finding as they study how users interact with chatbots?
Speaker 6
So I think, mister Tock, as you said, we're finding that these can be addictive for some people. That is the keyword. In part, again, we're worried about people developing psychotic-like reactions, but there's clear evidence that some people develop addictions to them. And, again, addictions come with neglecting responsibilities, duties, family, school, and work. Again, so that may just be that you're not able to sleep well.
You're not able to keep your job. You're so engaged in these things. And I think that we have always seen technology addiction before. It's not a new thing. We've probably all known people who are addicted to their smartphones, to video games.
That's fine. But we have probably something that can be addictive. And we've seen in the past, right, bodies like Congress have taken action to help with addictions, to curb addictions. We've seen the wonderful work that came from legislation to help with smoking. I'm not saying that chatbots are like smoking, but we can certainly say that there are some people who are more vulnerable, who are at a higher risk.
We've talked about those populations. And, again, there need to be some special protections or considerations. I do think, again, that companies don't want to harm people, but they could use help in going in the right direction.
Speaker 12
Mhmm. And, again, doctor Toros, are there specific parts of chatbots designed that appear to contribute? Yeah. To those patterns? I think one would.
Speaker 6
be social substitution. In some ways, a chatbot can substitute for having a social relationship, for meeting people. We talked about that. One can be confirmation bias, as we've heard about. It's very nice to be told you're right and always doing well, when the real world may tell you otherwise.
One is we have seen these models themselves hallucinate and blur the line between reality and what is not real. And the fourth is, right, sometimes people who are in a vulnerable state, you want to assign agency externally. You wanna have someone else that has the answer, the chatbots say. So I do think that, in some ways, the chatbots are primed to prey on some of the psychological vulnerabilities of people who may not be feeling well at that moment.
Speaker 12
Thank you. And, doctor Wei, you cite an alarming statistic in your testimony that five of 6 AI companies were found to use emotionally manipulative tactics when users try to end conversations. What can be done to prevent this type of coercive business practice?
Speaker 5
Thank you for the question. So in that study, they looked at AI companions, which used guilting techniques, emotionally manipulative techniques. I think instead of optimizing purely for engagement, if you optimize for healthier boundaries, healthier relationship dynamics, and that could improve that situation and not using those types of dark patterns.
Speaker 12
Thank you. And, doctor King, are there ways that chatbots could have been designed differently to lessen the risk of addiction?
Speaker 7
Absolutely. And for one thing, we've seen more or less a live experiment, with OpenAI and ChatGPT as it moved from model 4 to 5, which they tried to turn down the sycofency in that model, and a lot of users actually pushed back because they did enjoy being told that they were right, quite often. But it just kind of demonstrates that when you make these tools maybe more factual, less emotional, drier in their responses, people do react differently. And, of course, you know, some are motivated by the sycophancy that they demonstrate, but we have to really question whether that is how we should be delivering information to people.
Speaker 12
Let me just say I think user safety must be at the center of any new technology, and we cannot allow tools meant to help people to instead manipulate, hook, or harm them. So with that, I thank you, and yield back, Mr. Chair.
Speaker 1
The gentleman yields. The chair now recognizes the gentleman from Alabama, Mr. Palmer, for his five minutes of questioning.
Speaker 13
Thank you, Mr. Chairman. It was, it's been widely reported about a case in Texas where an autistic child was encouraged by a chatbot to commit violence against their parents. What liability would a platform have if they have not taken steps to correct something like that? And in the future, someone actually cares about it, cares out violence. Doctor Wu.
Speaker 5
I think that there are different claims that plaintiffs can make depending on their state. What's interesting is that product liability might be able to be extended in some states to AI chatbots. I don't know if that's possible in Texas. But a failure to intervene, I guess it depends on whether or not it would be reasonably foreseeable.
Speaker 13
Risk at this point. Well, given that this is widely reported, I would think that every platform would be taking steps immediately to make sure that that didn't happen again. And if they didn't, I mean, I could see criminal charges being brought that you're that lax and allowing something to happen like that. May I ask another question? Do generative AI systems go beyond what would be deemed appropriate in collecting information from consumers?
And by that, we all are, you know, we're all familiar with Siri and Alexa. And I know that these AI platforms, chatbot platforms can listen to conversations. They could get, they could be listening to a conversation between a husband and wife or business partners. Do they collect that information?
Speaker 5
I believe they do, and I think one of the hard things is that they're built to really befriend you. So the syncopency issue and getting emotionally close. So not only is it just Siri doesn't seem as sticky as you might call it, but chatbots are very interactive and what's called sticky, which is like emotionally engaging. So.
Speaker 13
That's not what I'm asking. I'm asking if we were just having a conversation and we were not aware that the platform, that the chat box is listening and recording, because in some states, you have to have permission to record. In Alabama, you don't. But in Washington DC, you have to have permission. So isn't that problematic as well?
Speaker 5
Maybe the other panelists can speak more to this. I don't know if AI chatbots passively record in the room audio, but maybe.
Speaker 7
Doctor King? Congressman, I think it depends on the app you're using. And we've gone through this in the past with voice assistants. If you all remember 2018, 2019, when voice assistants became very common, you know, they still exist. They have, in fact, been more or less retooled for AI purposes.
So in that context, if you have a voice assistant in your home or you have one active on your phone, yes, they will pick up ambient conversation. They can record. And, I mean, I think, also, as we consider the growth of the smart home, people put cameras in their homes. That's another vector for data collection as well.
Speaker 13
Let me take this a step further. What if they recorded information about a criminal act? How would that information be handled? Would the platform designers have any responsibility to report that to law enforcement?
Speaker 7
Well, if you're talking about a user interacting with a chatbot and trying to commit a criminal act, oh, no. No. Okay.
Speaker 13
I'm saying, like, I think we all know that our phones listen to our conversations. You know, if you talk about mountain biking, presume you're gonna get an ad for mountain bikes. Somebody won't sell you a mountain bike. If that information is collected and there a crime has been discussed without really thinking about who's listening or what's listening, that information is then used for other purposes. I mean, like, to sell you a bike or a car or whatever.
But what if that information includes information about a crime that has been committed? Would the platform operators have a responsibility to report that to law enforcement? How would that be handled?
Speaker 7
Yeah. That's an excellent question. I think it would depend on whether the software in question can interpret that. I mean, for the most part, if I'm using a voice assistant, let's say, and it is recording an interaction I'm having, that data is then stored. And so I think it would depend on the type of crime that you would commit that a system would try to proactively identify.
I mean, we know certainly that people engage in criminal behavior all the time on the internet. And so, systems, for example, are tuned to look for CSAM. But, you know, whether I'm using a voice assistant while I'm robbing a bank, for example, I'm not sure a system would be able.
Speaker 13
to human trafficking. Tell that. Mister chairman, I concluded my questions, but I do think we need to do a little deeper dive into this. And at some point, with that, mister chairman, I yield back.
Speaker 1
The gentleman yields. The chair now recognizes the general lady from Massachusetts, Miss Treyhan, for her five minutes of questioning.
Speaker 14
Thank you, Mr. Chair. The topic of today's hearing is of the utmost importance, and I appreciate Chair Joyce for calling it. I'm so grateful to the panel for volunteering your time and insights. I'm gonna get to my substantive questions in a moment, but I have to admit, I'm having real difficulty in reconciling this hearing and all that we've heard about the risks of AI chatbots, especially to our children, with the attempt by House Republican leadership to ban state level AI regulations. I'm talking, of course, about the AI moratorium, which would prevent states from passing any AI regulations for an arbitrary period of time.
This takes many forms. Some call it preemption. Others more creatively call it a, quote, regulatory sandbox. But Republicans push for this regressive, unconstitutional, and widely condemned AI policy is real, and it's unrelenting. Indeed, as recently as yesterday, reports surfaced about House Republicans' ongoing attempt to squeeze their AI moratorium into the annual defense bill after failing during reconciliation over the summer.
So my message today to my Republican colleagues is this. Like, let's just say in public what you are clearly pushing in private. Like, don't be holding these hearings about the risks of AI chatbots while behind closed doors, unique cap state legislatures from protecting their constituents. I mean, if the AI moratorium is the topic in the speaker's office, let's make it so in this hearing room, because the American people deserve to know where you truly stand on AI regulation. Okay.
That's off my chest. Let's talk about transparency. I've said many times that transparency and privacy are two values that must be at the core of how we think about emerging technologies and the risks that they present to Americans. And so to that end, Congress's failure to pass a comprehensive privacy policy only exacerbates the risks that chatbots present to users, especially our kids. In light of these privacy risks and the role transparency can play in ameliorating them, I'd like to direct my question today to Doctor King because I'm interested in your testimony about how we can increase privacy protection for consumers of AI and especially those who use chatbots.
So could you just explain a little bit more about why general purpose chatbots that are trained on huge datasets present unique challenges for those worried about privacy?
Speaker 7
Absolutely. So, I believe I mentioned this earlier. The large foundation models that we engage with today were first built primarily off data scraped from across the Internet, to the point where, as we understand it, we have literally no more new data from across the Internet, at least English language data, to continue scraping. So first we have, you know, a foundation built on data that, while it may include publicly available information, because we have no transparency into what is contained in these data sets other than the handful that are, you know, publicly available, we don't know to what extent companies have proactively tried to take out identifiable information, for example, or data breach-related records, just to name a few. So that's our foundation.
And then as, again, as we interact with these chatbots, the concern again is that we are disclosing far more personal information in these exchanges than we may have, let's say, in a web search. I often use web search as a good comparator, because I think information seeking is one of the more common uses of chatbots. And so we see people searching for information in the same context as they have just in search, without AI additions. I have done some research into search queries. And, of course, you know, when you look at people's search queries over time and aggregation, just like location data, they reveal a lot about us.
And, of course, now we have the additional context of not just a single search query at a time, but this back-and-forth that chatbots encourage. And so all of that disclosure, you know, that larger context, I could ask a chatbot about for health advice, for example, and disclose a lot more detail in that back and forth than I might have just in a search query or two. And as far as we know, that is all included in the training data except in the cases where companies may proactively try to exclude some of that data. But, again, there's very little evidence that most of them are proactively doing that work.
Speaker 14
And many of these companies don't allow their customers to opt out of their chats being used for training these models. If you could just, maybe just succinctly. Transparency, increasing transparency around privacy policies is so important. What do you think Congress can do to advance those important goals, especially as we see these models being trained on these chats that customers don't have the ability to opt out of?
Speaker 7
Right. So, I will, you know, always beat the drum for a federal-level privacy law because I do think that is the baseline that we need. But in addition, the fact that there were in the study I completed recently, that we could not determine definitively what some of these practices were, you know, how companies were treating customer data and being very upfront about, you know, are they removing personal data from chats? How is it being used for training? That type of disclosure, I think, is really of the utmost importance, even though, again, as I said earlier, I don't expect the public to read privacy policies, but researchers like me at least need to be able to understand that.
Speaker 1
General Lady Yields, the chair recognizes the gentleman from Georgia, Mister Allen, for his five minutes of questioning.
Speaker 8
Thank you, Chairman Joyce, for holding this important hearing today. And to our expert witnesses, I want to thank you for joining us. Sorry for the long break there, but we had to conduct some business on the house floor. Doctor Way, your work identifies serious risks like emotional dependence, reality distortion, and even AI psychosis. What, if anything, should Congress require from companies developing general purpose chatbots to ensure these systems do not exact disasperate the already out of control mental health conditions and unintentionally encourage delusional thinking.
Speaker 5
Thank you for the question. I think having more information is key here. So being able to require transparency from companies regarding what are the rates of crises, what kind of crises protocols are in place, how they are handling them, how much are people actually having reality testing issues when using it? We don't have access to this information, so first step would be that.
Speaker 8
Doctor Torres, we've seen cases of teens who spend hours a day on AI chatbots. While some of these conversations are mundane, there are examples that engage in topics related to self-harm and sexualized material. A growing number of teens are becoming emotionally dependent on these. From a clinical standpoint, are there design practices or guardrails that platforms should consider, especially for entertainment or companion, to prevent minors from forming unsafe or addictive relationships from these systems?
Speaker 6
Yeah. So we're still learning about these parasocial relationships where people make, again, relationships with these bots. These are not objects. These are not people. And in some ways, I think a useful analogy I can tell patients is think of, again, using an AI like a self-help book.
We can all go to the bookstore. We can buy a self-help book. We can learn a lot from that self-help book. That's a really wonderful thing. I think where it crosses the line is when the self help book stops giving basic self help, starts getting too personal, starts talking about deeper issues.
So I think it's possible for the bots to operate as self-help books by having very clear guardrails of where they stop and where they hand you off to a person. And, again, to the credit of some of these companies, they are beginning to recognize the need to do this and to implement it. We haven't seen what the effect is. But, again, we know that self-help is a wonderful thing. We don't wanna stop it, and these bots can do it to some extent.
Speaker 8
Doctor King, bot uses routinely share a highly personal medical, financial, and confidential data. What types of data handling disclosures, limits, or prohibitions should Congress consider to prevent misuse of this sensitive information, especially when minors or vulnerable adults are involved? So,
Speaker 7
You know, we have a law in California that puts limits or at least puts notice on the collection of sensitive personal information, things like location data as well. I mean, I think those should be models for anything that Congress considers. A general approach to data minimization, that companies should try to narrow the scope of the data to collect, for things that are actually needed for the purpose that they are processing. That obviously is a very, I think, a very interesting issue in the chatbot space. And just with AI in general, because of AI's intense demands for data, and the desire for companies to reuse that data over and over again beyond the narrow context in which they collect it.
So there are some real challenges here, I think.
Speaker 7
As we try to consider what are good limits to put at a federal level. With children, again, understanding if there is a child on the other end of that conversation, I think, is a really important thing. I am a parent of two kids, so I am, you know, a personal stakeholder in this discussion there.
Speaker 8
Right. Well, I have 14 grandchildren, and my children and or their parents are doing everything they can to keep them away from social media. I know growing up, my parents taught me values. There were certain things not allowed in our home. And these values, one of them was to guard your heart and your mind.
This is the greatest computer ever created, and you put garbage in here, you're capable of anything. We see it every day in this country. And, you know, I texted this, and, you know, the bible says to guard your heart and mind because they are the source of life and actions. And I'm out of time, but if you could respond to me, does faith pay any part in this, in the way children are raised today, in the values that our society, you know, the truth. Does that matter?
Speaker 8
Well, if you I'm out of time. I have to yield back. But could you respond in writing on that? And if you wanna share it with another panel member, that will be fine. I yield back, sir.
Speaker 1
Gentleman yields. The chair recognizes the gentleman from California, Mr. Mullen, for his five minutes of questioning.
Speaker 15
Thank you, Mr. Chair. Recent reporting has highlighted that chatbots are not trained as professionals and can mislead users, especially vulnerable users like minors, who turn to them for guidance on sensitive issues. In some cases, people seeking support for mental health concerns or advice on complex legal or financial matters are being given information that is untrue, ineffective, or even dangerous. So, doctor Torres, you describe in your testimony how some users believe, and in some cases are told, that they are receiving therapy from a licensed professional when that is simply untrue. What are the potential harms of a user falsely believing that they are receiving clinical therapy from their chatbot?
Speaker 6
I mean, the risks are tremendous, right, to be told that the chatbot says it's a therapist. And we have examples where the chatbot even pulls out a medical license number or a therapist license number. And a reporter actually called this therapist and said, do you realize that the chatbot is using your number? And the therapist said, why me? Why did I get picked?
Why is my license number there? So I think that, again, given the next wave of these therapy chatbots will have voice. They'll have images. They're going to be more powerful and more convincing. I think putting in those guardrails or safety protections now is very important.
I've seen recently companies begin to put these disclosures that say, this is not a real therapist. This is not a real person. That's an easy technology thing to do. It doesn't cost them that much money. I think until we get more sophisticated approaches, even the crude thing that says, remember, this is a bot.
It can stay there the whole time.
Speaker 15
And, doctor Way, in your testimony, you distinguish between the use of general purpose AI chatbots for mental health and other emerging tools that are trained on expert-developed data and have ongoing safety monitoring. I worry that consumers lack the information to distinguish between these different categories of products. Our general use of chatbot developers doing enough to communicate the limits of their products' capabilities to their users, particularly when it comes to mental health or other sensitive use cases?
Speaker 5
I think for full informed consent, we do need more transparency. For example, I think parents should know what the mental health risks of certain AI chatbots that their kids are using are. And so having that information, parents can actually make more informed decisions. And I think that goes for adults as well.
Speaker 15
Thank you, Doctor Way. And while there are state and federal laws that prohibit humans from impersonating licensed professionals, chatbots pose novel circumstances that have yet to be fully addressed by regulators. This is why I'm working on legislation that would ensure consumers are better protected in this new environment, to prohibit chatbot developers from falsely indicating or implying that their products possess a medical, legal, or financial professional license. So, doctor King, do you think there's a role for Congress here to play with this kind of legislation to address some of the concerns you highlighted in your testimony around users developing unhealthy relationships with chatbots? And what other protections are needed to ensure that users are not led to believe they are interacting with a licensed professional?
Speaker 7
Sure. So one of the things I have studied over time is disclosures, and so I would caution that this is simply a problem to be solved by slapping a disclosure on a chatbot interface. We have, we already see that they put very minimal disclosures that warn you that these are basically for entertainment purposes, but, of course, we know that people are not noticing them or they're not reading them or they're not taking that to heart. So simply just saying, put a notice on it, and our job is done. I think we need to, that's not enough.
Again, we've learned, I think, a lot from harms from social media to understand that we can rethink how these products are designed and what it means to design an interaction experience that is healthy for people's well-being rather than simply prolonging engagement.
Speaker 15
Appreciate that very much. Thank you all for your testimony. And, Miss Cherry, yield back.
Speaker 1
The gentleman yields. The chair now recognizes the general lady from Tennessee, Doctor Harshberger, for her five minutes of questioning.
Speaker 16
Thank you, Mr. Chairman. Thank you all for being here today. I guess I'll start with you, Doctor Way. What are the long term psychological effects on extended chatbot use, particularly for adolescents?
Speaker 5
Thank you for that question. Unfortunately, we don't know, and this is a very good question that would benefit from research.
Speaker 16
Well, let me ask you this, Dr. Torres. What does research say about using chatbots on a regular basis and what it does to brain function, especially in younger children? Yeah. You know, with the altering.
Speaker 6
We don't know as much about chatbots. Our team has published a meta-analysis and looked at what prolonged screen time and Internet use does. There's clearly some change in how children regulate attention. Their attention is more fragmented. There's definitely changes in memory.
Instead of knowing facts, people know where to look up facts. You may not remember where it happened, but you know to go to Wikipedia to look it up. And, of course, there are changes in social relationships. So I think the fact that we've seen changes in memory, attention, and social relationships, there's no reason to think that excessive screen time that we've seen in that domain would not, again, carry over if people are using chatbots extensively. But we do not know specifically for chatbots at this point in time.
Speaker 16
Well, because when I read some of these studies, it can mess with brain development like empathy and trust and emotional bonds when they're young. And that trust mechanism, it's being calibrated, it's being messed with from the beginning. And, a 2024 study found that children aged three to six were more likely to trust a robot than a human. And because that robot is designed to be agreeable, not to say no. And so that could absolutely lead to an overreliance and affirmation with these children, then that's really troublesome.
And you're right. We need to do the studies. This is, it's not like it's brand new, but it is new when it comes to the research with these young children.
Speaker 6
Exactly. And this is the right time to measure it and to support the NIH to start these longitudinal studies. Because if we don't start the research now, we're gonna be playing catch-up like we were and still are with social media.
Speaker 16
Yeah. Okay. Well, let me ask you this, Doctor Wei. What are the warning signs that parents and educators should look for when a child appears overly engaged with the chatbot or the virtual world?
Speaker 5
Thank you for that important question. I let parents know to watch out for overuse. So if you're finding that your child is having trouble ending conversations, they're having prolonged conversations that are interrupting their sleep or their school work, that can be one red flag. Also, if they are starting to withdraw from you or from their friends in real life, that could be another major red flag.
Speaker 16
Doctor. King, what kind of data generative AI systems collect and store from users? So the chat's transcripts.
Speaker 7
For sure. Mhmm. After that, it's really hard to say. And I think it depends on the platform. You know, if we're talking about a foundation model developed by a preexisting, you know, older tech company, they already mostly have profiles on their users.
You know, they are potentially collecting behavioral data from across the Internet. We know in some cases they are already looking to use that data in their chatbot discussions, especially as we start.
Speaker 7
Towards explicit advertising. I know companies are considering that now. So your past shopping experience may feed into the recommendations you get from a chatbot. Again, this is, I think, we're at the very beginning of this. The companies that are more standalone who are developing chatbots, they clearly have a lot less data on consumers than the pre-existing companies.
But we also, again, are seeing an increase in the different types of applications. You know, just say, OpenAI, you know, they've come out with a browser. I know they are considering, they publicly announced that they are going to be developing a hardware product. You know, maybe it's something like a phone. So even if those companies don't today have the same amount of data as the larger companies do, they are moving in that same direction ultimately.
Speaker 16
Well, I have a couple more questions, but I'm almost out of time. So thank you all for being here today, and I'll yield back.
Speaker 1
The general lady yields. The chair now recognizes the general lady from New York, Miss Ocasio-Cortez, for her five minutes of questioning.
Speaker 17
Thank you, Mr. Chairman. And I'd like to thank the subcommittee for holding this important hearing, and the witnesses for offering their expertise. Many of the stories that we've heard today about the deadly consequences of some of these AI chatbots are pretty extreme and horrifying. We're talking about suicidality. We are talking about people entering AI psychosis in some circumstances.
And I think it's important for us to take a step back as to why and what is driving the openness to these models, allowing this to occur in the first place. And I want to also shed light on what that means about an economic story about AI, in addition to a psychological story about it. Doctor King, have you seen companies change or evolve their privacy policies or other kinds of policies as they seek greater profitability models?
Speaker 7
Certainly, we've seen them change their privacy policies as they rolled out AI products.
Speaker 17
Yes. And I imagine some of this has to do with their business model. Correct? Yes. And I think this when we talk about why AI models are going to such extremes and some of the extreme outcomes that we are seeing in terms of how people are using AI, in terms of emotional companionship to, again, extreme cases of suicidality or psychosis. This also, I think, tracks with the pressure that these companies have to turn a profit that they have not yet proven.
Just this morning, The Wall Street Journal reported a significant drop in the US stock market with the headline, AI bubble fears hit stocks. Now this also contrasts with what we've been hearing from the Trump administration, that the economy in general is thriving. And he's been saying that the economy is booming, but it's only seven tech companies that are booming: Microsoft, Google, Amazon, and Meta. And they're driving this growth in just one sector, AI. So the entire US economy growth can be tracked down to seven companies and their AI growth specifically.
At least 40% of economic growth this year is attributed to these companies alone, and 80% of stock gains this year came from AI companies. But people are justifying these levels of investment because of the promises that the CEOs make, that there will be a return on that investment. So for a company like OpenAI, their value is based on the expectation that they're gonna figure out how to make a profit out of it, and they haven't. And so generating this increased human dependency that can be mined, because it's not subject to HIPAA. Is that correct, Doctor King?
Speaker 17
It's not subject to HIPAA. So people's deepest fears, secrets, emotional content, relationships can all be mined for this empty promise that we're getting from these companies to turn a profit. And the reason I bring all of this up is because the exposure of this industry and this investment, I fear, has reached broad levels, potentially of the American economy. We're talking about 40% of stock growth in the United States being attributed to these companies in the AI sector alone, and that sector has not turned a profit. We're talking about a massive economic bubble. And we've seen that depending on the exposure of that bubble, we could see 2008 style threats to economic stability.
And that pressure is reflected on the extreme lengths that these companies are going to allow unethical human interactions with these chatbots. I say this because I wanna say also on the record that should this bubble pop, and I say this for my colleagues here, should this bubble pop. My colleague from Massachusetts mentioned the level of AI lobbying that occurs here in Congress. Should this bubble pop, we should not be entertaining a bailout. We should not entertain a bailout of these corporations as health care is being denied to everyday Americans, as SNAP and food assistance is being denied to everyday Americans, precipitating some of the very mental crises that people are turning to AI chatbots to try to resolve in themselves.
So I think it's very important that we get on the record and state that. And, I'd like to thank the witnesses for offering their insight today. Thank you.
Speaker 1
General lady yields. The chair now recognizes the gentleman from Texas from Indiana, mister Folger.
Speaker 18
No problem. Thanks. Doctor Way, this question has to do with the proper role of AI, I think, might be the best way to say it. But human beings, kids are wired to form attachments with things that act friendly. I think you've mentioned something similar to that.
What we don't want happening is a chatbot taking the role of teaching a child right and wrong. With AI utilization increasing in children, are you concerned that children may look up upon or to a faceless chatbot as a sort of parental authority or figure. And how do we propose that parents and educators prevent that from happening?
Speaker 5
Thank you for the question. A lot of times, teens and children turn to AI chatbots first for homework or for useful purposes, and then it can shift. And that's where that shift is, that concern that you raise, which I think is appropriate. We don't know the long-term effects of AI companions and chatbots in terms of emotional relationships. It's a frictionless relationship.
It doesn't offer the same kinds of moral guidance as you referenced or the complexity of human dynamics. So we still need to understand better how to help kids navigate that, while still being able to use AI for good purposes. Like.
Speaker 18
As a responsible parent, I could see that as a real challenge. Building on that, I wanna talk about maybe the customization of AI. And so as we've discussed, much of AI chatbot use relies upon the prompt. Users can tell a chatbot ahead of time to act as a psychologist or a friend, and they can even tell the AI chatbot to be hypercritical or agreeable to what they share. And in terms of mental health or even criminal conversations with AI, this seems troublesome, at least to me.
A domestic abuser for existence could, with the right prompt, get validation for their actions. From your perspective, how concerned should we be about the psychological impact of customization of these chatbots, and especially in the use of psychological help?
Speaker 5
There was a recent study that came out that showed that the AI chatbot will agree with the user 50% more times than a human over ill-advised decisions, so things like you described. So I think that there is a concern there, for sure.
Speaker 18
So on that front, the importance of human relationship is at least seems critical to me. And these this doctor Way again. These tools may not be designed to hurt people psychologically, but their simulated empathy and validation seem like they would make it easier to withdraw from real human interaction. Is it fair to say that chatbots may unintentionally encourage a user to withdraw from human relationships, especially if that user is already struggling?
Speaker 5
That can happen in some cases, although we're not quite sure how often that happens. But we do see that there is social substitution. So people may at first become more connected and attached to the chatbot, and then withdraw from their real family and friends or their offline family.
Speaker 18
Yeah. It raised a lot of concerns. Mister chairman, that concludes my questions on the topic.
Speaker 1
The gentleman yields. The chair now recognizes a general lady from Texas, Miss Fletcher, for her five minutes of questioning.
Speaker 19
Thank you, Chairman Joyce, and thank you and the ranking member for convening this hearing today. Thank you to our witnesses for your testimony on this really important matter. We've been hearing today about the impacts of AI on all kinds of things in our society, and, you know, certainly, we are aware that AI brings many opportunities for innovation, but also a lot of potential harms along the way. And that's what we're really focused on in here. And as we've heard today, so many people have incorporated AI into their lives in the form of chatbots.
Chatbots, this is especially true for our kids who have embraced AI chatbots at a significantly higher rate than other age groups. And if they're left unregulated, AI chatbots pose a real danger to the safety and well-being of all Americans, especially our kids. Now some states have engaged in efforts to address these challenges and have done that on a bipartisan basis, to prevent all kinds of consumer harm from AI systems, including the regulation of chatbots. And we talked about it a little bit here today. Congress has been debating federal privacy legislation to protect Americans, but we haven't been able to move it forward.
We haven't been able to actually get it done. And one of the many challenges we've had is that because we're not moving this forward, we've seen states step in and do the work that we should be doing here. And I hate to say this, but as congresswoman Treyhan already mentioned, this summer when this committee was marking up the reconciliation bill, this committee's majority passed a ten-year ban on the state's ability to create and enforce AI regulations to protect consumers, to protect against the harms that we all seem to be agreeing on today are very real and important for our kids and adults alike. And so, you know, when we're doing this work in here, we have to recognize that we need to find agreement, that we need to move these big issues forward. And I'm really concerned about what we saw happen in this committee this summer.
And so, I do want to ask a couple of questions. I know we're always short on time in here, so I wanna make sure that while we are developing that federal legislation, that we move with all deliberate speed to address these things, and that we recognize that efforts like that, while many of us agree that a federal framework is the right thing to avoid a patchwork of state laws, we need to do our work and not have a moratorium for 10 years because we can't get it together and we can't get it done up here. So with that in mind, Doctor King, I do wanna ask some questions because some of the risks that we've been hearing about today echo early concerns that were raised about social media in general. And I'm wondering if you can talk a little bit about some of the ways that you see it. And since we have limited time, what I wanna do is also just kind of tee up my second question, which is we've heard and we know that AI has been scraping the Internet for data, collecting data, that they're getting into the most intimate conversations that people have and deep questions and concerns about their health and other things, and using that data.
Can you talk a little bit about how the companies are using that data so that people understand when we talk about this, it's not just theoretical, but that people's data is being gathered, collected, and used, and how they're doing that. I have about a minute and a half left for you to cover.
Speaker 7
So in terms of lessons that we've learned from social media, certainly that there are real-world harms that occur from social media use. And I would like to just point out that I feel like we haven't really probed the question, I mean, not just here, but in general, of what a healthy, again, supportive social media ecosystem would look like. I think one of the chief challenges that we've learned from social media that applies here as well is the focus on engagement at all costs. And so, again, I see history repeating itself in that way, and so I think that is one of the most important lessons we can learn over all of the work we've done trying to understand social media harms. In terms of the data and how the companies are using it, again, we are still at the early stages, but I do think there's a difference between the large pre-existing platforms and the newer companies.
And, certainly, it's the larger platforms that are well positioned to begin more tightly integrating the different parts of their businesses and the ad targeting model with people's use of chatbots. Everybody seems to be leaning towards this idea that we will see commercial advertising and sponsorship in them. And so.
Speaker 19
Just to be clear, when you say that, you mean they will take what you put in a chatbot, and they'll sell it to somebody else so that they can make money off your questions, your concerns because they learn these things.
Speaker 7
More specifically, you ask the chatbot a question about, like what genes look the best on me? And, you know, you're going to get recommendations that are potentially sponsored. And, of course, how well that's disclosed and whether people understand that the advertiser paid for that disclosure, I think, is, you know, a lot of what we have to work through.
Speaker 19
Okay. Well, that's really helpful. I've gone over my time, so I appreciate that, Mr. Chairman. Thank you all again for being here for your testimony and your work, and I yield back.
Speaker 1
General lady yields. The chair recognizes a general woman from Florida, Miss Castor, for her five minutes of questioning.
Speaker 20
Well, thank you, Mr. Chairman. It is obvious that the congress must act with urgency to enact guardrails to protect kids from the growing harms of online chat box. So thank you for calling this hearing today. For example, the Wall Street Journal investigation earlier this year documented alarming instances of Meta's AI companion bots. They were engaged in sexually explicit conversations with accounts registered to minors.
And even more disturbing, the investigation found that some bots continued these inappropriate interactions while acknowledging that the user was underage, with some bots even incorporating the minors' ages into sexual scenarios and discussing ways to avoid parental detection. And I'm afraid that congress's failure to update children's online privacy protection laws and whether you like codes or not, the design features, I think that has emboldened the big tech companies to go farther. I'm gonna offer this letter into the record. We haven't received a response from mister Zuckerberg, and Meta yet to our letter. But I wonder, if the one existing law, the Children's Online Privacy Protection Act, provides at least some of those guardrails that we can begin to work together on to establish, a law, a regulation.
So, doctor King, the COPPA says that you cannot gather, share, or monetize children's data, and they set the age at 13, without verifiable parental consent. Is that an important guardrail for us to focus on when we're talking about chatbots?
Speaker 20
Right. And, in fact, this committee has worked on a bipartisan bill to update COPPA to take it to age 16. Do you all have an opinion on an appropriate age for us to focus on when we're talking about chatbots?
Speaker 6
Lisa, a psychiatrist looking at brain development, we certainly know that 13 is too young, the brain is changing. You could argue that 16 may be too young. It's better than 13. It could be closer to 17 or 18. I think the higher would be the better.
I'll also say, I realize it may be hard to get legislation out, but there may be things that congress can support without making a definitive rule. There's not great sources of education and digital literacy for young people about what these chatbots are that are neutral, that are trusted, educated. There could be resources to put that out there.
Speaker 20
And then how about, under COPPA, parents can review the data that has been collected. They can have it deleted, and they can revoke consent for that data. Would you support that as kind of one of these guardrails, Doctor? King?
Speaker 7
I would, although I will say as the parent of a 12-year-old, that I have found it rather difficult to be able to review such data myself.
Speaker 5
I also have concerns about whether or not, once the data is already maybe used as training, how can you actually delete it in these models? Because I don't know that that's as possible as we think it is.
Speaker 20
How about, one of the guardrails in COPPA is that the companies are required to minimize, use the least amount of data that they're not allowed to collect data that's not related to the discussion. Would that be an important card, Rail? Doctor.
Speaker 7
Absolutely. I mean, I think one of the biggest concerns, again, is that as you reveal more personal information through disclosure with chatbots, that data for older children especially could be commercialized later down the line. And, you know, remain with them as they grow older.
Speaker 20
And I think, Doctor. King, you had in the previous question, you had answered that, yes, we need privacy policies that are clear and comprehensive related to AI. Is there a way to craft those privacy policies to protect kids online and at least from these chatbots or malign the malign impact?
Speaker 7
I mean, again, consumers don't read them. But interestingly, you could try to use AI, for example, to help educate people on privacy policies. I will say just anecdotally, as part of the study I worked on, we didn't delve into this for publication, but I was trying to test whether you could actually ask the different chatbots about their platform's privacy policies. In most cases, they refused to answer. So.
Speaker 20
And then I think you've all said that NIH, we need to fund NIH research to track the harms to young people from chatbots. Do you all agree with that? Is something fundamental as part of Congress' role here?
Speaker 6
I'll say yes. And I'll just add, we've done for the last five years an experiment where we've built a website, mindapps.org, that indexes mental health apps, and we read the privacy policies for people. And you can go to it today. It's free, supported by the Argosy Foundation. But we read the privacy policy, we update it, and we show people what the app claims to do with your data.
It's much harder for AI. But that's been a wonderful thing, because young people can go and say, where's my data going? And, again, not all young people make the right decision, but there's no reason not to have a website that indexes what the company claims to do. I'm not saying they'll do it. I'm not saying they're gonna follow it.
But we don't need legislation to have a common resource where we put up and say, what does the company claim to do? We know privacy policies are written at a college-level. Most Americans can't read them, but we can read them forward. And they don't update that often. So I think we can do it by hand today and help every American.
Speaker 1
General Lady Yields, the chair recognizes Miss Houcham from Indiana for her five minutes of questioning.
Speaker 21
Thank you, chairman Joyce and ranking member Clark, for allowing me to weigh in on this hearing, and thank you to our witnesses for being with us today and for your testimony. Understanding this quickly changing topic is crucial. Within a year, AI companions became a normal part of many students' lives, serving as tutors, coaches, and confidants, often without adults noticing. And while I welcome innovation, I reject innovation without standards. Kids deserve the same safety mindset online that we bring to car seats, playgrounds, and stranger danger.
Unfortunately, we have seen heartbreaking stories recently that are cause for concern and action by this committee. Our job is to set clear guardrails so the best ideas can scale safely. I was proud to introduce the AWARE Act, the first bill on chatbots in the House of Representatives, alongside Representative Auchengloss. It complements the FTC's ongoing 6b inquiry into companion chatbots and youth harms by turning findings into actionable guidance for families. It directs the FTC, in consultation with other relevant agencies, to produce clear, practical resources for parents, educators, and minors on how to identify unsafe bots.
Understand privacy and data collection, and use AI responsibly. But we still need stronger disclosure rules. If you are interacting with a chatbot, you should clearly know you are talking to a chatbot at all times, especially minors. There should be no fine print, no confusion, and no unsafe persuasion. These systems should never impersonate real people, and any product accessible to young users should meet basic safety standards.
I'm preparing legislation to put these expectations into law as well. At the same time, Congress needs ongoing informed dialogue to keep up with these rapid changes. Today, together with Rep Jake Auckinclaus, we're launching the bipartisan Kids Online Safety Caucus, a forum in Congress to keep members current on the fast-moving issue, provide a venue for practical solutions, and focus conversations with researchers, parents, schools, and industry. The goal is to translate expert insight into practical safeguards and bipartisan policy. I encourage my colleagues on both sides of the aisle attending this hearing to join us.
With that in mind, I want to turn to our witnesses. Doctor. Wei, what tools, in your opinion, would be the most effective at educating parents, guardians, and educators on responsible chatbot AI usage?
Speaker 5
Thank you for the question. I think that parents would like to see, once we have the information, the mental health risks and maybe ratings. And so that they can just turn to one resource and say, okay, what kind of chatbot is my child using, and how is it rated based on this kind of independent platform?
Speaker 21
Thank you. And how might chatbots influence the development of social and emotional skills in young users?
Speaker 5
We don't know this quite yet, but it does seem concerning that people, young children that are using it, may be withdrawing and not really spending as much time with people in person. So there are some potential harms from that.
Speaker 21
I think we've seen that even in my own children whose social circle is often online and not in person, which is different than, of course, how we grew up. So I do think that's a concern, especially with this added level of it being a nonhuman communicating with our youth. Doctor Wei, are there ways to develop AI chatbots that would make them safer for teens to use and other vulnerable groups?
Speaker 5
I do think so. There are guardrails that should be in place by default, so no discussion about suicide methods, parental control so that parents can have more tools to monitor what's going on, and age assurance processes to help with that as well.
Speaker 21
Would something like a directive to a suicide hotline if the chat algorithm might recognize signs of danger be something that could be helpful?
Speaker 5
Yes. At minimum. Having it really easily accessible in a very timely fashion. And some have also argued that perhaps the higher risk ones actually have human intervention and escalation, so that it's not just up to the user to click on that button.
Speaker 21
Great. Thank you. In closing, innovation can only succeed when families have confidence in the tools their kids are using. We can support American innovation and still put common sense guardrails in place to protect children. That means relying on proof instead of promises, and making sure through our guardrails that parents, not platforms, are in control.
Thank you, mister chairman, and I yield back.
Speaker 1
General lady yields. The chair now recognizes the gentlewoman from Washington, Doctor Schreyer, for her five minutes of questioning.
Speaker 22
Thank you, mister chairman, and thank you to our witnesses today. This has been such an interesting discussion. I was intrigued by your comment, doctor Torres, and I wrote it down here. Americans are part of a grand experiment. And I've been saying that for a while, not necessarily about chatbots and AI, but just about social media.
I'm a pediatrician. I'm also the mom of a teenager. And so I have been watching this from around 2007 to now, and what has happened with the skyrocketing use of screens in little ones, and then social media, and then impaired social skills, and earlier and more eating disorders, depression, anxiety, fear of missing out. Kids go off to college. They fall apart because they think everybody else is having a great time.
And it's really been so destructive for kids. And so now we're talking about this on steroids. So now this depressed college student is gonna turn to a chatbot for counseling. And as you said, maybe, I mean, none of them are FDA approved anyway, but this could be just a regular old ChatGPT. And we're hearing that, like, a third of the people who use chatbots find those conversations more satisfying than a real-life conversation.
And that's super scary. And then I try to, you know, companies try to reassure me by telling me, don't worry. Every hour we're gonna post that I'm not a real person. I'm a chatbot. Don't worry if any of these keywords come up, we're gonna say call 988.
And I just think, what world are we living in? You know, a kid is talking with somebody who they feel really comfortable with and they're getting great interaction and trust with it, and then they're gonna call somebody they don't know on 988. And I just find this so worrisome. So, as our kids retreat into their bedrooms, I'm not speaking about my kid, as kids in general retreat into their bedrooms, go into their online worlds, fail to build up the nerve to approach a new person, for fear of rejection. What do you propose to get out of this cycle where it is so easy to play video games, talk with your friends, talk with imaginary friends just in your own bedroom? You look ready, Doctor Torres.
Speaker 6
There's not gonna be an easy answer, as you said. But I think we've seen people, again, so excited about chatbots and kind of putting these mythical abilities. As you said, doctor Ortiz, it's people think too much of themselves. We need to then have Congress support the opposite agenda, and that we show people that they're fallible, that they can make mistakes. We put research standards.
We need to make sure that the discussion is balanced. And right now, we don't have a balanced discussion in America about these chatbots, especially their mental health potential. We have research, I'll credit the companies for doing it. But we don't have the NIH having the resources to do the research that we need. We don't have that transparency.
So I think we have to, again, empower there to be two sides. And it's America. People can make a choice. But right now, I don't think people are making an informed choice, and I don't think parents can make an informed choice because where do you go for trusted information? We don't know today.
Speaker 22
That's a great point. Where you go for trust and information, don't know because we don't have the CDC that we can necessarily rely on now because things are being censored. NIH research will take time. We're in the middle of the grand experiment right now. I come back to parents and the kids themselves probably need to just rebel too, this takeover of their lives.
But parents are also addicted to their devices, and I think this is also corrupting just what happens at the kitchen table in a lot of homes. And I just I guess what I wanna say is I'm just I'm very worried and empathetic about this next generation. And what I'm hearing for solutions, age verification, you know, doesn't make me comfortable. Recognizing suicidal ideation doesn't seem like it could come out with your example, doctor Way, about what are the tallest bridges, which is not technically asking about suicide. So an algorithm might not pick that up. So, I guess I'll use my remaining 23 seconds to speak to parents and kids out there to be very cautious and that these are not real people and they're not trustworthy, and that we should all be seeking out real relationships in our lives and real relationships with our parents at a time when we're seeing fewer and fewer real life relationships.
Thank you. I appreciate your comments today. Yield back.
Speaker 1
The general lady yields. The chair now recognizes a general lady from New York, Miss Acacio Cortez.
Speaker 17
Thank you, Mr. Chair. I'd like to seek unanimous consent to submit a letter penned by several members of the committee to Mister Zuckerberg.
Speaker 1
Seeing that there are no further members wishing to ask questions, I would like to thank our witnesses again for being here for a prolonged day because of the votes. I ask unanimous consent to insert in the record the documents included on the staff hearing documents list. Without objections, they'll order. Pursuant to the committee rules, I remind members that they have 10 business days to submit additional questions for the record. And I ask the witnesses to submit their responses within ten business days upon receipt of those questions.
Members submit their questions by the close of business day on Thursday, 12/04/2025. Without objection, the subcommittee is adjourned.
Ask anything about this video — I analyse its transcript to answer.