AI Chatbots
AI chatbots can conduct basic therapy sessions but struggle to adapt to individual needs
A recent study published in the journal Computers in Human Behavior: Artificial Humans suggests that artificial intelligence chatbots can conduct basic therapeutic conversations. According to the study, a language model performed similarly to human practitioners in high-quality research settings, but its ability to adapt specialized psychological methods to individual needs remains inconsistent.
Study Details
The study involved 65 university students who were experiencing mild to moderate psychological distress. Each participant completed a single 30-minute in-person session with a locally hosted artificial intelligence chatbot, exchanging an average of 49 messages. The chatbot was programmed to deliver Cognitive Behavioral Therapy, a widely used, problem-oriented treatment that helps individuals identify and change unhelpful thoughts and behaviors.
The researchers found that the chatbot's overall competence score fell slightly below the generally accepted threshold for adequate clinical performance. The standard rating scale defines an acceptable level of competence as a score of 40 out of a possible 66 points. The chatbot achieved this minimum threshold in 30 of the 65 sessions, showing a high degree of variability from one conversation to the next.
Comparison to Human Practitioners
The researchers compared the chatbot's performance to an established baseline of human competence using a meta-analysis of 18 prior studies. The results showed that the chatbot scored somewhat lower overall than human practitioners, with an average score of 38.1 points compared to 40.3 points for human professionals. However, the performance gap disappeared when the researchers looked only at the most rigorously conducted human studies.
Limitations and Future Directions
The study's findings do not indicate that artificial intelligence is ready to replace human practitioners. The study evaluated the chatbot based on a single session with young adults experiencing only mild distress, and clinical populations often present more complex challenges. Additionally, the study faced challenges in consistently rating the chatbot's text-based transcripts, and raters knew they were evaluating an artificial intelligence, which may have influenced their scoring.
According to study author Arthur Bran Herbener, the chatbot's performance highlights the difference between following a predetermined structure and tailoring a method to a specific person. Herbener explained that good observable skill in delivering therapy is not the same as clinical effectiveness, and that more comprehensive assessments of LLM-chatbots' therapeutic competence may require entirely new assessment approaches. The study was reported by PsyPost, and its findings suggest that while AI chatbots show promise in delivering therapy, there are still challenges to be addressed in adapting to different individuals and ensuring consistently competent care. The chatbot's ability to challenge clients' beliefs is often considered important for fostering clinical change, and understanding this is crucial for the development of effective AI-powered therapy systems.
