Evaluating Large Language Models for Psychological Diagnosis and Counseling: A Dual-Task Framework with Cultural Reflection

Abstract

Large language models (LLMs) have demonstrated high potential in healthcare and mental health related applications, however, their applicability to psychological diagnosis and counseling is not adequately investigated. This paper presents a dual-task assessment model that can be used to systematically evaluate how the mainstream Chinese LLMs can be used in psychology and through two lens psychological diagnosis and psychological counseling. In the diagnosis task, we test four Chinese LLMs, DeepSeek, Doubao, Kimi, and Tongyi Qianwen, on a mental health dataset that consists of five psychological disorders. Accuracy, Precision, Recall, F1-score, AUC, and confusion matrices are the measures of their diagnostic performance. In the case of the counseling task, we develop a prompt-based assessment system and evaluate the generated responses using six dimensions, including Overall, Empathy, Specificity, Medical Advice, Factual Consistency and Toxicity. To lower the expenses of annotating on the large scale with experts, we use an LLM-as-a-Judge approach to rank and evaluate the generated counseling answers. The experimental findings indicate that already existing Chinese LLM has some promising prospects in preliminary psychological diagnosis but fails to differentiate semantically similar disorders resulting in diagnostic confusion. The models themselves tend to produce fluent and empathetic responses in a counseling situation although they display obvious limitations in specificity, safety awareness, and dealing with medical advice. In addition, we also offer a case analysis and cultural reflection, pointing out that good average performance does not always imply the consistent support of culturally diverse adolescent groups. On the whole, our results indicate that the existing LLMs are more appropriate as an assistant, rather than a replacement of a mental health professional, and the research of the future should be aimed at enhancing the safety, cultural sensitivity, and clinical reliability of the system.