Who Can be Your AI Doctor?: Evaluation for Disease diagnosis on Large Language Models

Citations

SCOPUS

4

초록

Large Language Models (LLMs) have demonstrated outstanding performance in general domain knowledge inference such as arithmetic reasoning, commonsense reasoning, and open-domain question answering. Traditional LLMs were mainly developed and serviced by big tech companies as commercial, making it nearly impossible for researchers to access parameters. However, the recent emergence of non-commercial LLMs has led to ongoing research efforts to surpass commercial LLMs in specific areas using task-specific or domain-specific LLMs. In this study, among many tasks or domains, we focus on the high-level knowledge-demanding medical domain, particularly Automatic Diagnosis System (ADS). We evaluated whether current representative LLMs can perform disease diagnosis effectively and whether they have the potential to assist doctors in clinical situations. Moreover, we also observe the overall quality of responses to whether each LLM can accurately diagnose based on understanding the patient's basic information such as age, sex, underlying diseases, and family history, and explore whether non-commercial LLMs are likely to outperform commercial LLMs in ADS. © 2023 IEEE.

키워드

Automatic diagnosis system; large language models; medical domain knowledge
제목
Who Can be Your AI Doctor?: Evaluation for Disease diagnosis on Large Language Models
저자
Kim, Jonghyeon; Ju, Chan-Yang; Lee, Dong-Ho
DOI
10.1109/ICTC58733.2023.10392305
발행일
2023-10
유형
Conference Paper
저널명
International Conference on ICT Convergence
페이지
154 ~ 159