Pre-Trained Multimodal Large Language Model Enhances Dermatological Diagnosis Using Skingpt-4
Author Type(s)
Student
Document Type
Article
Publication Date
12-1-2024
DOI
10.1038/s41467-024-50043-3
Journal Title
Nature Communications
Disciplines
Medicine and Health Sciences
Abstract
Large language models (LLMs) are seen to have tremendous potential in advancing medical diagnosis recently, particularly in dermatological diagnosis, which is a very important task as skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases. Here we present SkinGPT-4, which is an interactive dermatology diagnostic system based on multimodal large language models. We have aligned a pre-trained vision transformer with an LLM named Llama-2-13b-chat by collecting an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors’ notes, and designing a two-step training strategy. We have quantitatively evaluated SkinGPT-4 on 150 real-life cases with board-certified dermatologists. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identify the characteristics and categories of the skin conditions, perform in-depth analysis, and provide interactive treatment recommendations.
Recommended Citation
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., & Gao, X. (2024). Pre-Trained Multimodal Large Language Model Enhances Dermatological Diagnosis Using Skingpt-4. Nature Communications, 15 (1). https://doi.org/10.1038/s41467-024-50043-3
