Pre-Trained Multimodal Large Language Model Enhances Dermatological Diagnosis Using Skingpt-4
Author Type(s)
Student
Document Type
Article
Publication Date
7-5-2024
DOI
10.1038/s41467-024-50043-3
Journal Title
Nature Communications
Abstract
Large language models (LLMs) are seen to have tremendous potential in advancing medical diagnosis recently, particularly in dermatological diagnosis, which is a very important task as skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases. Here we present SkinGPT-4, which is an interactive dermatology diagnostic system based on multimodal large language models. We have aligned a pre-trained vision transformer with an LLM named Llama-2-13b-chat by collecting an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes, and designing a two-step training strategy. We have quantitatively evaluated SkinGPT-4 on 150 real-life cases with board-certified dermatologists. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identify the characteristics and categories of the skin conditions, perform in-depth analysis, and provide interactive treatment recommendations.
Recommended Citation
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., & Gao, X. (2024). Pre-Trained Multimodal Large Language Model Enhances Dermatological Diagnosis Using Skingpt-4. Nature Communications, 15 (1), 5649-5649. https://doi.org/10.1038/s41467-024-50043-3
