Derm-T2IM: Harnessing Synthetic Skin Lesion Data via Stable Diffusion Models for Enhanced Skin Disease Classification using ViT and CNN

This study explores the utilization of Dermatoscopic synthetic data generated through stable diffusion models as a strategy for enhancing the robustness of machine learning model training. Synthetic data plays a pivotal role in mitigating challenges associated with limited labeled datasets, thereby facilitating more effective model training and fine-tuning. In this context, we aim to incorporate enhanced data transformation techniques by extending the recent success of few-shot learning in text-to-image latent diffusion models. The optimally tuned model is further used for rendering high-quality skin lesion synthetic data with diverse and realistic characteristics, providing a valuable supplement and diversity to the existing training data. We investigate the impact of incorporating newly generated synthetic data into the training pipeline of state-of-the-art machine learning models, assessing its effectiveness in enhancing model performance and generalization to unseen real-world data. Experimental results demonstrate the efficacy of the synthetic data rendered through stable diffusion models helps in improving the robustness and adaptability of CNN and vision transformer (ViT) models on different real-world skin cancer datasets. The dataset along with the trained model are open-sourced on our GitHub https://github.com/MAli-Farooq/Derm-T2IM.