When Dermatology AI Fails Darker Skin: Foundation-Model Embeddings Recover Diagnostic Signal on the Diverse Dermatology Images Benchmark
Riddhi Pritam Singhvi
Abstract
Background: Skin tone can affect how well dermatology AI classifiers perform. When a classifier does poorly on a subgroup, it is hard to tell whether the image itself lacks the information needed for diagnosis or whether the model fails to use the information that is there.
Methods: We tested four configurations on 656 biopsy-confirmed lesions in the Diverse Dermatology Images (DDI) dataset: native Deep Derm, a recalibrated Deep Derm, and logistic probes trained on frozen Derm Foundation and DINOv2 embeddings. All used near duplicate-grouped nested five-fold cross-validation. We measured performance by subgroup AUC, predictive information, and how many cancers Deep Derm caught at a fixed threshold.
Results: At a threshold set for 90% specificity in light skin, native Deep Derm flagged 13 of 49 light-skin cancers but caught none of the 48 dark-skin cancers. Dark-skin AUC was near chance (0.538; 95% CI, 0.453-0.626). Rescaling the scores did not help (AUC 0.510; 95% CI, 0.422-0.604). Both Derm Foundation (AUC 0.807; 95% CI, 0.727–0.881) and DINOv2 (AUC 0.786; 95% CI, 0.706–0.863) did substantially better on dark skin, raising AUC by 0.269 and 0.248 over Deep Derm. There was not enough data to tell whether predictive information was the same across skin groups.
Interpretation: Frozen foundation model embeddings could distinguish cancers from benign lesions in DDI's images, while Deep Derm's released scores carried almost no useful ranking information for dark skin. Because swapping in a foundation model also swaps out the classifier, and Deep Derm's internal features were not available, the experiment cannot pin the difference on representation alone. These results matter for auditing subgroup performance, not for deploying any of these models clinically
