Debiasing Computer Vision Models using Data Augmentation based Adversarial Techniques

Deep learning models in computer vision, such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), have been found to exhibit significant biases related to factors such as gender and ethnicity. These biases often originate from inherent imbalances in training data predominantly sourced from the internet. In this study, we aim to address gender bias in computer vision models by curating a specialized dataset that highlights gender-related disparities. Additionally, we measure dataset diversity across six datasets (FFHQ, WIKI, IMDB, LFW, UTK Faces, diverse dataset), five professions (CEO, engineer, nurse, politician, and teacher) and different query retrieval tasks using the Image Similarity Score (ISS). To reduce learned gender biases and increase data diversity, we propose adversarial data augmentation techniques that specifically target facial regions within images. These techniques, named Partial Mix (PM), that partially mixes two gendered faces in a squared pattern, and Noise Addition (NA), that adds noise to the facial region, are designed to mitigate bias. Our experimental results demonstrate increased data diversity across the six datasets and professions, along with reduction in gender bias for CNN-based models. However, these adversarial techniques were less effective in reducing bias for Vision Transformers. This discrepancy highlights the unique challenges for bias mitigation posed by ViTs. Consistent with prior research, our findings indicate that ViTs learn from a broader set of visual cues compared to CNNs. This increased sensitivity makes ViTs more prone to amplifying biases, emphasizing the need for tailored bias mitigation strategies when deploying these models in real-world applications.