Social B(eye)as over Time (SBT) Dataset

Many eyes have scrutinized the social behaviors of computer vision services, given their popularity with researchers and developers. When analyzing images depicting people, their descriptions often reflect social inequalities and stereotypes, yet the proprietary nature of these services mean that it is difficult to anticipate or explain their behaviors. Mechanisms providing oversight of these processes can enable more responsible use, allowing stakeholders to audit their behaviors and track potential changes over time. Previously, in 2019, we audited image tagging algorithms for social bias when processing images of people. In this work, we i) present data from an audit of the same services three years later, with ii) additional outputs for input images depicting other racial/ethnic groups and iii) a toolkit enabling several fully-automated analyses on the algorithms' behaviors across time.