Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models

Indre Zliobaite*, Bart Custers

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

31 Citations (Scopus)

Abstract

Increasing numbers of decisions about everyday life are made using algorithms. By algorithms we mean predictive models (decision rules) captured from historical data using data mining. Such models often decide prices we pay, select ads we see and news we read online, match job descriptions and candidate CVs, decide who gets a loan, who goes through an extra airport security check, or who gets released on parole. Yet growing evidence suggests that decision making by algorithms may discriminate people, even if the computing process is fair and well-intentioned. This happens due to biased or non-representative learning data in combination with inadvertent modeling procedures. From the regulatory perspective there are two tendencies in relation to this issue: (1) to ensure that data-driven decision making is not discriminatory, and (2) to restrict overall collecting and storing of private data to a necessary minimum. This paper shows that from the computing perspective these two goals are contradictory. We demonstrate empirically and theoretically with standard regression models that in order to make sure that decision models are non-discriminatory, for instance, with respect to race, the sensitive racial information needs to be used in the model building process. Of course, after the model is ready, race should not be required as an input variable for decision making. From the regulatory perspective this has an important implication: collecting sensitive personal data is necessary in order to guarantee fairness of algorithms, and law making needs to find sensible ways to allow using such data in the modeling process.

Original languageEnglish
Pages (from-to)183-201
Number of pages19
JournalARTIFICIAL INTELLIGENCE AND LAW
Volume24
Issue number2
DOIs
Publication statusPublished - Jun 2016
MoE publication typeA1 Journal article-refereed

Keywords

  • Data mining
  • Fairness
  • Non-discrimination
  • Personal data
  • Regression
  • Sensitive data

Fingerprint Dive into the research topics of 'Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models'. Together they form a unique fingerprint.

Cite this