Research Algorithmic bias detection and mitigation: Best practices and policies to reduce consumer harms
Introduction
The private and public sectors are increasingly turning to artificial intelligence (AI) systems and machine learning algorithms to automate simple and complex decision-making processes.The mass-scale digitization of data and the emerging technologies that use them are disrupting most economic sectors, including transportation, retail, advertising, and energy, and other areas. AI is also having an impact on democracy and governance as computerized systems are being deployed to improve accuracy and drive objectivity in government functions.
The availability of massive data sets has made it easy to derive new insights through computers. As a result, algorithms, which are a set of step-by-step instructions that computers follow to perform a task, have become more sophisticated and pervasive tools for automated decision-making.While algorithms are used in many contexts, we focus on computer models that make inferences from data about people, including their identities, their demographic attributes, their preferences, and their likely future behaviors, as well as the objects related to them.
In the pre-algorithm world, humans and organizations made decisions in hiring, advertising, criminal sentencing, and lending. These decisions were often governed by federal, state, and local laws that regulated the decision-making processes in terms of fairness, transparency, and equity. Today, some of these decisions are entirely made or influenced by machines whose scale and statistical rigor promise unprecedented efficiencies. Algorithms are harnessing volumes of macro- and micro-data to influence decisions affecting people in a range of tasks, from making movie recommendations to helping banks determine the creditworthiness of individuals.In machine learning, algorithms rely on multiple data sets, or training data, that specifies what the correct outputs are for some people or objects. From that training data, it then learns a model which can be applied to other people or objects and make predictions about what the correct outputs should be for them.
However, because machines can treat similarly-situated people and objects differently, research is starting to reveal some troubling examples in which the reality of algorithmic decision-making falls short of our expectations. Given this, some algorithms run the risk of replicating and even amplifying human biases, particularly those affecting protected groups.For example, automated risk assessments used by U.S. judges to determine bail and sentencing limits can generate incorrect conclusions, resulting in large cumulative effects on certain groups, like longer prison sentences or higher bails imposed on people of color.
In this example, the decision generates “bias,” a term that we define broadly as it relates to outcomes which are systematically less favorable to individuals within a particular group and where there is no relevant difference between groups that justifies such harms.Bias in algorithms can emanate from unrepresentative or incomplete training data or the reliance on flawed information that reflects historical inequalities. If left unchecked, biased algorithms can lead to decisions which can have a collective, disparate impact on certain groups of people even without the programmer’s intention to discriminate. The exploration of the intended and unintended consequences of algorithms is both necessary and timely, particularly since current public policies may not be sufficient to identify, mitigate, and remedy consumer impacts.
With algorithms appearing in a variety of applications, we argue that operators and other concerned stakeholders must be diligent in proactively addressing factors which contribute to bias. Surfacing and responding to algorithmic bias upfront can potentially avert harmful impacts to users and heavy liabilities against the operators and creators of algorithms, including computer programmers, government, and industry leaders. These actors comprise the audience for the series of mitigation proposals to be presented in this paper because they either build, license, distribute, or are tasked with regulating or legislating algorithmic decision-making to reduce discriminatory intent or effects.
Our research presents a framework for algorithmic hygiene, which identifies some specific causes of biases and employs best practices to identify and mitigate them. We also present a set of public policy recommendations, which promote the fair and ethical deployment of AI and machine learning technologies.
This paper draws upon the insight of 40 thought leaders from across academic disciplines, industry sectors, and civil society organizations who participated in one of two roundtables.Roundtable participants actively debated concepts related to algorithmic design, accountability, and fairness, as well as the technical and social trade-offs associated with various approaches to bias detection and mitigation.
Our goal is to juxtapose the issues that computer programmers and industry leaders face when developing algorithms with the concerns of policymakers and civil society groups who assess their implications. To balance the innovations of AI and machine learning algorithms with the protection of individual rights, we present a set of public policy recommendations, self-regulatory best practices, and consumer-focused strategies–all of which promote the fair and ethical deployment of these technologies.
Our public policy recommendations include the updating of nondiscrimination and civil rights laws to apply to digital practices, the use of regulatory sandboxes to foster anti-bias experimentation, and safe harbors for using sensitive information to detect and mitigate biases. We also outline a set of self-regulatory best practices, such as the development of a bias impact statement, inclusive design principles, and cross-functional work teams. Finally, we propose additional solutions focused on algorithmic literacy among users and formal feedback mechanisms to civil society groups.
The next section provides five examples of algorithms to explain the causes and sources of their biases. Later in the paper, we discuss the trade-offs between fairness and accuracy in the mitigation of algorithmic bias, followed by a robust offering of self-regulatory best practices, public policy recommendations, and consumer-driven strategies for addressing online biases. We conclude by highlighting the importance of proactively tackling the responsible and ethical use of machine learning and other automated decision-making tools.
Examples of algorithmic biases
Algorithmic bias can manifest in several ways with varying degrees of consequences for the subject group. Consider the following examples, which illustrate both a range of causes and effects that either inadvertently apply different treatment to groups or deliberately generate a disparate impact on them.
Bias in online recruitment tools
Online retailer Amazon, whose global workforce is 60 percent male and where men hold 74 percent of the company’s managerial positions, recently discontinued use of a recruiting algorithm after discovering gender bias.The data that engineers used to create the algorithm were derived from the resumes submitted to Amazon over a 10-year period, which were predominantly from white males. The algorithm was taught to recognize word patterns in the resumes, rather than relevant skill sets, and these data were benchmarked against the company’s predominantly male engineering department to determine an applicant’s fit. As a result, the AI software penalized any resume that contained the word “women’s” in the text and downgraded the resumes of women who attended women’s colleges, resulting in gender bias.
Causes of bias
Barocas and Selbst point out that bias can creep in during all phases of a project, “…whether by specifying the problem to be solved in ways that affect classes differently, failing to recognize or address statistical biases, reproducing past prejudice, or considering an insufficiently rich set of factors.”Roundtable participants focused especially on bias stemming from flaws in the data used to train the algorithms. “Flawed data is a big problem,” stated roundtable participant Lucy Vasserman from Google, “…especially for the groups that businesses are working hard to protect.” While there are many causes, we focus on two of them: historical human biases and incomplete or unrepresentative data.
Historical human biases
Historical human biases are shaped by pervasive and often deeply embedded prejudices against certain groups, which can lead to their reproduction and amplification in computer models. In the COMPAS algorithm, if African-Americans are more likely to be arrested and incarcerated in the U.S. due to historical racism, disparities in policing practices, or other inequalities within the criminal justice system, these realities will be reflected in the training data and used to make suggestions about whether a defendant should be detained. If historical biases are factored into the model, it will make the same kinds of wrong judgments that people do.
The Amazon recruitment algorithm revealed a similar trajectory when men were the benchmark for professional “fit,” resulting in female applicants and their attributes being downgraded. These historical realities often find their way into the algorithm’s development and execution, and they are exacerbated by the lack of diversity which exists within the computer and data science fields.Further, human biases can be reinforced and perpetuated without the user’s knowledge. For example, African-Americans who are primarily the target for high-interest credit card options might find themselves clicking on this type of ad without realizing that they will continue to receive such predatory online suggestions. In this and other cases, the algorithm may never accumulate counter-factual ad suggestions (e.g., lower-interest credit options) that the consumer could be eligible for and prefer. Thus, it is important for algorithm designers and operators to watch for such potential negative feedback loops that cause an algorithm to become increasingly biased over time.
Incomplete or unrepresentative training data
Insufficient training data is another cause of algorithmic bias. If the data used to train the algorithm are more representative of some groups of people than others, the predictions from the model may also be systematically worse for unrepresented or under-representative groups. For example, in Buolamwini’s facial-analysis experiments, the poor recognition of darker-skinned faces was largely due to their statistical under-representation in the training data. That is, the algorithm presumably picked up on certain facial features, such as the distance between the eyes, the shape of the eyebrows and variations in facial skin shades, as ways to detect male and female faces. However, the facial features that were more representative in the training data were not as diverse and, therefore, less reliable to distinguish between complexions, even leading to a misidentification of darker-skinned females as males.
Turner Lee has argued that it is often the lack of diversity among the programmers designing the training sample which can lead to the under-representation of a particular group or specific physical attributes. Buolamwini’s findings were due to her rigor in testing, executing, and assessing a variety of proprietary facial-analysis software in different settings, correcting for the lack of diversity in their samples.
Conversely, algorithms with too much data, or an over-representation, can skew the decision toward a particular result. Researchers at Georgetown Law School found that an estimated 117 million American adults are in facial recognition networks used by law enforcement, and that African-Americans were more likely to be singled out primarily because of their over-representation in mug-shot databases.Consequently, African-American faces had more opportunities to be falsely matched, which produced a biased effect.
Bias detection strategies
Understanding the various causes of biases is the first step in the adoption of effective algorithmic hygiene. But, how can operators of algorithms assess whether their results are, indeed, biased? Even when flaws in the training data are corrected, the results may still be problematic because context matters during the bias detection phase.First, all detection approaches should begin with careful handling of the sensitive information of users, including data that identify a person’s membership in a federally protected group (e.g., race, gender). In some cases, operators of algorithms may also worry about a person’s membership in some other group if they are also susceptible to unfair outcomes. An examples of this could be college admission officers worrying about the algorithm’s exclusion of applicants from lower-income or rural areas; these are individuals who may be not federally protected but do have susceptibility to certain harms (e.g., financial hardships).
In the former case, systemic bias against protected classes can lead to collective, disparate impacts, which may have a basis for legally cognizable harms, such as the denial of credit, online racial profiling, or massive surveillance.In the latter case, the outputs of the algorithm may produce unequal outcomes or unequal error rates for different groups, but they may not violate legal prohibitions if there was no intent to discriminate.
These problematic outcomes should lead to further discussion and awareness of how algorithms work in the handling of sensitive information, and the trade-offs around fairness and accuracy in the models.
Algorithms and sensitive information
While it is intuitively appealing to think that an algorithm can be blind to sensitive attributes, this is not always the case.Critics have pointed out that an algorithm may classify information based on online proxies for the sensitive attributes, yielding a bias against a group even without making decisions directly based on one’s membership in that group. Barocas and Selbst define online proxies as “factors used in the scoring process of an algorithm which are mere stand-ins for protected groups, such as zip code as proxies for race, or height and weight as proxies for gender.”They argue that proxies often linked to algorithms can produce both errors and discriminatory outcomes, such as instances where a zip code is used to determine digital lending decisions or one’s race triggers a disparate outcome.
RELATED NEWS
