By Colleen Dorsey
There are so, so many ways that artificial intelligence and machine learning (AI/ML) can improve our quality of life. If you take any sort of deep dive into the subject, you will find amazing possibilities and/or discoveries that are already happening or are in the making. I listen to a podcast called TWIML AI hosted by Sam Charrington.[1] Recently, I learned about a tool being developed by a researcher at MIT that is using AI/ML to recognize emotion in speech in order to assist children with autism, who often have difficulty mastering this skill.[2] While this particular tool is still in development, it gives us a sense of the extent to which AI/ML can enhance our lives and the lives of those we love.
For all of the potential good it can do, AI/ML also has potential for significant harm. The harm to which this article refers is that of bias and discrimination. In addition to offensiveness from a human sensibilities and dignity standpoint, bias and discrimination also has a real impact on our nation’s economy, at least if we take the view most leaders take about what should be measured to determine the health of a nation’s economy. For instance, if we measure a nation’s GDP (gross domestic product), at least in the United States, one important contributor is consumer spending. And because we are such a consumer-driven society, there is an impact on spending when consumers experience bias and discrimination. If certain categories of citizens (who are, incidentally, poised to represent most of the population in the United States by 2045[3]) are discriminated against with respect to their ability to get approved for loans, our consumer-driven economy suffers. If these same groups are discriminated against with respect to hiring decisions, our consumer-driven economy suffers. Unemployed and underemployed consumers spend less.
It might be useful at this point to explain what AI/ML is and how it’s created. The Department of Defense (DoD) recently published, through the Defense Innovation Board, recommendations on the ethical use of AI by the DoD.[4] The DoD’s definition of AI is broad, but good, considering the many ways to define AI/ML. The DoD defines AI as “a variety of information-processing techniques and technologies used to perform a goal-oriented task and the means to reason in the pursuit of that task.” The reasoning portion of the task is the machine-learning part. Data is inputted into a computer, along with corresponding outputs. The inputs and outputs are called training data. This training data creates the algorithm that matches the inputs to the outputs and is used by the computer to teach itself and improve on a specific task without being explicitly programmed to do so. For example, a computer program designed to recognize a dog over another animal would be trained by inputting thousands of pictures of dogs. Once the results of that training are complete, we have a model. We can then put new inputs into the model to get the outputs that are the computer’s best guess at whether the image is a dog or not.
This is a very simple example; so, to further understand how advanced AI has become, it is useful to have a basic understanding of linear algorithms. A linear search is the most basic of searching algorithms. Think about searching through a paper phone book, one name at a time, until you find the name you are looking for. This is an example of a linear search. Linear searches have been used for decades. Developers have now created much more sophisticated algorithms that mimic the human brain, called artificial neural networks (ANNs). ANNs attempt to imitate how the human brain processes information. The brain can take multiple inputs and further process them to infer hidden, as well as complex, nonlinear relationships. The brain is also capable of learning from its past mistakes. ANNs attempt to duplicate how this brain neuron system functions. The neurons are created artificially on a computer. Connecting many such artificial neurons creates an artificial neural network.
With a general understanding of what AI is and how it attempts to mimic the human brain, it is a bit easier to see how far-reaching it can be in our everyday lives. Consider an algorithm built to determine whether a customer might default on a loan. If the only training data being inputted into the system to teach the system what a low-risk loan looks like is data on loans made to white males, the system may unfairly flag loans to other races and genders as high risk. Or how about whether a male or a female will be a better hire? If the training data being inputted to teach the system what a good hire looks like consists only of resumes of men, the system will spit out resumes of women as not being a good hire.
By now, I hope you have a feel for the importance of the training data and how biased, wrong, or skewed information can result in unintended consequences. Just as (in most instances), people who get caught up in corporate scandals do not set out intending to break the law or perform unethical acts, so is it that developers and others involved in creating AI/ML systems are not consciously seeking to create biased or discriminating systems. This lack of awareness makes education on the subject that much more important. “I did not intend for that to happen” isn’t going to help if your program and product discriminate against a whole segment of the population.
The following examples of inadvertent bias built into AI/ML systems illustrate the importance of open dialogue on these issues. Google created an AI system called BERT. The intent of BERT was to teach computer systems “the vagaries of language in general ways and then apply what they learned to a variety of specific tasks.”[5] The end uses of this technology ranged from completing sentences and automatically analyzing contracts to assisting our beloved digital assistants like Alexa. The data BERT learns from includes decades of biases. Data inputted into the systems include books, Wikipedia entries, and news articles. These sources, it turns out, are filled with bias. For instance, BERT is more likely to associate men with computer programming and generally does not give women enough credit.[6] Cade Metz wrote about the issues with BERT in an article in the New York Times. He describes a computer scientist, Robert Munro, who one afternoon fed 100 English words into BERT: jewelry, baby, horses, house, money, action.[7] “In 99 cases out of 100, BERT associated the words with men rather than women. The word mom was an outlier.[8]Munro also examined other language applications being offered by cloud-computing services from Google and Amazon Web Services. He found that neither of these services recognized “hers” as a pronoun though they did correctly identify “his.” When alerted to these issues, Google and Amazon Web Services stated they were aware of the problem and working on resolving it.[9]
Another example involves a program developed for a large university hospital system to address health concerns before they become chronic.[10] The program rated patients on a risk scale to determine if they qualified for a VIP-type care coordination program. The idea was if high-risk patients received certain care and health attention, it would help to head off chronic conditions before they occurred—or got worse. A group of black patients and white patients were assessed a risk score, and, if they hit the appropriate level, they were put into this care coordination program.[11] A couple of researchers from Berkeley, in reviewing the health conditions of those with the same risk scores, noted that the white patients were getting better but that the black patients were actually getting sicker. After delving into the situation a bit more, the researchers found that one of the correlations used to determine whether a patient (white or black) would get access to the program was the number of federal health dollars spent on them on an annual basis. In digging even further, they discovered that, on average, the black patients had approximately $1,800 less in health care dollars spent on them per year than their white counterparts.[12] Because of this, they were not making the cut for this elite care coordination program. Let that sink in: fewer federally available dollars are allocated to black patients each year. And then think about how much money, in terms of health care costs, $1,800 represents. Perhaps it was an MRI they were refused? Perhaps they were told they did not need X rays?
The data or training set that the programmers used was pulling on historically cemented bias—the federal dollars allocated to white patients vs. black patients. The researchers went back and looked instead at the health of the patients as measured by “avoidable costs” such as emergency visits and hospitalizations and flare ups of chronic conditions over the year.[13] After experimenting with a model that combined both health and cost prediction, they were able to reduce the bias by 84 percent.[14] What does this mean? It means that a lot of biased data is out there and that it starts with the fact that people are biased.
This shouldn’t really come as a surprise. After all, we wouldn’t need civil rights laws if we were, as a world society, great at treating each other equitably. Bias has existed for a long, long time and is not likely to go away in its entirety any time soon. If we all agree this is the case, then it’s not a great leap to see that the data sets we use to train computers are likely to be polluted with some level of discriminatory bent. Perhaps we should build into the AI/ML product development processes a presumptionthat the data is tainted? Cross-functional teams including engineers, product developers, marketers, sales, human resources, legal and compliance and ethics folks could be created and required to insert pause points into the production phases wherein they brainstorm every possible way that the data could be skewed against a protected class.
Because of this potential for (or certainty of, depending upon how we look at it) skewed data, it’s also very important for teams to understand and fully articulate in writing the purpose or intended use of the system being contemplated or created. Wrong goals or unarticulated uses can lead to unintended consequences. Some companies will push back that implementing these sorts of processes and discussions around AI/ML creation is not practical and that businesses just don’t have the time to do it, especially as it relates to technology, which moves at warp speed. But what are meetings for if not to debate why a project should be implemented? Isn’t this a process already baked into our corporate culture? This is certainly what business leaders and boards do all the time. Putting these decisions into a documented process whereby they can show potential complainants the rigor their organization put into the decision-making processes is never a bad use of time.
Colleen Dorsey
Ms. Dorsey joined the University of St. Thomas School of Law in 2015. She previously spent nearly 16 years as in-house counsel for Land O’Lakes, where she developed and implemented the company’s first compliance and ethics program. At St. Thomas, she is charged with managing the Organizational Ethics and Compliance programs’ visibility and growth, and supporting student recruitment, advising and employment.
[1] Podcast, TWIML AI with Sam Charrington, episode 312 dated October 28, 2019, entitled “Using AI to Diagnose and Treat Neurological Disorders with Archana Venkatarama.”
[2] Id.
[3] William H. Frey, “The US will become ‘minority white’ in 2045, Census projects,” March 14, 2018, The Avenue, The Brookings Institute; https://www.brookings.edu/blog/the-avenue/2018/03/14/the-us-will-become-minority-white-in-2045-census-projects/.
[4] AI Principles: Recommendations on the Ethical Use of Artificial Intelligence by the Department of Defense,
Defense Innovation Board; https://media.defense.gov/2019/Oct/31/2002204458/-1/-1/0/DIB_AI_PRINCIPLES_PRIMARY_DOCUMENT.PDF
[5] Cade Metz, “Finally, a Machine That Can Finish Your Sentence,” New York Times, November 18, 2018.
[6] Id.
[7] Id.
[8] Id.
[9] Id.
[10] Amina Khan, “When computers make biased health decisions, black patients pay the price, study says,” October 24, 2019; https://www.latimes.com/science/story/2019-10-24/computer-algorithm-fuels-racial-bias-in-us-healthcare
[11] Id.
[12] Id.
[13] Id.
[14] Id.