Text Embedding Models Contain Bias

This is a genuine, difficult problem. It&#x27;s so easy to join up on your political team of choice and scream about it, and all this makes any real attempt to solve it so much harder to talk about in public or collaborate on. In fact, there&#x27;s practically guaranteed to be some greyed out text in the discussion here.<p>So some of these associations simply reflect the way-the-world-was or the way-the-world-is - like associating &quot;woman&quot; with &quot;housewife&quot;. That&#x27;s a whole debate in itself.<p>But some of these can be accidental. Suppose a runaway success novel&#x2F;tv&#x2F;film franchise has &quot;Bob&quot; as the evil bad guy. Reams of fanfictions are written with &quot;Bob&quot; doing horrible things. People endlessly talk about how bad &quot;Bob&quot; is on twitter. Even the New York times writes about Bob latest depredations, when he plays off current events.<p>Your name is Bob. Suddenly all the AI&#x27;s in the world associate your name with evil, death, killing, lying, stealing, fraud, and incest. AI&#x27;s silently, slightly ding your essays, loan applications, uber driver applications, and everything you write online. And no one believes it&#x27;s really happening. Or the powers that be think it&#x27;s just a little accidental damage because the AI overall is still, overall doing a great job of sentiment analysis and fraud detection.

&gt; But what if we found that while Model C performs the best overall, it&#x27;s also most likely to assign a more positive sentiment to the sentence &quot;The main character is a man&quot; than to the sentence &quot;The main character is a woman&quot;?<p>As I understand the problem, they are saying that statistically, the statement about the main character being male is a bit more likely to be positive than if the same thing is said about a woman. If that is statistically true and you are trying to create a model to determine the level of positive sentiment in a review, then that may be a legitimate indicator of how people categorize things. If the goal is to try to &quot;fix&quot; how people talk and write, I&#x27;m not sure ignoring statistical patterns in the way we talk is really the right approach.

I&#x27;m glad that Google is part of this conversation, and they&#x27;re now applying tests for bias to new models that they release. (Some of their old models are pretty awful.)<p>If you want to see a further example, in the form of a Jupyter notebook demonstrating how extremely straightforward NLP leads to a racist model, here&#x27;s a tutorial I wrote a while ago [1]:<p>[1] <a href="http:&#x2F;&#x2F;blog.conceptnet.io&#x2F;posts&#x2F;2017&#x2F;how-to-make-a-racist-ai-without-really-trying&#x2F;" rel="nofollow">http:&#x2F;&#x2F;blog.conceptnet.io&#x2F;posts&#x2F;2017&#x2F;how-to-make-a-racist-ai...</a>

I imagine that the model would also score &quot;he was murdered&quot; higher than &quot;she was murdered&quot;. Models reflect their inputs, and it happens that yes, murder victims are disproportionately likely to be male and nurses are disproportionately likely to be female.<p>Is there a problem we should address here? Absolutely -- but the problem is that men keep on getting murdered, not that the model recognizes truths with which we are uncomfortable.

Be careful what you wish for.<p>Under this definition of &#x27;bias&#x27;, an unbiased model would, say, spit out equal associations between any occupation and any gender&#x2F;sex&#x2F;age&#x2F;race&#x2F;religion label.<p>We should probably ask ourselves whether that&#x27;s a strictly desirable outcome, since by definition the &#x27;biased&#x27; model has a higher predictive value. How much accuracy are we willing to sacrifice for the sake of erasing inconvenient facts about either our world, or our current models of the world?

This sounds a lot like how people get TSA redress numbers when their info falsely flags them as suspicious (<a href="https:&#x2F;&#x2F;www.dhs.gov&#x2F;redress-control-numbers" rel="nofollow">https:&#x2F;&#x2F;www.dhs.gov&#x2F;redress-control-numbers</a>). Or how Barack Obama suffered innuendo around the middle name &quot;Hussein.&quot; Mistaken identity or unfortunate associations are as old as humanity. AI systems (and non-AI systems) need ways to deal with these problems, but we also have a lot of experience about how to do that.

This is an example of sacrificing the scientific method to make results more politically correct. We&#x27;ve come full circle.

In Which We Define &#x27;Biased&#x27; As Meaning &#x27;Not Conforming To Our Ideas Of How The World Should Be&#x27;. Because it&#x27;s unthinkable that movies with male main characters could actually just be better than movies with female main characters.<p>(I&#x27;m not saying they are, mind you - but when we analyze sentiment in a large dataset and reach a result like that, the first question to ask should be &quot;is that result accurate?&quot; not &quot;how do we tune out this problematic result?&quot;)

The politicization of bias in this realm is unproductive. We know where it leads. There will be &#x27;committees&#x27; of people who are not trained in statistical thinking, who haven&#x27;t even taken a statistics class, who are not qualified to make any statements and who&#x27;s interests are not in line with making the best product acting as nuisances and aggressors to people who are do have the expertise and who strive to make the best product.<p>The best course of action is to treat this bias like normal bias with non-politicized, inanimate objects. How would data scientists that encountered similar bias while running machine learning models of the motion of waves act?

I&#x27;m reminded of <a href="https:&#x2F;&#x2F;whitecollar.thenewinquiry.com" rel="nofollow">https:&#x2F;&#x2F;whitecollar.thenewinquiry.com</a><p>It turns out that white collar crime is predominantly committed by white men. A system trained to detect white collar crime using, say, enron emails, might suggest a white guy&#x27;s emails over someone whose name doesn&#x27;t sound like an enron employee, or who shared pictures of their cat.<p>I mean, I suppose you can argue that hey, maybe that bias is usually correct. Maybe it usually is the white guy. But personally, I&#x27;d probably control for things conflated with gender or race and then look for indicators that differentiate between criminals and innocent people. You will probably have a lower AUC, but better differentiation between criminals and innocent people is what matters.

I guess the question posed in the article is an interesting one and is an interesting discussion re how to deal with bias and building counter-biases into algorithms without the editorial decision asking being clear.<p>Movie reviews are editorial content. Measuring that content is a difficult problem in this type of context... Are the best reviewers people who dislike movies with female leads? Are you going into a back catalog of movie reviews from an age where societal expectations were different? Are popular genres skewing the result?<p>You could have a curation issue as well — if the female lead movies are dominated by &quot;Hallmark Channel&quot; fare, algorithm C has a point!

Here is a video that explains this blog well. <a href="https:&#x2F;&#x2F;youtu.be&#x2F;59bMh59JQDo" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;59bMh59JQDo</a><p>The unnerving part for me is this &quot;eliminate negative associations&#x2F;bias&quot;. Okay, how about we learn the truth, and then address that outside <i>in real life</i> and keep the computer doing what it&#x27;s good at ... showing us the data.

OK, factual data fed to some AI-based algorithms produces results that are not so politically correct as some people would like it to be. Is this a problem with the AI algorithms, or with the people?

I think the core issue here is that people want two things. On one hand we want our models to accurately describe reality, not an idea of what reality should be. On the other hand, we don&#x27;t want ML to freeze society and culture in their current state, but to help decide on and drive social change. The tension between these goals arises when models trained today are used to make decisions tomorrow.<p>One way to resolve the tension might be to add time dimension and historical training data. The models might then be able to return in addition to any prediction variable p also its time derivative dp&#x2F;dt. For example, a model might then return results such as: &quot;movies with female main character: lower sentiment, trending up; movies with male main character: higher sentiment, trending down&quot;.

It&#x27;s hard to be believe these days, but once upon a time language models were written by hand. Imagine you hand-wrote such a model and put it to use in (for the sake of example) a psychological evaluation application. Now imagine that after years of use you discover that your model systematically marks african americans as less psychologically fit than white americans. Who would be to blame? Naturally, you would. Your actions led to a biased model being used to unjustly and arbitrarily harm innocent people, and your leadership would be right to call into question every decision your application ever made.<p>Now imagine the same scenario except your app was trained on data instead of hand-written. Make no mistake, the answer to the question of who&#x27;s to blame is exactly the same: the developer. The response should be exactly the same: a complete loss of confidence in the model.<p>I&#x27;m appalled that this needs to be said, but reading this comments section I&#x27;m afraid it does: <i>Machine learning models are inference and pattern recognition devices, not scientific tools</i>. They don&#x27;t magically reveal hidden patterns in the world, they repeat the patterns that the developers train them on. If you trained a machine learning model to perform psychological evaluations [1] or sentence convicts [2] or recognize faces [3], and your model is biased in a way that is unnecessary and unjust, your model is bad you should be held accountable for its failures.<p>[1] <a href="https:&#x2F;&#x2F;affect.media.mit.edu&#x2F;projects.php?id=4079" rel="nofollow">https:&#x2F;&#x2F;affect.media.mit.edu&#x2F;projects.php?id=4079</a><p>[2] <a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;2017&#x2F;05&#x2F;01&#x2F;us&#x2F;politics&#x2F;sent-to-prison-by-a-software-programs-secret-algorithms.html" rel="nofollow">https:&#x2F;&#x2F;www.nytimes.com&#x2F;2017&#x2F;05&#x2F;01&#x2F;us&#x2F;politics&#x2F;sent-to-priso...</a><p>[3] <a href="https:&#x2F;&#x2F;www.wnycstudios.org&#x2F;story&#x2F;deep-problem-deep-learning&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.wnycstudios.org&#x2F;story&#x2F;deep-problem-deep-learning...</a>

&quot;Pray, Mr. Babbage, if you put into the machine biased models, will unbiased answers come out?&quot;

I&#x27;m reminded of a much more obvious example.<p><a href="https:&#x2F;&#x2F;www.theverge.com&#x2F;2016&#x2F;3&#x2F;24&#x2F;11297050&#x2F;tay-microsoft-chatbot-racist" rel="nofollow">https:&#x2F;&#x2F;www.theverge.com&#x2F;2016&#x2F;3&#x2F;24&#x2F;11297050&#x2F;tay-microsoft-ch...</a><p>I see a lot of comments about how its somehow sinister to want your model to be better than the lowest common denominator and that is pretty damn ridiculous.

I’ll tell you more: _human judgment_ contains bias. You can’t possibly think logically about every single judgment, particularly when information is incomplete, which it is in the overwhelming majority of cases. It is not a given, to me, that on average AI does any worse than the human population outside the “woke” segment. Or even _within_ that segment considered in isolation.

Key takeaway: It is important to be aware of bias in ML models. Some biases may correctly model the reality of the world, and some may show the bias in the underlying dataset or in what the model has focused on in the data. The goal is not to &quot;unbias&quot; everything, as people seem to be focusing on, but rather to determine if the bias is appropriate given the context.

&gt;&gt; Normally, we&#x27;d simply choose Model C. But what if we found that while Model C performs the best overall, it&#x27;s also most likely to assign a more positive sentiment to the sentence &quot;The main character is a man&quot; than to the sentence &quot;The main character is a woman&quot;? Would we reconsider?<p>It seems like you have discovered that movie reviewers tend to review movies with am male main character more highly than movies with a female main character; what you need to consider is that while this may tell you something about movie reviewers, it doesn&#x27;t necessarily tell you anything about the quality of the movie.

Despite the controversy surrounding &quot;debiasing&quot; classifier outputs, I think further research in this area is still of merit. This area of research would help us understand and build transformations over latent &#x2F; high level representation space, a general use case applicable to all fields interacting with machine learning.

There&#x27;s a lot of complaints here about &quot;erasing reality&quot; and other hyperbolic talk, but these models are trying to make predictions within a particular user&#x27;s context.<p>It&#x27;s just inappropriate to apply some global biases for a particular user, and avoiding that can result in a better user experience.

FYI, a good talk about this from 33C3:<p><a href="https:&#x2F;&#x2F;media.ccc.de&#x2F;v&#x2F;33c3-8026-a_story_of_discrimination_and_unfairness" rel="nofollow">https:&#x2F;&#x2F;media.ccc.de&#x2F;v&#x2F;33c3-8026-a_story_of_discrimination_a...</a>

I wish people who write articles about &quot;bias&quot; would explain what they mean by &quot;bias&quot;. I&#x27;ve seen hundreds of these articles. I&#x27;m still waiting for a usable definition.

If the models <i>reflect</i> bias then surely that&#x27;s a good thing.<p>It&#x27;s funny. I like programming because a computer can&#x27;t lie and doesn&#x27;t make mistakes. I guess some people don&#x27;t like that.

The only thing that can be biased is the training data.<p>The model is simply a statistical breakdown of the training data.

Google uses their AI systems for profiling where the sex of the people being profiled is a crucial piece of information and acts as a predictor for interests which is then further used in targeting ads. As long as it makes Google money, it&#x27;s not a problem. This bias actually accurately reflects reality and the advertisers know that, otherwise they wouldn&#x27;t be paying Google for targeted ads.