> Employees at the center referred her to the online system. Uncomfortable with the technology, she asked for help with the online forms and was refused.<p>Seems like it was humans that failed her. I'm not sure what algorithms have to do with this.
I take from this that there needs to be some flexibility in how the results of the algorithms are applied. I also find the first example in the article unconvincing as a real mistake. It states that Robert McDaniel had:<p>> a misdemeanor conviction and several arrests on a variety of offenses—drug possession, gambling, domestic violence<p>then it seems like it's calling the algorithms a mistake that<p>> branded Robert McDaniel a likely criminal<p>Maybe I"m just sheltered, but a history of arrests, drug possession, and domestic violence tell me that the person is probably a criminal (though whether that rises to the level of bring one of Chicago's top 420 criminals I can't say).
How does the number of people negatively affected by algorithm compare to the number of people who would be negatively affected by human processor?<p>I mean, someone could probably write hundreds of similar articles about negative interactions with callous or incompetent human officials. Having dealt with at least an average number of DMV type officials over the years, I can't see that machines could do a whole lot worse.<p>I do agree with several points of the article though. Let the algorithms be open to public critique. This is democracy and it should lead to improvement (eventually). And of course there should always be recourse to human intervention.
I'm not sure if most folks really understand the nightmare we're setting ourselves up for. It's the domestic policy equivalent of drone warfare.<p>The western legal system was built and functions inherently on the precondition that it's people who use, administer, and maintain it. There's a lot of slack and human interpretation built into the process, and no laws are constructed such that they are enforced in a mechanical fashion. In addition, there's the premise that the folks doing the work of enforcing the laws are virtually the same as those being policed. Finally, severely unjust or unpopular laws are many times ignored by both the population and the enforcers.<p>All of that goes away with machine application of criminal/administrative law. The system was not built for this.
"Decision-making algorithms are politics played out at a distance, generating a troubling amount of emotional remove."<p>This is absolutely key. Adding distance[1] between the point where a decision is made and where the consequences of that decision are realized make it harder for any feedback from those consequences to affect the person making the decision. This makes the decisions worse (from lack of information) <i>and</i> the implementation worse (error must be much larger before the feedback from that error reaches the decision maker).<p>You see this effect in many areas. An obvious example is the law enforcement mentioned the article (or military), where "just following orders" to the modern variant of "just following an algorithm" end up causing problems.<p>A more interesting example might be the existence of the derivatives market and the invention of increasingly-exotic financial instruments. A bank giving someone a loan has some fairly well-known possible behaviors, and is (probably) close enough to allow feedback between the parties for things like capitalism to work (if you don't like the bank's behavior, you let them know that isn't acceptable by refinancing at a different bank). On the other hand, bad decisions bundled up and hidden in collateralized debt obligations sheltered these bad decisions until the problem blew up and introduced the world to the phrase "too big to fail".<p>A very interesting discussion of this problem - focused on how this kind of distance relates to human <i>honesty</i> (and rationalization) - is this RSA Animate featuring Dan Ariely: <a href="https://www.youtube.com/watch?v=XBmJay_qdNc" rel="nofollow">https://www.youtube.com/watch?v=XBmJay_qdNc</a><p>[1] measured in either number-of-hops or time
Using algorithms for support decision making and and putting a bad UI barrier between the users and the managers are two different things.<p>Anyway, public administrations should indeed make publish how their algorithms work in order to ensure they are reflecting the official policies.
This is not a problem of "algorithms" but rather of stupid policies. A programming manager at Google would have been fired if he had put such obvious errors in PageRank (or whatever they call it these days).<p>Algorithms and data can only improve effectiveness of these systems and agencies. However, their use has been combined with drastic funding cuts. These cuts and the resulting malfunctions aren't exactly a fundamental problem with data science.
I've been called as asked to take a survey about my interactions with an employee at a company I do business with. I could tell that the survey as constructed would not capture my actual concerns with the business processes and would instead reflect poorly on the employee that I did business with. The failure of the system would end up being used to mark the employee down even though he did a good job with the constrains he had.<p>It's unfortunate that these sorts of automated processes are ending up targeting edge cases, like things that should be covered by the ADA.
In the CS community a new (sub)subfield has emerged called "Fairness, Accountability, and Transparency in Machine Learning" (FATML). It's a young research topic, but I find it quite interesting.<p><a href="http://www.fatml.org/" rel="nofollow">http://www.fatml.org/</a>
This seems to make the false premise that you need computers to make decisions algorithmicly. If someone writes out a set of hard rules as to who can apply for a welfare program, the result will be the same if a human or machine makes the determination.<p>Long before computers existed, people complained about "rigid bureaucracy", which is effectively a complaint that government or business employees stuck to a process (an algorithm) that had some problems.
I have sat across from a call centre in a government office and listened to the workers running through the written version of algorithms. I've spoken to people who worked there and heard how crushing it was to know that someone was getting screwed but to be totally unable to do anything about it because the policy dictated their reaction. And I've worked with charities and listened to the other end of those phone calls; people screaming that their kids are going to be taken away because their benefits have been delayed and they can't afford to feed them.<p>The underlying assumption of this piece seems to be that turning decision making over to algorithms reduces positive discretion. But the humans in these situations frequently have no more discretion than the machine does, and inefficiency also has a human cost. It seems false to me to pretend that what these algorithms are doing, at least in terms of the majority of their immediate effect, is qualitatively different.<p>What you're losing when you encode something as an algorithm is the insight that you get from having humans in the loop. Intuition; the things that people haven't thought to measure yet. That's the weakness in any statistical technique - you need a human to lend numbers relevance; to say what is important to know the relationships of; otherwise they're just a sequence of events.<p>But you need to start off with a system that leverages human strengths in order for that criticism to make sense. Human judgement only has an advantage in a system designed to use the different sorts of value that it offers. If your call centre worker is not truly responsible for the outcome of the call, and if you don't regularly attempt to get feedback from them to inform policy decisions, then it makes no difference if they are replaced by a machine. They were being treated as one to begin with, and the value that they added to the organisation by virtue of being human; of having professional judgement; was being thrown away anyway.<p>All this does, in a lot of cases, is make existing flaws more obvious.<p>The exception I can think of to this is the criminal justice system, where there are examples of positive discretion. However, there are also examples of negative discretion there. There are many stupid laws on the books, and selectively enforcing those laws allows you to screw, more or less, whoever you want. It's not surprising that a system that would mechanically implement those laws would produce undesirable outputs, it's just that it's finally being applied to people who have the power to say something about it, (and, perhaps, have their concerns taken seriously enough to alter policy.)<p>For all that there is a loss in the case of the criminal justice system, there is also a gain: Encoding something as an algorithm makes the flaws in the process more apparent.
This reminds me of the terrifying epistolary short story Computers Don't Argue [1] by Gordon R. Dickson<p>[1] online here: <a href="http://www.dave.rainey.net/calendars/dystopias/process3.html" rel="nofollow">http://www.dave.rainey.net/calendars/dystopias/process3.html</a>
See also: <a href="http://en.wikipedia.org/wiki/Therac-25" rel="nofollow">http://en.wikipedia.org/wiki/Therac-25</a>
compare and contrast this with Tim O’Reilly essay proposing Algorithmic Regulation <a href="http://beyondtransparency.org/chapters/part-5/open-data-and-algorithmic-regulation/" rel="nofollow">http://beyondtransparency.org/chapters/part-5/open-data-and-...</a>