Cool post. I thought this was the most interesting point:<p>"As an outside agency, we can’t (and shouldn’t) make this decision for Zidisha. However, it IS our job to inform them well enough to make it. The ROC curve is abstract and hard to interpret for this purpose, so we translated its information into a plot that directly measures the trade-off at every possible threshold value."<p>So many of the standard techniques in machine learning assume an objective function that's different from what the end-user actually cares about. Good practitioners like Everett know to translate between the algorithm's loss function (e.g., squared error) and the end goal. I'm surprised there's not more research to let ML algorithms optimize the ultimate loss function directly!
I'm the founder of Zidisha, and I can't thank the Bayes Impact team enough for this project. Like most small nonprofits, we don't have the resources to hire in-house data scientists. Bayes Impact brought data science expertise within our reach for the first time, and they've had a transformative impact on Zidisha.
Great job with Bayes Impact and thanks for the interesting post.<p>"So moving from a 20% fraud reduction to 50% will block about 200 additional fraudulent loans and also 200 honest loans. Is that okay?"<p>From the accompanying chart it looks more like ~700 additional honest loans blocked?<p>Any information about which threshold they picked? Looks like a difficult decision for a non-profit, from the chart it looks like about 15% of applications have been fraudsters but model accuracy is obviously limited given the available features.
I'm a fan of what Bayes Impact, but the article opening seems really naive about what the financial situation is for a pretty big chunk of US residents.<p>>> <i>As a Westerner, getting a credit card is only slightly more complicated than tying my shoes. My world is raining with opportunities to borrow money to go to school, open a store, consolidate loans, or buy an iPhone 6.</i><p>A little under 10% in the US are without a bank. And the "Underbanked", people have some type of banking but still use Money Orders and stuff like Payday loans, is over 20%. I wouldn't say "Raining with opportunities" lines up too closely with those people.<p>Like I said, it's great to see what Bayes Impact is trying to help, but just needed to clarify a bit about the US.<p><a href="https://www.fdic.gov/householdsurvey/2012_unbankedreport.pdf" rel="nofollow">https://www.fdic.gov/householdsurvey/2012_unbankedreport.pdf</a><p>Edit: Updated to clarify that this was directed to the tone of the article opening, not the author. I had previously used the word "author" instead of "article"