I'm one of the creators of the quiz!<p>Note that the projects were <i>not</i> chosen based on their results, and certainly not because the results are counterintuitive. Rather they are the ten best understood social programs we could find.<p>You can read about the findings of our research into whether people can guess what works and what doesn't ahead of time: <a href="http://www.vox.com/2015/8/13/9148123/quiz-which-programs-work" rel="nofollow">http://www.vox.com/2015/8/13/9148123/quiz-which-programs-wor...</a>
I was thinking about why this website triggered my skeptic alarms and came up with a few things.<p>1. The web interface makes it hard to go back. The first thing I want to do when I see a surprising answer is reread the question. I understand this is likely just bad engineering, but it makes the whole site <i>feel</i> less trustworthy.<p>2. Some questions get summarized poorly. For example, the mindfulness question asks "What effect does mindfulness based stress reduction have on self-reported mental health (rates of anxiety, stress, and depression)?" but then summarizes the choices to "reduction rates of mental health issues." You can't drop a word like self-reported from a question, after all physically disabled people self-report being happier after their disability (e.g. <a href="http://www.ncbi.nlm.nih.gov/pubmed/21870935" rel="nofollow">http://www.ncbi.nlm.nih.gov/pubmed/21870935</a>).<p>Also in the "Drug Substitution Programs" question the text indicates that the research is based off of cases where "Addicts were given heroin or substitutes such as methadone or buprenorphine, based on their needs," however the choices are formed "Positive effect - Prescribing heroin to addicts reduces crime rates," [note that it dropped the <i>or substitutes</i>]. This feels like going for shock value.<p>3. At the end the website is selling very hard about some newsletter. Apparently the website seems to be focused around a career guide? If you truly have no axe to grind then present high-quality information and I'll naturally explore the site more.<p>4. If your objective is really to help raise awareness about how often media publicity for social interventions doesn't reflect efficacy as measured in journals then I would think at the end you would propose a plan of action , such as "When hearing about a social program, you can use google scholar to tap into research findings..."
This is great! One of the interesting things about social sciences (and engineering, although just like computer hacking, social engineering has unfortunately strong negative connotations) is that the real world is often counter-intuitive. In particular, people tend to believe in punishment a lot, while it commonly escalates the problem (I like the quip "emphasis on punishment is the sign of an obedience frame").<p>I am dismayed that I didn't do better than random chance, even though I like to read about social issues. I think this really shows the importance of empiricism in social sciences and engineering.
The Elderly care question said that there was no effect on 3 year mortality, which is fine and I'm sure that is accurate but it doesn't answer the question of "Did it have a net positive/negative effect on quality of life for those 3 years".<p>Not been a negative nelly just an observation that something as complex as social interaction problems can't always be summed up with a clear net gain/loss.<p>Loved the idea and implementation though.
Having conclusive evidence that a program doesn't work (or even is harmful) is significant. How effective is it for actually getting such programs stopped? Have any been?
Home visits for older adults is a little disingenuous. The purpose is not really to reduce death risk, but to increase quality of life. I don't think that counts as "no effect" just "no effect on mortality" which is unsurprising.
I got way fewer of these right than I would have hoped. It's clear that I need to update my understanding of the questions presented. I owe a debt to those who put this together. Thanks very much.
It would be nice to see a summary of the projects and their effects after finishing the quiz. By that time, I had already forgotten about some and wondered what the answers were.
I find it a bit unintuitive that "no change" errors are valued the same as "opposite" errors (e.g. answering "positive" when the change was actually negative). I find it far less interesting when the discrepancy is whether there is any change or not than when the effect is actually opposite to what one would apparently expect.<p>Additionally as others have pointed out some of the questions and answers aren't very clear (though maybe it's just my reading comprehension that's failing). I too am unhappy about the way the elderly question is posed -- it's not clear whether all of the programs were actually focussing on reducing mortality. In fact the introduction mentions that it's merely one of several goals and from anecdotal evidence I would expect that the mortality metric is thrown off by patients stabilizing once their health has deteriorated enough to require them to be placed under permanent care -- that you're less (or equally likely) to die doesn't mean you're more (or equally) healthy.
It's really hard to tell "negative effect" and "no effect" apart, I got two of them switched.<p>Besides, I would consider both of those failed programs, not even sure if there's a point in distinguishing.<p>I got 7/10 right, only one successful program I got wrong.
This may be specific to me (though I'm using the latest chrome, latest mac os, so I doubt it), but the "share on facebook" link didn't work at all for me, and the facebook "like" link took me to a page to share the post on my timeline, not just "like" it. Didn't see any easy route to report such things, so here it is!
Ben Goldacre (of Bad Science) wrote a paper with the UK Government on A/B tests of policies.<p><a href="http://www.badscience.net/2012/06/heres-a-cabinet-office-paper-i-co-authored-about-randomised-trials-of-government-policies/" rel="nofollow">http://www.badscience.net/2012/06/heres-a-cabinet-office-pap...</a>
While the content is great, the tech seems bloody awful. Why is it so slow to load? It takes 5-10 seconds to load on both my laptop and my desktop (both < 1 year old) and I have nice fast fibre. Why would you start with a loading screen in this day and age? And worse a loading screen that doesn't tell you what it's loading. Flash is dead.<p>It severely impacts the usability, it looks like nothing is happening and I can imagine a lot of people just close it before it loads.<p>And it's (virtually) static content! Why doesn't it load with the opening page at least?