> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.)<p>This coupled with people acting on its predictions is a kind of self fulfilling prophecy.<p>which is to ask, are AI safety folks building models of this pattern? :)
If this is already such a problem even in the professional discipline and vocation whose <i>sine qua non</i> is the accurate analysis of physical reality, I'm really nervous about the next few years. And I was nervous already...
In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)
People treating tools like they're infallible has been a problem since computers were invented, but IMHO the biggest difference with AI is how confident and convincing it can be in its output. Much like others here, I already have had to convince, very carefully, many otherwise-decently-intelligent people who believed ChatGPT was correct.<p>Thus I think the biggest success of AI will be the arts, where imprecision is not fatal, and hallucinations turn into entertainment instead of "truths".
AI is a tool … a fool with a tool is still a fool … For natural sciences, there is no need to worry since nature would provide the ultimate check … for social “sciences”, it is entirely a different story.
Wow I came into this article angry, idk if their book title accurately conveys the sober, expert analysis it contains! In case anyone else is curious why they’re talking about “leakage” in the first place instead of the existing term “model bias”, here’s the paper they cite in the “compelling evidence” paper that started these two’s saga with the snake oil salesmen: <a href="https://www.cs.umb.edu/~ding/history/470_670_fall_2011/papers/cs670_Tran_PreferredPaper_LeakingInDataMining.pdf" rel="nofollow">https://www.cs.umb.edu/~ding/history/470_670_fall_2011/paper...</a><p>Crux passage:<p>> Our focus here is on leakage, which is a specific form of illegitimacy that is an intrinsic <i>property of the observational inputs</i> of a model. This form of illegitimacy remains partly abstract, but could be further defined as follows: Let <i>u</i> be some random variable. We say a second random variable <i>v</i> is <i>u</i>-legitimate if <i>v</i> is observable to the client for the purpose of inferring <i>u</i>. In this case we write <i>v € legit{u}</i>.<p>> A fully concrete meaning of legitimacy is built-in to any specific inference problem. The trivial legitimacy rule, going back to the first example of leakage given in Section 1, is that <i>the target itself must never be used for inference:</i><p>> (1) <i>y !€ legit{y}</i><p>So ultimately this all about bad experimental discipline re: training and test data, in an abstract way? I’ve been staring at this paper for way too long trying to figure out what exactly each “target” is and how it leaks, but I hope that engineering-translation is close
Scientists have been obsessed with over-optimzing for FOMO for the past decade - what papers should I read that I don't have time for, what grants should I apply for that I don't know about, what projects should I work on that will give me the best ROI, who in my field is poised to disrupt or make a big leap, etc.<p>Some even think that the end goal is actually an autonomous research agent that can make decisions about what questions to ask and why, and that's one of the true marks of AGI. That to me is insane and misses the entire point of science altogether, even once we reach that technical feasibility. We ask questions about the universe to expand our human relationship with the universe, not to just amass more research capital for the sake of it. And the fact that the AI snake oil has infected big chunks of science reveals which parts of it are just gold rush speculation and which aren't.<p>There's a more fundamental challenge of training scientists to understand why we ask the questions we ask. You can't just offload that to some background task and trust that it makes sense.
I would have thought scientists weren’t going to use these tools to do research considering they as a group are far more exposed to things like peer reviews and critical thinking than general society.<p>What worries me the most about these AI solutions, however, is their usage in the public sector. They can certainly be useful helpers, like, they can scan images for cancer and if added to existing processes involving humans, often lead to enhanced results. They can’t replace any existing methods, however, as we learned here in Denmark a few years ago. Unfortunately that lesson hasn’t been learned across the public sector. I think medicine and healthcare learned it, but right now, we’re replacing actual human controls, audits and sometimes decision making with AI or an unwarranted trust in AI results. Which is going to lead to some really terrible results considering how bad things like LLMs often are at being lucky in even “common knowledge” situations. It’s further enhanced by how some of the work it’s tasked to do isn’t as black-and-white as writing code is. We use AI tools in our daily work, and they are ok, but as anyone who’s used them for programming probably knows by now, they aren’t exactly great at being lucky. Sometimes they’ll hallucinate solutions that simply do not exist.<p>This is how they work, and as I said earlier, AIs can be great enhancers. They aren’t replacements though, and if we start treating them like they are, which is very tempting from a change-management and benefit-realisation perspective, we’re just going to get in trouble. This is unfortunately exactly what we’re doing, and why wouldn’t we? Most western public sectors have functioned on at least some form of new public management for two decades by now, sometimes longer. As a result the entire systemic culture is geared toward efficiency and cost reduction, even when it doesn’t really result in either efficiency and cost reduction on a broader perspective.<p>Now, if scientists are on board. Then what hope does a public bureaucracy have?
LLMs are basically Dissociated Press, but with deeper layers of statistics for a better function approximation than a simple Markov chain. It's really doing the same thing though: pick the next sequence of characters that best follows the foregoing characters.<p>Not something I'd trust as a "source of truth". Maybe a neat idea generator. And some of the deep learning algorithms can identify patterns that humans might miss -- patterns that could reveal useful insight. But they're not <i>doing</i> the knowledge work.
...but why wouldn't they use AI as an oracle? From an outsider's perspective, it seems that there's already plenty of incentive to test the margins of acceptable academic practice in order to produce more papers or publish more quickly. Sadly I feel like it'll become the norm to have a chatbot interpret your results and write your paper rather than using those expensive grad students.<p>I don't have answers; just the lingering question "why are we building this?"
Maybe they should also call it "curve fitting" instead of "AI" so they don't need to call a "poor fit" a "hallucination"
But surely if it's artificial intelligence then it'd know its limits and would respond appropriately? Oracle use no problem?<p>It is it because it's actually shit but it's the best thing we've seen yet and everyone is just in denial?
Not just scientists, but everyone!<p>My partner recently went a bit nuts writing an article with the help of GPT4. She was very proud of how productive she'd been until I asked if she'd actually searched for the papers GPT4 had referred to.<p>Of course, many of the referred to papers didn't exist...