GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

I don&#x27;t think I see enough discussion about what this means for privacy. There was some protection in the fact that it was prohibitively expensive to get someone to listen to every single one of our phonecalls&#x2F;read all our emails&#x2F;etc.<p>Worrying that this will no longer be the case.

So, uh, GPT-4 outperforms at labeling. What is that labeling used for?<p>&quot;Employing Surge AI&#x27;s top-tier human annotators at a rate of $25 per hour would have cost $500,000 for 20,000 hours of work, an excessive amount to invest in the research endeavor. Surge AI is a venture-backed startup that performs the human labeling for numerous AI companies including OpenAI, Meta, and Anthropic.&quot;<p>What could go wrong? Using GPT-4 to perform labeling used by OpenAI in order to train...uh, wait.

We need new political arrangements to distribute the gains of AI or things are going to get very bad very quickly.

Great to see this tech and the money invested in it being used to take low-paying jobs away from people with limited options, instead of something like drug discovery or cancer biology.

Interesting to see what the impact will be on crowdsourcing annotation companies like Scale AI, especially after reading this article: <a href="https:&#x2F;&#x2F;www.forbes.com&#x2F;sites&#x2F;kenrickcai&#x2F;2023&#x2F;04&#x2F;11&#x2F;how-alexandr-wang-turned-an-army-of-clickworkers-into-a-73-billion-ai-unicorn&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.forbes.com&#x2F;sites&#x2F;kenrickcai&#x2F;2023&#x2F;04&#x2F;11&#x2F;how-alexa...</a>

From reading the paper, GPT-4 also outperformed the researchers themselves in many categories, despite the researchers being the ones who created the dataset being used to perform the comparison.<p>In other words, the metrics are biased in the researchers’ favor — so GPT-4 would have beat them even more often (probably a majority of the time based on the numbers), if someone else had created the guidelines and golden labels.

This sounds awfully close to the bootstrap loop of singularity AGI.

NLP is solved, more or less. Either way, Bespoke NLP is on its way out. It&#x27;s pretty funny how buried this is in the original paper.

Buried in an arXiv paper was this nugget. Thought I&#x27;d share!

Very interesting. Until the day OpenAI has a problem in their systems and the entire world grinds to a halt. Or they put outrageous new prices. Which apparently never happened in other fields, seems.

So if AI can generate datasets better than it&#x27;s own datasets...well that&#x27;s pretty damn substantial.

If you look at the table, the GPT-4 model has better correlation with the expert ensemble than the crowd does, but only on some criteria. The GPT-4 model is closer for all of the ethics questions, but the crowd is closer for the utility level and economic impact questions.

Does OpenAI even have the compute to begin to meet demand?

When an AI &quot;outperforms&quot; the &quot;ground truth&quot;, it is by definition &quot;worse&quot;, not &quot;better&quot;.<p>And if your ground truth is problematic, then this is generally a problem of specification and quality control, <i>not</i> performance.

&gt;This breakthrough saved the researchers over $500,000 and 20,000 hours of human labor.<p>BTW, this is interesting. There is a lot of noise about AI carbon footprint. Now imagine how much humans would eat and fart for 20.000 work hours. It&#x27;s about 10 man&#x2F;years. Assuming 8h &#x2F; 5d &#x2F; 50 weeks schedule.

This is really interesting result. Immediate and direct application of LLMs, with significant financial benefits. I think LLMs will drive tremendous productivity increase.

“ Employing Surge AI&#x27;s top-tier human annotators at a rate of $25 per hour would have cost $500,000 for 20,000 hours of work”. That’s a wrap for Surge AI

What’s an elite crowdworker ? Top 1% sheep ? Or just the usual clickbait oxymoron ?