I work at Alphabet and I recently went to an internal tech talk about deploying large language models like this at Google. As a disclaimer I'll first note that this is not my area of expertise, I just attended the tech talk because it sounded interesting.<p>Large language models like GPT are one of the biggest areas of active ML research at Google, and there's a ton of pretty obvious applications for how they can be used to answer queries, index information, etc. There is a huge budget at Google related to staffing people to work on these kinds of models and do the actual training, which is very expensive because it takes a ton of compute capacity to train these super huge language models. However what I gathered from the talk is the economics of actually using these kinds of language models in the biggest Google products (e.g. search, gmail) isn't quite there yet. It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day when you take into account serving costs, added latency, and the fact that the average revenue on something like a Google search is close to infinitesimal already. I think I remember the presenter saying something like they'd want to reduce the costs by at least 10x before it would be feasible to integrate models like this in products like search. A 10x or even 100x improvement is obviously an attainable target in the next few years, so I think technology like this is coming in the next few years.
It's great, until people realize GPT-3 will generate answers that are demonstrably wrong. (And to make matters worse, can't show/link the source of the incorrect information!)
These are addressing two very different concerns but framed as a singular one. Google is first and foremost a search engine - it searches the web for answers, the key point being the answers need to exist on the web. The other is a machine learning model tasked with deriving answers, and sometimes - if not very often answers will be provided in an authoritative tone whilst being completely and utterly incorrect.<p>Google is working on the latter called LaMDA[1] which is arguably more impressive and extensive than GPT-3, but for the reasons discussed above can't just be rolled out to the public. (edit: as others have noted, the code snippets themselves are wrong, but the Twitter poster didn't verify this because they're not interested in the answer, just the lack of one from Google).<p>It's certainly an interesting discussion for sure. Mathematics help (homework) is being built into search presently and one day for sure code-snippets will be embedded on search. However at Google's scale and the amount of scrutiny it receives spitting out machine-learning based results without any curation or substantiation is dangerous. Legally it is much safer to delegate to websites, thus alleviating any blame to the host.<p>1: <a href="https://en.wikipedia.org/wiki/LaMDA" rel="nofollow">https://en.wikipedia.org/wiki/LaMDA</a>
These examples are terrific, but the framing is ridiculous.<p>- GPT-3 answers can be incorrect, and don't carry enough context with them for the reader to engage critically.<p>- Text is often an inefficient presentation of an answer and Google's knowledge card results can do more and more (while adopting the risk above).<p>- LLM's are a ways from being scalable at this quality to a fraction of the throughput of Google queries.<p>- Search increasingly benefits from user-specific context, which is even harder to integrate at a reasonable expense into queries at massive throughput.<p>- Google is also regularly putting forward LLM breakthroughs, which will of course impact productized search.<p>As an NLP practitioner who depends on LLMs, I'm excited as anyone about this progress. But I think some folks are jumping to a conclusion that generative AIs will be the standalone products, when I think they'll be much more powerful as integrated into structured product flows.
Another person who doesn’t realise AI language models are just making shit up. Google results are quite often full of wrong information, but at least it has mechanism for surfacing better content: inbound links, domain authority, and other signals. It doesn’t guarantee correctness, but it’s better than the pseudo-authoritative fiction GPT-3 and friends come up with.
Here's an example on how misleading ChatGPT is:<p>Prompt:<p>Can you explain what happens when i enter "ping 16843009" in a linux shell?<p>Answer:<p><i>When you enter the command ping 16843009 in a Linux shell, you are instructing the ping utility to send an Internet Control Message Protocol (ICMP) echo request to the IP address 168.43.9.0. [...]</i><p>The correct answer is that it will ping 1.1.1.1<p>Also ChatGPT missed that fact that 16843009 is bigger than 255 and failed to explain what happens and why.
This is the very definition of clickbait. Not the Tweeter's fault, but it's a gray area when sharing Tweets on HN, since Tweets do not have a "title" per se.<p>From the HN Guidelines:<p>> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
I don’t think so. Google is still a search engine first and a question answering machine second. And for the question answering I will still prefer links over a blob of text that can’t be inspected or verified.
Google is only "done" if you never actually use it to find sites on the web. For nearly all of these examples I was thinking "WHY DON'T YOU JUST TRY CLICKING THE TOP LINK?" E.g. the first link for writing differential equations in LaTeX, I thought the first result, <a href="https://priyankacool10.wordpress.com/2013/10/15/writing-differential-equations-in-latex/" rel="nofollow">https://priyankacool10.wordpress.com/2013/10/15/writing-diff...</a> , provided excellent, helpful examples.<p>That is, if anything, I'd be quite satisfied with Google getting back to being a <i>search engine</i> and not just trying to bypass all the results that actually come back.
In the replies someone asks a basic physics question.<p>"In a vacuum do heavier objects fall faster?"<p>The response from GPT is completely wrong and so confident, it is like an imposter trying to save face.
What terrifies me is the idea of someone building a GPT-based bot specifically targeting Wikipedia. If one would train a model on the existing wiki dataset, it could generate and submit plausibly looking but completely factually false edits and articles with plausibly looking citations. Given the history of long-lasting hoaxes, it shouldn’t be hard to achieve enough throughput to completely overwhelm the capacity of human editors to do any fact checking.
The AI community needs to get real. All this talk about "large language model based AIs" is just smoke and mirrors. The technology is nowhere near advanced enough to convince the majority of people that it can provide genuine value in our lives. Stop pretending like these systems are capable of doing anything more than parroting back pre-programmed responses. The hype is out of control.<p>(The above comment was generated using ChatGPT)
Since we are posting ChatGPT Twitter links, I like this one:<p><a href="https://twitter.com/goodside/status/1598129631609380864" rel="nofollow">https://twitter.com/goodside/status/1598129631609380864</a><p>> explain the worst-case time complexity of the bubble sort algorithm, with Python code examples, in the style of a fast-talkin' wise guy from a 1940's gangster movie
I'm actually really interested in an AI that gives the wrong answers. It is a great way to generate filler when building out flashcards with the correct answer that you find when studying something. Is there a good open source (docker image) available ChatGPT3 equivalent that I can use for that, does anyone know?
<a href="https://twitter.com/jdjkelly/status/1598143982630219776/photo/1" rel="nofollow">https://twitter.com/jdjkelly/status/1598143982630219776/phot...</a><p>I went and checked out the Borges fable mentioned here: <a href="https://kwarc.info/teaching/TDM/Borges.pdf" rel="nofollow">https://kwarc.info/teaching/TDM/Borges.pdf</a><p>Looks like the ChatGPT summary is completely wrong? The map gets discarded instead of rendering obsolete the territory.
Google’s PaLM is current SOTA, way better than GPT-3 (non-tuned). I’m sure Google has many “tuned” internal-only PaLM variants in prod or testing today.
AI is often over-hyped, especially during the recent months.<p>But I think that we've all noticed the progressive degradation of search engines, including Google.<p>It is often more efficient to search on Reddit or Wikipedia or event YouTube.<p>But a good interactive LLM based chat agent could be a game changer.<p>I've used the demo and it is very useful to quickly get structured data in plain English or French, with well written code examples when needed.<p>It is not 100% there yet, the agent should be connected to a search engine backend, and maybe keep some long-lasting state for each user.<p>This is promising.
I seriously don't get this argument. Google can implement this themselves! It's not like they can't train a large language model akin to GPT-3 (they already have) or deploy it. And as others pointed out, language models are seriously not reliable right now in terms of producing true information.
What are the engineering and considerations for serving this sort of model to billions of queries a day? Do the economics of a gpt-as-a-search-engine work?
Generative models will surely change the shape of the web. If a major effect of freely sharing something is to enable a big AI company to ingest it and show it to their users without attribution, people are going to share things less freely. Which will then mean that these models won’t be able to generate new things as well.<p>I don’t know exactly how that will manifest, but something of that shape seems to be on the way.
I've never seen Solidity before, but it sure looks like `onlyOwner` is an arbitrary modifier name, and you could use _any_ modifier that contains a require(msg.sender == owner) assertion to restrict the caller. So shouldn't the answer be "...you can add a modifier to the function.." rather than "...you can add the onlyOwner modifier to the function...".
If there is really some other method that is better, why can't google just use that behind the scenes to provide answers? At the end of the day, google is what people are used to. They just go there without thinking. I do agree that the search engines part of it has become less effective but authoritative answers is an evolving field and google will evolve as it does.
For the use cases of question and answering, especially regarding technology, ChaGPT is indeed more flexible and convenient compared to Google and will surely replace a large part of this use case. However, Google is still irreplaceable as an index for the entire internet, and it will remain how we find content created by other _people_.
But the problem is, the "AI" doesn't actually know anything about the answer it is giving. It is simply brute-forcing and randomly generating based on a huge lookup table.<p>So what might appear to be an accurate answer, could in reality just be total garbage. Whereas the google answer has at least been written by an actual person.
I asked it to show me an example code for a Websocket server using Axum and it spit out some .NET code.<p>But while using it, generally I had the feeling that this could one day (3-4 years?) replace Google almost completely for all my code-related searches, which make up more than half for all my Google searches.
I asked on Twitter: "Why do you assume Google (who has one of the largest AI teams around, plus DeepMind) won't also integrate this into search too?"<p>I mean really, do people really think Google isn't also working on stuff like this?
I think these are 2 separate use cases, one for organized knowledge and one for related links. Google doesn't compile knowledge as well, but it does good job on finding related links.
I don't get it. Why can't Google just train their own LLM and use that for answer cards?<p>The main value of Google Search is the ability to search the web for websites. Not to search for answers.
In the first example, the AI seems more focused on extraneous stuff about aligning the equation, while the search result starts off by answering the question asked
Google is literally the “Kleenex” of search<p>Aka “just Google that”<p>I imagine the brand and goodwill value will have remarkable staging power forward as consumers decide where to do their AI search
GPT chat confidently claimed that Bill Gates never flew with Lolita Express (Jeffrey Epstein’s plane), even when I cited a New York Times article.<p>So, your mileage may vary
Yea... when being proactive, in any way that is not adversarial... ChatGTP has shown me that it's capable of providing very specific insights and knowledge when asking about topics Im currently curious about learning. And it works, I learn the type of information I was seeking. When the topics are technical, GPT is very good at crawl, walk, run with things like algorithms. It's great at responding to "well what about...".<p>Not only do I learn simpler, I gain better communication style myself when figuring out how to communicate with GPT. GPT also has a nice approach for dialog reasoning.<p>It's filter system may be annoying, however you can easily learn to play GPT's preferred style of knowledge transfer... and it's honestly something we can learn from.<p>TLDR; IMO ChatGPT expands the concept of learning, and self-tutoring, in an extremely useful way. This is something no search engine of indexed web pages can compete with. Arguably, the utility of index web pages is really degraded for certain types of desired search experiences when compared to ChatGPT... which it seems obv that internet browsing will be eventually incorporated (probably for further reference and narrowed expansion of a topic)
scaling a large language model to serve thousands of queries per second and be continuously updated is not trivial.<p>I'm sure we'll get there at some point.
I wonder what this will do to misinformation. Seems like the next big culture war will be over AI. What seems very Utopian will quickly be framed as dystopian. If AI doesn't promote "opposing positions" it will definitely become the target of politicians ire, if not outright banning as <insert political party here> propaganda. For example, what would AI say in terms of the effectiveness of Ivermectin in combatting COVID-19? or Vaccine injury rates? Would AI argue that lockdowns are the most effective measure against a spreading pandemic?