FWIW I ran a startup that provided you with a program (single binary) that allowed you to run natural language queries on your database across most schemas. It had a full semantics layers which translated your query into a mixed-lambda calculus-prolog query, which is then translated into SQL as needed - you can see a sample of the semantics layer here: <a href="https://youtu.be/fd4EPh2tYrk?t=92" rel="nofollow">https://youtu.be/fd4EPh2tYrk?t=92</a>.<p>It's deep learning based with a lot of augmentation. Going from the OP's article to actually being able to run queries on any schema is quite a bit more work. I'd love to see GPT3 handle arbitrary schemas.<p>p/s: the startup failed. deep research based startups need a lot of funds.
<i>Woah. I never gave it my database schema but it assumes I have a table called "users" (which is accurate) and that there's a timestamp field called "signup_time" for when a user signed up.</i><p>I am definitely impressed by the fact that it could get this close without knowledge of the schema, and that you can provide additional context about the schema. Seems like there is a lot of potential for building a natural language query engine that is hooked up to a database. I suppose there is always a risk that a user could generate a dangerous query but that could be mitigated.<p>Not related to the article but what exactly is "open" about OpenAI?
This is a use case where AI-powered SQL is a solution in search of a problem, and introduces more issues than just doing boring SQL. For data analysis, it's much more important to be accurate than fast, and this article is unclear how many attempts each example query took. GPT-3 does not always output coherent output (even with good prompts), and since not 100% of the output is valid SQL the QA and risk tolerance of bad output affects the economics.<p>OpenAI's GPT-3 API is expensive enough (especially with heavy prompt engineering) that the time saved may not outweigh the cost, particularly if the output is not 100% accurate.
This seems to be consistent with my outsider view of AI demos.<p>1) Have a question<p>2) Figure out the answer<p>3) Have the AI figure out the answer<p>4) If the AI figured out your answer, be impressed, otherwise try again.
The problem with the curre t "AI" technology is it is only approximately correct (or rather, it is somewhat likely to produce a "good" result). This gives great use-cases when it comes to human perception, as we can filter out or correct small mistakes and reject big ones. But when used as input to a machine, even the smallest mistake can have huge consequences. Admittedly, this nonlinearity also applies when human beings "talk" to machines, but the input to and output of a single human being will always be constrained, whereas a machine could output billions of programs per day. I don't think it would be wise to follow that route before we have computational models that can cope with the many small and few big mistakes an "AI" would make.
You know what grinds my gears with GPT-3? The fact that I can't tinker with it. I can't do what this guy just did, or play around with it, or learn from it, or whatever. Access is limited.<p>I feel like I'm back in 95, when I had to beg faculty staff to get a copy of VB on some lab computer, only to be able to use it 1 hour a day. Restricting knowledge like this, in 2021, feels odd.
Am I reading it wrong, or was the output for this example wrong:<p>Input: how much revenue have we received from users with an email ending in 'seekwell.io' in the last 3 months?<p>The output conditions on:
users.signup_dt>= now() - interval '3 months'<p>But my interpretation would be to condition on charges.charge_dt<p>It reminds me of the criticisms of almost fully autonomous vehicles where the driver pays less attention. Now I'm curious if the queries written from scratch would be less likely to have errors than queries that use GPT3 as a starting point.
I've built this as well! I won an internal hackathon with this and so I ran up against <i>many</i> of the issues you'll find here.<p>1. There is unlimited flexibility in the prompt.<p>Seemingly irrelevant changes to the prompt can change whether you get out correct SQL or not. Sometimes you can just repeat things in the prompt and get different and better results. "Write correct SQL. Write correct SQL"<p>For any one input question you may be able to tweak the prompt to get the correct answer out. But you need to do this tweaking for each question (and know the correct answer you need). Tweaking one prompt may break all other input-output pairs.<p>2. Real questions involve multiple large schemas.<p>I deal with tables with thousands or tens of thousands of columns. There is no way you can get GPT-3 to deal with that scale with a simple input as shown here. And of course you want to join across many tables etc.<p>3. Syntax<p>Natural language is more robust than SQL, you can get <i>close</i> and get the point across. Most language models trained on general corpora are fundamentally not suited to the symbolic manipulation of languages like SQL.<p>This isn't to say that GPT-3 couldn't be part of a solution to this problem, but please restrain your exuberance, it's not going to solve this problem out of the box.
He's our fastest business analyst but sometimes on a hot day he'll just keep repeating "Syntax Error"...<p>Very cool work, I continue to be blown away by what GPT-3 can achieve.
"GPT-3, I need a React App that displays vital economic statistics from the World Bank API."<p>----<p>"Nice! can you add a drop-down for regional statistics when in country view?"<p>----<p>"Just one last thing. Can you make the logo bigger?"
Amazon pitched something like this at reinvent. I'm bearish because the art to good analytics is deeply understanding the data. The SQL queries are the easy part. It's knowing when to omit inactive states, or how accrual works in the system, or that field X was depreciated and isn't accurate anymore.<p>These systems solve none of those underlying questions, and in fact are more likely to make it worse. Because if you won't even bother to write SQL or look at your database, you are even less likely to make sure it's high quality.<p>(There is maybe an exemption here for queries that you don't really care of you get the real answer. I don't know what those questions would be, but that would be a good usecase.)
We should start with the caveat that the GPT3 API waitlist doesn't actually move, you literally need to get an employee to get you manually off the waitlist.
This is really cool, but it's clear that the person requesting the SQL has to know whether the generated SQL is correct for it to be of use.<p>If I'm a non-technical user and I ask a plain-language question and the generated SQL is incorrect, it's likely going to give the wrong answer -- but unless it's terribly wrong ("Syntax error", truly implausible values) the user may not know that it's wrong.<p>So I see this as more of a tool to speed up development than a tool that can power end users' plain-language queries. But who knows? Maybe GPT-4 will clear that hurdle.
It reminds me of why tools like Tableau are so useful. You dont' have to teach people SQL or whatever, they can build their own visualizations and Tableau will do the SQL for you.
Can you be 100% sure that GPT-3 will really understand what you want to query? Or do you have to double check every query before you run it?<p>As for the latter, it's no automation, it's just a nice "autocomplete" feature.
I constantly wonder how people are getting access to the GPT-3 API (as beta users) when so many are still on the waiting list. The answer to use the Adventure Dungeon game is quite lacking.
Can someone check if the page <a href="https://blog.seekwell.io/gpt3" rel="nofollow">https://blog.seekwell.io/gpt3</a> is causing CPU spikes in the browser?
I’m really curious as to how this would deal with a “Bobby Tables” scenario or deliberate malicious intent.<p>The ethics implications are... fascinating.
Teaching it CTEs, subqueries, generic column identifiers (1,2,3), unions, left outer join, like, regexp, JSON, array, array_agg, concat, coelesce, trunc, etc. would be fun.<p>But something else humans can do is look at the data to construct patterns as another way to extract data. To some degree it requires some (bad) inductive reasoning approaches like “I’ll assume maybe most or all data in this column has this format.” “Oops, it didn’t, so let’s tweak it to look like this, to cover it sometimes also being empty string or null.”