For context, I'm a experienced SWE working on some fairly complex things. I've been using programming GPT copilots for 6-8 months now and lately I've been using them less and less.<p>I think for complete beginners or casual programmers, GPT might be mind-blowing and cool because it can create a for loop or recommend some solution to a common problem.<p>However, for most of my tasks, it usually has ended up being a total waste of time and leads to frustration. Don't get me wrong, it is useful for those basic tasks for which in the past I'd do the google -> Stack Overflow route. However, for anything more complex, it falls flat.<p>Just a recent example from last week - I was working on some dynamic SQL generation. To be fair, it was a really complex task and it was 5pm so I didn't feel like whiteboarding (when in doubt, always whiteboard and skip gpt lol). I thought I'd turn to GPT and ended up wasting 30 minutes while it kept hallucinating and giving me code that isn't even valid. It skipped some of the requirements and missed things like a GROUP BY which made the generated query not even work. When I told it that it missed this, it regenerated some totally different code that had other issues.. and I stopped.<p>When chatGPT first came out I was using it all the time. Within a couple weeks though it became obvious it is really limited.<p>I thought I'd wait a few months and maybe it would get better but it hasn't. Who exactly are copilots for if not beginners? I really don't find them that useful for programming because 80% of the time the solutions are a miss or worse don't even compile or work.<p>I enjoy using it to write sci fi stories or learn more about some history stuff where it just repeats something it parsed off wikipedia or whatever. For anything serious, I find I don't get that much use out of it. I'm considering canceling my subscription because I think I'll be okay using 3.5 or whatever basic model I'd get.<p>Am I alone here? Sorry for the ramble. I just feel like I had to put it out there.
I think of them as a more intelligent autocomplete. I don’t lean on them too heavily but I find that they make my life 5% easier by autocompleting based on style and known names of things versus relying wholly on the LSP. (Copilot)<p>On the GPT-4 side I’ve had great luck with dealing with complex SQL/BigQuery queries. I will explain a problem, offer my schema or a psql trigger and my goals on how to augment it and it’s basically spot on every time. Helps me when I know what I want to do but don’t know precisely how to achieve it.
I've stopped trying to use GPTs for complex tasks like what you describe, but I find them to be invaluable for getting a lot of grunt work done on my hobby projects.<p>As a concrete example: GitHub Copilot has been absolutely life-changing for working on hobby programming language projects. Building a parser by hand consists of writing many small, repetitive functions that use a tiny library of helper functions to recursively process tokens. A lot of people end up leaning on parser generators, but I've never found one that isn't both bloated and bad at error handling.<p>This is where GitHub Copilot comes in—I write the grammar out in a markdown file that I keep open, build the AST data structure, then write the first few rules to give Copilot a bit of context on how I want to use the helper functions I built. From there I can just name functions and run Copilot and it fills in the rest of the parser.<p>This is just one example of the kind of task that I find GPTs to be very good at—tasks that necessarily have a lot of repetition but don't have a lot of opportunities for abstraction. Another one that is perhaps more common is unit testing—after giving Copilot one example to go off of, it can generate subsequent unit tests just from the name of the test.<p>Is it essential? No. But it sure saves a lot of typing, and is actually <i>less</i> likely than I am to make a silly mistake in these repetitive cases.
Threads like these often feel like reading a lot of "Does anyone else feel like your Phillips screwdriver just isn't very good at hammering in nails? It kind of works, but I'd rather not use it for that."<p>I really have zero desire to ever be programming without Copilot again, and have been writing software for over a decade.<p>It just saves so much time on doing all the boring stuff I wanted to tear my hair out on wondering why I even bother doing this line of work at all.<p>Yeah, you're right, it's not as good as I am at the complex more abstracted things I actually enjoy solving.<p>So it does the grunt work like writing the I/O calls, logging statements, and the plethora of "nearly copy/paste but just different enough you need to write things out" parts of the code. And then I do the review of what it wrote and the key parts that I'm not going to entrust to a confabulation engine.<p>My favorite use is writing unit tests, where if my code is in another tab (and ideally a reference test file from the same package) it gets about 50% there with a full unit test and suddenly my work is no longer writing the boilerplate but just narrowing in the tailored scaffolding to what I actually want it to test.<p>It's not there to do your job, it's there to make it easier in very specific ways. Asking your screwdriver to be the entire toolbox is always going to be a bad time.
I believe the mistake people use with copilot is that they attempt to write large projects or functions when they lack the knowledge base of the underlying technology.<p>I prefer to use it as more of an autocomplete on a line per line basis when writing new code.<p>Typically, I use it for small and concise chunks of code that I already fully understand, but save me time. Things like "Here's 30 lines of text, give me a regex that will match them all" or "Unroll/rewrite this loop utilizing bit shifting".<p>I also use copilot as a teacher. Like to quickly grok assembly code or code in languages that I do not use everyday. Or having a back and forth conversation with copilot chat on a specific technology I want to use and don't fully understand. Copilot chat makes an excellent rubber duck when working through issues.
I was writing a sudoku solver today and there was a bug that took me a while to track down (can't remember exactly how long - could be a few minutes to a couple of hours).<p>I asked ChatGPT to find the bug and it didn't find it. I also asked GPT4-Turbo to find the bug and it also couldn't find it. In the end I found the bug manually using tracing prints.<p>After I found the bug, I wondered if GPT4 could have found it so I gave the buggy code to GPT4 and it found the line with the bug instantly.<p>To me this shows that GPT4 is much better than GPT4-Turbo and GPT-3.5
<i>> Who exactly are copilots for if not beginners?</i><p>The thing is: in software engineering, you're very often "a beginner" when using new technology or operating outside your familiar domain. In fact, you need to learn constantly just to stay in the business.
In my experience:<p>- For basic autocompletion is ok, on lazy days I even find myself often thinking "why the ai is not proposing a solution to this stupid method yet?".<p>- For complicated coding stuff is worthless, I've lost a lot of time trying to fix some ai generated code to end up writing the stuff again, so I rely on google/stackoverflow for that kind of research.<p>- For architectural solutions or some research like looking for the right tool to do something I found it quite useful as it often present options I didn't consider or didn't know in the first place, so I can take them also in consideration.
I get a tremendous amount of value from ChatGPT, like you said for things where I would previously have to google -> stack overflow it's incredibly useful. It works as a insanely good search/autocomplete and that is worth a ton. I love being able to sketch a function with an example input/output and have it return something correct, or at least close 95% of the time. As an experienced dev it's easy for me to look at it and get to 100%.<p>It's also so helpful to be able to just ask questions of the documentation on popular projects, whether it be some nuance of the node APIs or a C websockets library, it saves me countless hours of searching and reading through documentation. Just being able to describe what I want and have it suggest some functions to paste into the actual documentation search bar is invaluable.<p>Similarly I find it's really helpful when trying to prototype things, the other day I needed to drop an image into a canvas. I don't remember off top exactly how to get a blob out of an .ondrop (or whatever the actual handler is) and I could find it with a couple minutes of google and MDN/SO, but if I ask ChatGPT "write me a minimal example for loading a dropped image into a canvas" I get the exact thing I want in 10 seconds and I can just copy paste the relevant stuff into MDN if I need to understand how the actual API works.<p>I think you're just using it wrong, and moreover I think it's MUCH MUCH more useful as an experienced engineer than as a beginner. I think I get way more mileage out of it than some of my more junior friends/colleagues because I have a better grasp on what questions to ask, and I can spot it being incorrect more readily. It feels <i>BAD</i> to be honest, like it's further stratifying the space by giving me a tool that puts a huge multiplier on my experience allowing me to work much faster than before and leaving those who are less experienced even further behind. I fear that those entering the space now, working with ChatGPT will learn less of the fundamentals that allow me to leverage it so effectively, and their growth will be slowed.<p>That's not to say it can't be an incredibly powerful learning tool for someone dedicated to that goal, but I have some fear that it will result in less learning "through osmosis" because junior devs won't be forced into as much of the same problem solving I had to do to be good enough, and perhaps this will allow them to coast longer in mediocrity?
I have been using Cody (sourcegraph.com/cody) for about 6 months now and it's completely changed the way I write code. But, there was an adjustment period to learn how to work with the tool. Expecting a code copilot to just give you working code 100% of the time is unrealistic today, we may get there eventually though.<p>I've been writing code for close to 20 years now across the full stack, I have written a lot of bad code in my life, I have seen frameworks come and go, so spotting bad code or spotting bad practices is almost second nature to me. With that said, using Cody, I'm able to ship much faster. It will sometimes return bad answers, i may need to tweak my question, and sometimes it just doesn't capture the right context for what I'm trying to do, but overall it's been a great help and I'd say has made me 35-40% more efficient.<p>(Disclaimer: I work for Sourcegraph)
A lot depends on what you use them for.<p>I don’t find them that great at large scale programming and they couldn’t do the hard parts of my work, but a lot of what I do doesn’t need to be “great.”<p>There’s the core system design and delivering of features. That it struggles with. Anything large seems to be a struggle.<p>But generating SQL for a report I do sporadically on demand from another team?<p>Telling me what to debug to get Docker working (which I am rarely doing as a dev)? Anything Shell or Nginx related (again, infrequent, so I am a beginner in those areas)<p>Generating infrequently run but tedious formatting helper functions?<p>Generating tests?<p>Basically, what would you give a dev with a year of experience? I would take ChatGPT/Copilot over me with 1 year of experience.<p>The biggest benefit to me is all the offloaded non-core work. My job at least involves a lot more than writing big features (maybe yours does not).
It’s incredible for my use case.<p>I have been involved in software and implementing technical things since the late 90s and from time to time have been pretty good at a few things here and there but I am <i>profoundly</i> rusty in all languages I sort of know and useless in ones I don’t.<p>But I’m technical. I understand at sort of a core level how things work, jargon, and like the key elements of data structures and object oriented code and a MVC model and whatever else. Like I’ve read the right books.<p>Without ChatGPT I am close to useless. I’m better off writing a user story and hiring someone, anyone. Yes I can code in rails and know SQL and am actually pretty handy on the command line but like it would take me an entire day and tons of googling to get basic things working.<p>Then they launched GPT and I can now launch useful working projects that solve business problems quickly. I can patch together an API integration on a Sunday afternoon to populate a table I already have in a few minutes. I can take a website I’m overseeing and add a quick feature.<p>It’s literally life changing. I already have all the business logic in my head, and I know enough to see what GPT is spitting out and if it’s wrong and know how to ask the right questions.<p>Unlike the OP I have no plans to do anything complex. But for my use cases it’s turned me from a project manager into a quick and competent developer and that’s literally miraculous from where I’m standing.
I'm the beginner you are talking about, I have mixed feelings about coding with AI.<p>I'm not a programmer, I'm a student in acc/fin, to use a weird analogy, if you are a chef, I'm a stereotypical housewife, and we think differently about knives (or GPTs).<p>I differentiate between tuples, lists and dictionaries not by the definition, but by the type of brackets they use in Python. I use Python because it's the easiest and most popular tool, and I use Phind and other GPT tools because programming is just a magic spell for me to get to what I want, and the less effort I have to spend the better.<p>But it doesn't mean that GPTs don't bring their own headaches too. As I get more proficient, I now realise that GPTs are now giving me bad or inefficient advice.<p>I can ask a database related question and then realise, hang on, despite me specifying this is for Google BigQuery, it's giving me an answer that involves some function I know is not available on it. Or I read the the code it recommends for pandas and realise, hang on, I could combine these two lines into one.<p>I still use GPT heavily because I don't have time to think about code structure, I just need the magic words to put into the Jupyter cell, so I can get on with my day.<p>But you don't, and you actually think about these things, and you are realising the gaping flaws in the knife's structure. That's life. YOu have a skill and there comes pros and cons with it.<p>Like a movie reviewer who can no longer just go to the cinema and enjoy something for the sake of it... you also can't just accept some code from a GPT and just use it, you can't help not analyse it.
GPT-4 was addictive for me. I subscribed to replace online language classes and it was an excellent instructor. After passing the exam I unsubscribed and I'm living fine with 3.5. My screen time definitely lowered :) I bought $5 of GPT-4 API credits to use when 3.5 really can't do the job, but it rarely happens. Asking MS Copilot is also another great way to use GPT-4 for free. On the job I mostly use GH Copilot for code completion and it's great as it provides suggestions that are in line with the code style of my team. On serious tasks all chat bots allucinate and I also feel I'm spending as much time correcting them as if I studied the topic from scratch, because I (want to?) believe what they say but end up wasting a lot of time fixing their suggestions. I'm also thinking about SQL, today it suggested me an `UPDATE table JOIN another_table SET column` and I was surprised I could use JOINs in update statements, but the bot was so sure it was the right keyword. I tried to understand where _else_ the syntax error could be, until I turned back to postgres official documentation and verified there's no JOIN, only FROM, just like I remembered.
Between your post and Air Canada’s learning they have to honor policies their chat bot hallucinates and relays to customers, it seems like the zeitgeist is starting to comprehend the inherent limitations and risks of LLMs.<p>I find that kind of heartening, honestly.<p>But it’s by no means a death sentence for AI. Plenty of dimensions for massive improvement.
While I’m more on the infrastructure side of things, I see a similar issue. Like you mentioned, it’s great for lookups of API documentation and getting examples etc. I have also used it for things like templates and boring boilerplate. I have come to look at it as a lookup tool and something that converts my thoughts into code. I could see myself sitting at home and doing a lot of coding by voice and a vr headset in the future if the tools continue to develop.
At the moment I think we just need to come up with a better way of integrating it into our workflow. I’m starting to wonder if something like visual programming could work well with the “ai” “figuring” out the content of the blocks we connect and basically lets us influence the generated code by the io. That could be a solution to coding on tablets and phones with minimum typing.
Not alone. I was pretty grouchy about it a few months ago. It seems to be getting better, though.<p>I code all over the stack, usually some bizarre mix of python, pyspark, SQL, and typescript.<p>TS support seems pretty nice, and it can optimize and suggest pretty advanced things accurately.<p>Py was hopeless a few months ago, but my last few attempts have been decent. I've been sent down some rabbitholes though, and been burned -- usually my not paying attention and being a lazy coder.<p>PySpark is just the basics, which is fine if I am distracted and just want to do some basic EMR work. More likely, though, I'll rummage my own code snippets instead.<p>The speed of improvement has been impressive. I'm getting enthused about this stuff more and more. :)<p>Plus, who doesn't enjoy making random goofy stuff in Dall-E while waiting for some progressbar to advance? That alone is worth the time investment for me.
My personal reason for not using GitHub Copilot / etc:<p>I was testing ChatGPT-3.5 with F# in 2023 and saw some really strange errors. Turns out it was shamelessly copying from GitHub repos that had vaguely related code to what I was asking - this was easy to discover because there's not much F# out there. In fact the relative sparsity of F# is precisely why GPT-3.5 had to plagiarize! It did not take long to find a prompt that spat out ~300 lines verbatim from my own F# numerics library. (I believe this problem is even worse for C numeric programmers, whose code and expertise is much more valuable than anything in .NET.) OpenAI's products are simply unethical, and I am tired of this motivated reasoning which pretends automated plagiarism is a-okay as long as you personally find it convenient.<p>But even outside of plagiarism I am really nervous about the future of software development with LLMs. So often I see people throwing around stats like "we saw a 10% increase in productivity" without even mentioning code quality. There are some early indications that productivity gains in LLM code assistance are paid for by more bugs and security holes - nothing that seems catastrophic, but hardly worth dismissing entirely. What is frustrating is that this was easily predictable, yet GitHub/OpenAI rushed to market with a code generation product whose reliability (and legality) remains completely unresolved.<p>The ultimate issue is not about AI or programming so much as software-as-industrial-product. You can quickly estimate increases in productivity over the course of a sprint or two: it's easy to count features cleared and LoC written. But if there are dumb GPT "brain fart" errors in that boilerplate and the boilerplate isn't adequately reviewed by humans, then you might not have particularly good visibility of the consequences until a few months pass and there seem to be more 5-10% bug reports than usual. Again, I don't think the use of Copilot is actually a terrible security disaster. But it's clearly a risk. It's a risk that needs to be addressed BEFORE the tool becomes a de facto standard.<p>I certainly get that there's a lot of truly tedious boilerplate in most enterprise codebases - even so I suspect a lot of that is better done with a fairly simple <i>deterministic</i> script versus Copilot. In fact my third biggest irritation with this stuff is that deterministic code generation tools have gotten <i>really good</i> at producing verifiably correct code, even if the interface doesn't involve literally talking to a computer.
I have been programming for over 40 years, mostly in fairly verbose languages but nowadays mostly in JavaScript and clojure, both of which can be very concise.<p>I find I spend most of my time thinking about the problem domain and how to model it in logic, and very little time just banging out boilerplate code. When I want to do the kind of task a lot of people will ask gpt for, I find it's often built into the language or available as an existing library - with experience you realise that the problem you're trying to solve is an instance of a general problem that has already been solved.
You are not alone.<p>At the core AI/ML is giving you answers that have a high probability of being good answers. But in the end this probability is based on avarages. And the moment you are coding stuff that is not avarage AI does not work anymore because it can not reason about the question and 'answer'.<p>You can also see this in AI generated images. They look great but the avarage component makes them all look the same and a kind of blurry.<p>For me the biggest danger of AI is that people put too much trust in it.<p>It can be a great tool, but you should not trust it to be the truth.
I've found that they offer poor solutions but good starting points. I learned a lot more about using Leaflet and d3 for a personal project because GPT4-Turbo gave me a solution, which I took as a starting point. It gave me insight that illuminated the documentation for each that was previously opaque to me. As such, I find value in GPTs as accelerators for learning. They're non-judgmental so you needn't worry about whether or not you asked your question correctly on SO.
In my experience, if I use github copilot chat as a natural language interface to programming it's basically flawless. That is, I tell it exactly what I want the code to he like but in english, and the llm just translates that into code for me. When using the inline github copilot, I find that whenever I am making changes that are obvious, it gets things 100% right. If there are no obvious mistakes or additions to my code, it makes up completely useless noise. I just ignore that.<p>I have had some experiences where I did not know the language or library that I wanted to work with, and the results were disastrous. It hallucinates exactly the api that I want, but unfortunately it doesn't exist. As an aside -- that api SHOULD exist. The api design is very obviously missing this extremely important feature. I think llms may help people to figure out what kind of apis and features should be intuitively expected and increase software quality. But it still wasted a ton of my time.<p>My conclusion is actually that llm copilots will be bad for beginners because they don't know enough to specify the request in enough detail, and they don't know enough to see any errors and be able to understand why they were generated.
Yeah it's obviously useless for anyone doing advanced programming work. I'm writing compilers (for AI models of all things), and it's just not there. It has some amazing wins. I have fed it a description of ISAs for proprietary chips and had it generate correct kernels, and such. That is impressive, and it's very useful for references for quick tutorials on common enough things (but do check the docs). However, it's not able to do abstract thinking really yet.<p>On the other hand? If I want to add payments to my web app (not a web dev, but I mess around with side projects for fun), and I don't want to read the stripe library docs for example, it's pretty good at getting me started installing the right library, and generating the boiler plate. Same with various boilerplate for code generation.<p>On the other hand, ChatGPT (which has been clearly trained on stuff I've read) has an uncanny ability to describe my esoteric side projects much better than I can, so there is that. I have some DSLs written in Haskell and it's actually really good at providing answers with it, and explaining things succinctly.
Mathematical reasoning only exists associativity in any pure language model so inherently any tasks that requires any sort of mathematical complexity isn't one you should be approaching without at least defining the logic, algo, interfaces, inputs, expected outputs, tests etc. yourself first IMO.<p>With that being said, IMO, you can optimize your dev work to get the most use from GPT4 by providing better context, examples etc. these don't really fit into the co-pilot workflow but can fit into instruction-tuned model workflows. (GPT4 / Mixtral / Mistral / CodeLLAMA etc.)<p>E.g. use the first 50k tokens in the context window to pre-prompt with relevant context from your code repo/sdk documentation/examples whatever prior to actually prompting with your task. And demonstrate a test in your prompt that displays what you want. Taking a "few-shot" approach with specified validation (in your test) results in considerable improvements in accuracy for programming (and other) tasks.<p>(YMMV)
I learned Haskell recently, and I thought ChatGPT was a great support for that effort. It helped me with definitions, standard recipes and disambiguation. What it wasn't very good at was coding: it couldn't solve anything I did not know how to solve, but it was quite good for things I did know how to solve but couldn't be bothered to. For example, I could ask it to re-write a messy snippet of maps and filters as a list comprehension, and it would. The comprehension it produced may not work, but I could still glean enough from the approach that it would be useful.<p>I'd also ask it to re-write existing code "more canonically" or "more succinctly". Again, it fails to do this almost every time, but often it uses something that I didn't know existed whilst doing it, which I found to be valuable.
As a newb to programming I find it helpful for boilerplate and “smart” autocomplete tasks (which makes sense, as a LLM is simply autocomplete on steroids). I have found it helpful for Regex as well, which is a topic that still catches me. It can also be interesting (albeit typically somewhat useless) to bounce ideas off of it and have it attempt to solve problems. It rarely helps, but in my experience humans haven’t been much better outside of my professor.<p>I’ve found its best at completing repetitive but simple tasks, such as organizing data or making simple scripts. Anything more complex or that cannot be easily explained in a way it “understands” will run into problems. For instance, telling it to program a custom variant of Dijkstra’s Algorithm for my specific input data? No go, it doesn’t work. However, a simple script to replace any text from column 1 in a CSV with its neighbor text in column 2? Works first try.
Once CoPilot can understand entire workspace and have deep understanding of your project, it will likely seem a lot smarter than it seems now.<p>All I use it for is to avoid repetitive stuff as it's exceptionally good at guessing my next step.<p>The autocomplete bits feel wrong most of the time, and as fast as API updates happen it's mostly a wash in terms of productivity.
1) Some areas where it was a great success and saved me a lot of time:<p>* For my current gig, I recently had to play the best computer game of all time: Linux. ChatGPT was invaluable here. "How do I figure out where the network connections are managed on this device?"<p>* Also ChatGPT managed to assist me in creating a task breakdown for a prototype that I was building<p>2) Area where it was of some help. I think using traditional methods would have been just as good in these cases:<p>* Sample code for the usage of TS clients for MS Graph and notion<p>* generating an almost working OPC UA dummy server in TS using "node-opcua"<p>3) Areas where it was of no help and mostly a one way box where I could enter my thoughts. Not sure if using ChatGPT here was a win or a loss, though - you could still think of the process as some form of notetaking with feedback:<p>* Working on some complex tree recursion things<p>* trying to come up with the interface for an infrastructure-as-code framework
Having some "AI" that generates code in your IDE is roughly equivalent to copy-pasting code from the first google hit. Great for search/autocomplete/filling out obvious patterns. The grunt work. But terrible for doing anything novel or interesting - which presumably is the entire reason you have a software job.
It being useful for the basic Google/Stack Overflow tasks makes it worth it for me. It saves me a lot of time being able to simply bring up the exact answer to whatever small "how-do-I-do-this-again" problem that fits into my code without rewriting it.<p>You're right though that it isn't very useful for complex problems. Complex SQL queries in particular are hopeless. But I probably only spend 10-20% of my time on stuff where I <i>really</i> have to think about what I'm doing anyway.<p>The autocompletion is the best part for me. When I start writing a function called `mapUsersToIds` or whatever and it autocompletes the 3 or 4 lines of code that I now don't need to write, that saves me a ton of time - I'd say I'm 30% more productive now that I don't need to spend time typing, even if the autocompleted code is exactly what I was going to write in the first place.
I used Copilot, but the code it generated was either bad or it made up things that didn't exist. The same goes for ChatGPT.<p>Both tools work if you don't care about the quality of the code and are working with boring tech, otherwise, it's a total disaster.
I agree, they aren't great at programming, but I've found value from LLMs for rubber-ducking and quickly getting into new framework / library / language by using your existing knowledge and let the LLM augment that for you. "How would I do X using Y", "Is this performant, and does it have these pitfalls". It works surprisingly well even though the output may not be 100% correct, but it will get you started faster / point you usually at the correct direction. Also used chatgpt to generate marketing text which I absolutely cant write. [1]<p>1: <a href="https://mouselocker.cloudef.pw/" rel="nofollow">https://mouselocker.cloudef.pw/</a>
As someone building in the AI space I can give my unbiased opinion. I do not think AI is the big productivity boost we expect them to be yet.
I have been a software engineer for around 6 years now and I the only AI tooling I prefer and have really can not live without today is autocomplete (a better and smarter LSP).<p>When it comes to making a more complicated change, things are never easy because of the limited context window and the general lack of reasoning and inherent knowledge of the codebase I am working with.<p>Having said this, GPT4 has been really good for the one off questions I have about either syntax or if I forget how to do "the thing I know is possible I am just not sure" or the mundane things like docker commands or some other commands which I need help with.<p>But... if you guys have seen Gemini-1.5 Pro I was seriously mind blown and I think the first time I felt a LLM is better than me and that has to do with code search. I have had my fair share of navigating large codebases and spending time understanding implementations (clicking go-to-reference, go-to-definition) and keeping a mental model.. the fact that this LLM can take a minute to understand and answer questions about codebase does feel like a game changer.<p>I think the right way to think about AI tooling for programming is not to ask them to go and build this insane new feature which will bring lots of money for you, but how they can help you get that edge in your daily workflows (small quality of life changes which compound over time, just like how LSP is taken for granted in the editor now a days).<p>Another point to mention here which I believe is a major miss is that these copilots write code without paying attention to the various tools we as humans would use when writing code (LSP, linters, compilers etc). They are legit writing code like they would on a simple notepad and that is another reason why the quality is often times pretty bad (but copilot has proved that with a faster feedback loop and the right UX its not too big a hassle)<p>We are still very early in this game and with many people building in this space and these models improving over time I do think we will look back and laugh how things were done pre-AI vs post-smart-AI models.
i tried it once. it kept lying to me and misunderstanding trivial sentences. maybe its better these days, but i hate it with a passion.<p>me: Can you tell me how to implement XYZ without using standard library?
gpt: use function y from the standard library.<p><i>throws computer out of window</i><p>it can't even properly understand basic language. it remembers occurences and does statistical analysis... it just prentends it understands, in all cases, even if its accidentally right. a multi-billion dollar project to get answers from a toddler blurting out random words it's heard around it...
Anecdotally people are saying it may actually have gotten significantly worse recently. So it’s possible that the results you might have got from GPT-4 a month ago are no longer anything like as accurate or useful.
It can help with complicated tasks. When prompting via the chat interface, it comes more naturally to prompt with a full description of the problem.<p>With too much assumed context, it only does a good job of spitting out the answer to a common problem, or implementing a mostly correct version of the commonly written task similar to the one requested.<p>When you use copilot, are you shaping your use to its workflows? Adding preceding comments to describe the high-level goal, the high-level approach, and other considerations for how the code soon to follow interacts with the rest of the codebase?
I haven't used copilot much at all, but I use chat gpt occasionally. One of the queries I find it does the best with is: "What's an idiomatic way to do XYZ in language ABC".
Similar experience. I really only still use Copilot to:<p>- generate short blocks of low-entropy code (save some keystrokes)<p>- get me off the ground when using a new library (save some time combing through documentation)
I’ve had good success with ChatGPT and with Copilot, but I’ve found it really depends on how I use it and my expectations.<p>It is NOT good at generating complex functions or code but simple cases and I do not use copilot for “comment driven development” as my experience with that wasn’t good.<p>I use it as a fancy autocomplete, I use it to explain code, I use it for refactoring code (eg it’s good at tasks like “change all variables backed to snake_case”), I use it to provide api usage examples for functions, I use it to generate basic docstrings (which I then hand edit to add the detail I need), I use it to point out flawed in code (it’s bad at telling me if code is good, but it’s pretty good at spotting mistakes if the code is bad).<p>I’ve also used it to generate functions or algorithms that I wouldn’t have been able to do by myself and that includes complex SQL (TimescaleDB windowed queries), in an iterative process. I find the best results are when you generate short snippets at a time with as few requirements as possible but with as much detail on what you’re asking for as possible, and then it’s an iterative process. You also need to guide it, it works if you already know how you want to approach it and just want the AI to figure out the nitty gritty specific details. I’ve used this to generate a concurrent bitset-based memory allocator. I don’t think I could have done it by myself.<p>I’ve also had success generating little Python scripts in their entirety, although to get the exact behaviour I wanted was a process of repeatedly pointing out problems and eventually noticing it was not where the LLM was editing. So it’s important you understand every bit of code that it generates and don’t accept code until you’re certain you understand it.<p>As an example of a recent script it generated for me, it reads a TOML file containing a table, it looks for a key that you specify as a command line arg. It then generates a CSV for all keys it finds. Eg it turns this:<p><pre><code> [test.a]
x = 1
y = 2
[test.b]
x = 3
z = 4
</code></pre>
Into this:<p><pre><code> x,y,z
1,2,
,3,4
</code></pre>
It did this a lot faster than it would have taken me to do it myself.<p>I’ve also had good experience using ChatGPT to do pros and cons list of approaches or do some basic research for new eg into algorithms (now that it can search the web).
You're not alone, I have the exact same experience.<p>I already gave up on anything complex, but it also fails at relative simple things.<p>It goes like this:
The first answer does something useful, but is not the full solution or contains a bug.<p>When told, it apologizes and gives code that does not even compile.
Then when trying to get it in the right direction, it gets worse and worse.<p>Then it hallucinates a non-existing library,that should solve the problem.<p>And in the end I end up writing the code myself...
yes that are only good for simple cases, but at least for me those are not satisfying to solve<p>so letting it write some boilerplate (copilot) or solve simple problem that I don't want to waste my time for is providing me enough value to be worth spending money<p>also I find it very useful in filling out tests, where it mostly correctly fills other test cases given single example - again, not hard but time consuming<p>for harder problems I often find it useful when I debug something that I'm not familiar with, it usually give half of very generic advices but often suggest some valid starting points in investigation
(eg recently I had to debug openvpn server and I didn't touch them in years, issue was manifested with error about truncated packets and gpt was able to correctly suggest TLS negotiation)<p>sometimes I also use it rewrite/refactor code to see if maybe I would like different style/approach - this is rather rare, at begining I found it useful, now I think gpt started to often corrupt code or replace sections with comment with makes it much less useful
I have cancelled Copilot after trying for few months because sometimes it used to take forever to suggest code. Now I use Duet AI and it is code suggestion on steroids. Sometimes it suggests some good code and I feel how terrible I am for not thinking of that approach. But I think we are progressing fast with this AI things and it won't be long before everyone can code with these tools.
I feel you. In my experience the paid ChatGPT's answers does not outperform the basic google stackoverflow route for solved problems - which is not surprising considering how those models are created. For more complicated stuff, in my case, for example, weired Java dependency injection use cases, it gave me equally wrong answers I could found in the internet anyway. Only faster.
I don't know how to code. I wish I did, but it is a skill I have never been able to pick up. I am hoping that eventually, at some point, AI is able to be a zero code interface so I can create something with help of course.<p>I can't code, but I undersand coding concepts like variables, Booleans, etc.
> or learn more about some history stuff where it just repeats something it parsed off wikipedia or whatever.<p>Be careful with that too, it will also spit out whatever urban legend it read on a subject without making a difference between the facts it got on Wikipedia and the bullshit it read elsewhere.<p>Keep in mind, they are <i>language</i> models, not <i>knowledge</i> models.
I've had times where the AI saved me lots of minutes of typing. Example:
I have a piece of data that need to be mapped to an almost equal piece of data, but types differ slightly enough that I need to type the equality for each of them. Having it auto complete all of it (tens of lines) at once is such a relief.
it’s for satisfying the wet dreams of management in large companies dreaming of replacing developers with AI.<p>the issue is that it will take a long time before they wake up and scramble to re-hire the people they laid off. in the meantime i predict a lot of developers will have pivoted to other careers, retired, became AI devs etc.
You're not alone. This has been the experience of myself and nearly all the people on my team. Basically, everyone has stopped bothering with using LLMs for this purpose. In our experience, using these tools doesn't increase productivity, but does decrease code quality.
Since I don't see it mentioned here at all, I highly recommend trying out <a href="https://cursor.so" rel="nofollow">https://cursor.so</a> - GPT4 powered and absolutely helpful with refactoring code and adding new features, not just finishing the current line.
I agree, but I think of it more like a dirty work kind of thing, that it can do for me, like to repeat switch cases, instead of just copy pasting them, and change. I'd say its worth the 3-4% benefit it gives me.
I haven't used GitHub Copilot in awhile but I find ChatGPT is decent at generating the boilerplate for a new project but it does degrade the more I use it on that project. Often I have to either give up using it or start a new chat and give it my latest code.
I'm using it happily in all projects I'm involved but I'm not writing code often.<p>It helps tremendously.<p>Also when you look what Google is doing internally: gpt might not be perfect right now, but I would try to keep up with testing/using it regularly enough to not loose sight.
There are two use cases that work for me:<p>1. Complete tedious / repetitive code<p>2. Help with new/rarely used things. For instance, I'd ask it: write a function that fetches the price of the Apple stock from Bloomberg for the year 2023. And then I can use that a tweak it as I want.
The real advantages come when you move off of copilot or GPT onto something without their restrictions. Sure it can’t go pull code from github but it can give you more accurate results than the dumbed down prompts the SaaS providers give you.
Co-pilot is junk. The only thing that works passably for coding is ChatGPT using GPT-4 and prompts where you ask it write and then implement a software spec, stopping after each code section to ask for review and revisions.
I asked it for help with solving an issue with a small C function a few days ago, it suggested that instead of using `head.prev` I use `head.prev`. Not a typo, it just echoed my own code as a solution
It's hit or miss. Sometimes, I give it relatively complex requirements, and it spits out a running program. Worst case, it at least points me in the right direction.<p>I save well over an hour a month with it, so it's worth it.
If I have a linux problem and copy/paste relevant parts of the log into an AI, the result is more often than not useful and quick. It's a real help for admin tasks. Is it programming though ? :)
I can't tell if I'm imagining it, but I feel like the quality of ChatGPT has dropped for software dev.<p>I used to get very helpful responses but recently find myself correcting it a lot.
i turned off copilot when i did a trial of an alternative and when the trial ran out i just stopped using either. i still use chatgpt over stackoverflow but i don't miss the proactive prompts.
This is just another facet of the current problem with 3-month boot camp graduates thinking they are software engineers and technically illiterate managers hiring them for SWE positions. Of course, for that level of competence, GPT copilots are true marvels because the code quality is the same or even better.
I mean most of the time coding is not spent writing code or fast . its finding bugs or weird edge cases<p>some days i write 2 lines of code, or just try to find stuff. all this copilot talk feels like its never about real software development