I'm tired of fixing customers' AI generated code

&gt; Helping a customer solve challenges is often super rewarding, but only when I can remove roadblocks for customers who can do most of the work themselves.<p>One thing I loved about doing technical enterprise sales is that I’d meet people doing something I knew little or nothing about and who didn’t really understand what we offered but had a problem they could explain and our offering could help with.<p>They’d have deep technical knowledge of their domain and we had the same in ours, and there was just enough shared knowledge at the interface between the two that we could have fun and useful discussions. Lots of mutual respect. I’ve always enjoyed working with smart people even when I don’t really understand what they do.<p>Of course there were also idiots, but generally they weren’t interested in paying what we charged, so that was OK.<p>&gt; Helping a customer solve challenges is often super rewarding, but only when I can remove roadblocks for customers who can do most of the work themselves.<p>So I feel a lot of sympathy for the author — that would be terribly soul sucking.<p>I guess generative grammars have increased the number of “I have a great idea for a technical business, I just need a technical co founder” who think that an idea is 90% of it and have no idea what technical work actually is.

I had a related episode at work when my coworker asked me why his seemingly trivial 10 line piece of code was misbehaving inexplicably. It turned out he had two variables `file_name` and `filename` and used one in place of another. I asked him how he ended up with such code, he said he used copilot to create it. Using code from a generative AI without understanding what it does is never a good idea.

I&#x27;m always a little surprised at how many people out there want to develop software yet haven&#x27;t put in the effort to gain even the most basic computer nerd programming chops. You see this all the time in the more newbie-friendly game engine communities.<p>Maybe you don&#x27;t want to pursue a career in software, but anyone can spend a week learning Python or JavaScript. I suspect&#x2F;hope a lot of these people are just kids who haven&#x27;t gotten there yet.

Cryptocurrent trading tools? The susceptibility of people to get-rich-quick scams and the desire to not do even the minimum of work is surely correlated.<p>Stop poisoning the well and then complaining that you have to drink from it.

I would suggest adding a help section with advice for people using ChatGPT that sets expectations and also gives a pre-written prompt for them to use.<p>Something like.<p>Some of you may use ChatGPT&#x2F;Copilot to help with your coding. Please understand this service is designed for professional programmers and we cannot provide extensive support for basic coding issues, or code your app for you.<p>However if you do want to use ChatGPT here is a useful starting prompt to help prevent hallucinations - though they still could happen.<p>Prompt:<p>I need you to generate code in ____ language using only the endpoints described below<p>[describe the endpoints and include all your docs]<p>Do not use any additional variables or endpoints. Only use these exactly as described. Do not create any new endpoints or assume any other variables exist. This is very important.<p>Then give it some examples in in curl &#x2F; fetch &#x2F; axios &#x2F; python &#x2F; etc.<p>Maybe also add some instructions to separate out code into multiple files &#x2F; endpoints. ChatGPT loves to make one humungous file that’s really hard to debug.<p>ChatGPT works fairly well if you know how to use it correctly. I would defo not trust it with my crypto though, but I guess if some people wanna that’s up to them. May as well try and help them!

Someone mentioned &quot;hallucination-based API design&quot; on twitter (I cannot find it now). It&#x27;s designing API by LLM hallucination. If there is a common hallucination API call, just add that API. This will make the API more &quot;friendly&quot; and resemble common similar APIs.<p>Considering that LLM can hallucinate in different ways unpredictably, not sure whether it will work in practice.

I can guarantee that all these users are following some &quot;hustle university&quot; course peddled by a Twitter influencer. Crypto and AI are the two favorite words of all these get rich quick scams.

I‘m tired of fixing my colleagues‘ AI generated code. They churn out a lot of code in a short amount of time, but then we loose the saved time again because during pull request review they often cannot explain what this code is actually doing. Maybe I should use an AI for code review, too?

A tiered approach to sales can help here.<p>Cheap version offers minimal support. (Although you still have to sift &quot;bug reports&quot; into my problem&#x2F; your problem.)<p>Standard version allows for more support (but still rate limited.)<p>Developer version charges for time spent, in advance.<p>This helps because people only expect free support if you don&#x27;t explicitly offer something else. If you offer paid support them it&#x27;s reasonable and expected that there are limits on free support.

&gt; Often this takes the form of trying to access an endpoint that does not exist, or read a property off the API response that does not exist. After probing a bit more, my suspicions are usually confirmed — ChatGPT hallucinated that endpoint or property<p>In some cases, you might be able to use this to your advantage to improve the product.<p>When working with third-party APIs, I&#x27;ve often run into situations where my code could be simplified greatly if the API had an extra endpoint or parameter to filter certain data, only to be disappointed when it turns out to have nothing of the sort.<p>It&#x27;s possible that ChatGPT is &quot;thinking&quot; the same thing here; that most APIs have an X endpoint to make a task easier, so surely yours does too?<p>Over time I&#x27;ve sent in a few support tickets with ideas for new endpoints&#x2F;parameters and on one occasion had the developer add them, which was a great feeling and allowed me to write cleaner code and make fewer redundant API calls.

I&#x27;ve been saying for a while now that there&#x27;s an absolute gold mine waiting for people who want to specialize in fixing AI generated applications.<p>A lot of businesses are going to either think they can have generative AI create all of their apps with the help of a cousin of the wife of the accountant, or they unknowingly contract a 10x developer from Upwork or alike who uses generative AI to create everything. Once they realize how well that&#x27;s working out, the smart ones will start from scratch, the not-so-smart will attempt to fix it.<p>Develop a reputation for yourself for getting companies out of that pickle, and you can probably retire early.

Another concern is around reviewing it.<p>I can&#x27;t tell in a pull request what someone wrote themselves, or to what level of detail they have pre-reviewed the AI code which is now part of the pull request before allowing it to get to me. I can tell you I don&#x27;t want to be fixing someone else&#x27;s AI generated bugs though... Especially given that AI writes more&#x2F;less dry&#x2F;more verbose code, and increases code churn in general.

Seems to me like you have an opportunity to develop a couple SDKs in your customers&#x27; favorite languages (probably python and typescript) and a simple &quot;Get Started&quot; template that could alleviate a lot of these requests. Show them the base case, let them figure out how to work with your api via an SDK instead of directly with an API and let the advanced users build their own SDKs in whatever language they prefer. Since its, as OP claims, a simple HTTP API, the SDK could be generated with OpenAPI client generation tools.

This seems like a support issue, not an AI issue. AI is how the code was written, but the issue would be the same if it was amateurs writing bad code. If all you want to do is support your API, a support article outlining the issues you see over and over would be something to point your customers to. Warrant your API against errors, but point out that anything more is billable work. If you&#x27;re not interested, partner with someone to do that work. You could even offer support contracts that include some amount of customization.

I have a business fixing broken code&#x2F;systems (especially if it is stressful and last minute); if you are tired&#x2F;annoyed of something in the software market, just up your fees. For us not much changed; a lot of badly (throw the spec over the wall) outsourced software was fairly bad since forever; AI generated code is similar. I guess this will grow even faster though, as normally solid developers might take on a lot more work and get sloppy with AI.

There&#x27;s room for all of us in this industry. What someone is unwilling to do is just an opportunity for someone else to pick up the yoke.

I can empathize, but am also wondering if some of these are feature requests in disguise. For “how to call api” docs, sample code, and even client libraries can be generated from your OpenAPI specs. Link to the docs in every reply. The more complex asks could be translated “build this endpoint and charge me for it”. If all else fails, set up partnership with devs who want to fix&#x2F;build customers crap and figure out some copy to direct the more complex asks to them.

Hilariously, the target market for the author&#x27;s API seems to be the same as the top post on HN today[0]: &quot;traders&quot;.<p>I think amateur &quot;trading&quot; attracts a specific brand of idiot that is high&#x2F;left on the Dunning Kruger curve. While taking money from idiots is a viable (even admirable) business strategy, you may want to fully automate customer service to retain your sanity.<p>[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41308599">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41308599</a>

lol this is just like trying to help people in programming discords that are literally using AI on screen to write and rewrite the entire app as they go. then they run into an issue, ask for help, and don&#x27;t understand when you say &quot;there&#x27;s a memory leak 50 lines down&quot; or &quot;you have to define that variable first&quot;.<p>AI is a great tool to help someone start an idea. When it goes past that, please don&#x27;t ask us for help until you know what the code is doing that you just generated.

I&#x27;m on the other side of this when it comes to the C programming language. I&#x27;ve avoided it for decades, preferring Pascal, or even Visual Basic.<p>The single best thing to help someone in my place is clear and coherent documentation with working examples. It&#x27;s how I learned to use Turbo Pascal so long ago, and generally the quickest way to get up to speed. It&#x27;s also my biggest gripe with Free Pascal... their old help processing infrastructure binds them to horrible automatically generated naming of parameters as documentation, and nothing more.<p>Fortunately, CoPilot doesn&#x27;t get impatient, and I know this, so I can just keep pounding away at things until I get something close to what I want, in spite of myself. ;-)

There&#x27;s already a big market for taking out AI garbage, and it expect it to grow as the AI bubble bursts. The best thing a consultant can do today is learn the common issues and failure modes of AI generated code.<p>Providing an API service to customers means you will get terrible client code. Sometimes it&#x27;s dumb stuff like not respecting rate limits and responding to errors by just trying again. One option, if you don&#x27;t want to fix it yourself, is to partner with a consultant who will take it on, send your customers to them. Bill (or set your API access rates) appropriately.<p>Sometimes you have to fire customers. Really bad ones that cost more than they bring in are prime candidates for firing, or for introducing to your &quot;Enterprise Support Tier&quot;.

I always make gpt fix the code it created. How else would I learn.

Yeah so every tech revolution does this right.<p>ATMs were meant to kill banking jobs but ah theres more jobs in banking than ever.<p>The Cloud was meant to automate away tech people, but all it did was create new tech jobs. A lot of which is cleaning up after idiots who think they can outsource everything to the cloud and get burned.<p>LLMs are no different. The &quot;Ideas Man&quot; can now get from 0 to 30% without approaching a person with experience. Cleaning up after him is going to be lucrative. There are already stories about businesses rehiring graphic designers they fired, because someone needs to finish and fine tune the results of the LLM.

&gt; Often this takes the form of trying to access an endpoint that does not exist, or read a property off the API response that does not exist. After probing a bit more, my suspicions are usually confirmed — ChatGPT hallucinated that endpoint or property<p>This is an opportunity. Add another tier to your pricing structure that provides an AI that assists with coding the API connection. Really simple Llama 3.1 that RAGs your docs. Or perhaps your docs fit in the context already if the API is as simple as it sounds.

As the author says, the errors are very easy to fix. Easy enough for GPT! He should set up a support chatbot. Only seems fair? ;)<p>I&#x27;m half joking, but in most cases I found that GPT was able to fix its own code. So this would reduce support burden by like 90%.<p>I hate chatbot support as much as the next guy, but the alternative (hiring a programmer to work as a customer support agent on AI generated code all day?) sounds not only cruel, but just bad engineering.

OP should set up an AI chatbot to triage his customer support. It probably wouldn&#x27;t be that hard to send that code right back to GPT and get a fix suggestion to the customer instantly. Stick your documentation for each endpoint in a vector database, and use RAG to give them a quick fix. If it doesn&#x27;t work let them go to level 2 support <i>for a fee</i>.

This is only the beginning. Imagine this when AI bot chains and &quot;agents&quot; replace conveniently junior devs on scale. Someone will &quot;hack&quot; an API&#x2F;Lang Chain&#x2F;Insert-LLM-Framework Solution.<p>The next decade of support business is here. Fix my &quot;hallucination&quot; market. Thank you, Microsoft. You did it again.

The customer support issue with selling SaaS is not just about AI, I think. It&#x27;s widespread across the industry. There are customers who have almost no idea about how their software works, and expect you to guide them through their own specific setups. One issue we <i>regularly</i> have is people complain that our TLS certificate keeps changing and that they want us to notify them in advance of any change. We try to tell them that two&#x2F;three-month duration TLS certificates is a best practice and that they should ensure that their trust store knows about standard root certificates. Then it turns out their software only supports <i>one</i> root cert at a time and they want us to jump into a call and guide them through changing it in <i>their</i> software.

If you are building an API and have decent docs, it&#x27;s a totally ok trade off to say &quot;i&#x27;ll lose some customers this way, but I&#x27;m not providing support&quot;. And just be upfront about it. Some stores have a no return policy with no exceptions. They lose some customers, it&#x27;s ok.

I think this is the point to establish a &#x27;community&#x27;, and in particular to only &#x27;offer&#x27; community support (or charge more for spending your own time on it).<p>Some people will enthusiastically fix other customers&#x27; generated crap for free, let them.

I&#x27;m wondering if he could just adjust the API to give an error message that an LLM (or a person) could understand and then correct. It might be worth putting some effort into beefing that up, to cut down the support requests.

I&#x27;m tired of fixing my own AI generated code.

Well, if you are annoyed by a particular maintenance task related to your business, find a way to automate it!<p>In this case, you could create examples for your API in common programming languages, publish them on the product website, and even implement an automatic test that would verify that your last commit didn&#x27;t break them. So, most of the non-programmer inquiries can be answered with a simple link to the examples page for the language they want.<p>As a bonus point, you will get some organic search traffic when people search for &lt;what your API is doing&gt; in &lt;language name&gt;.

&gt; The worst is when a request starts out simple — I help them fix one hallucination — but then that customer wants to build more complex logic, and somehow I’ve set the expectation that I will provide unlimited free support forever. I’ve gotten a number of angry messages from customers who essentially want me to build their whole app for free.<p>Annoying, but a good way to weed out bad customers. Life’s too short to deal with people who feel this entitled.

I am no partiuclar fan of AI (and actually haven&#x27;t used copilot or similar myself ever), but the real problem here is not AI but:<p>&gt; I’ve gotten a number of angry messages from customers who essentially want me to build their whole app for free.<p>I guess AI maybe encourages more people to think they can build something without knowing what they are doing. i can believe that for sure. There was plenty of that before AI too. Must be even worse now.

&gt; My API is just a few well documented endpoints. If you can figure out how to send a POST request using any programming language, you should have no problem using it. But that seems to be too high a bar for the new generation of prompt-engineer coders.<p>Nice. The time has come that we need to design the API with the LLM in mind. Or, to rephrase that, test your API that it is working with the popular LLM tools regularly.

I often think about code when I&#x27;m driving. I&#x27;ve thought about a way to brainstorm code ideas out loud and have the llm generate a code file with my idea. I&#x27;m talking about a fairly granular level, where I&#x27;m specifying individual if blocks and the like. Then when I have my hands available I can correct the code and run my idea

Every problem hides opportunities! I can see a future where documentation will be replaced by a plugin to an AI code service. Instead of providing users with documentation on how to use the package, devs will be training an LLM on how to assist the user in generating the interface code. An elaborate Chat GPT prompt for instance.

Retail &quot;traders&quot; are the textbook definition of mentally challenged obnoxiousnes. Go meet them on forums like EliteTrader.com and you will soon realize who you are dealing with.<p>It&#x27;s your fault really. You don&#x27;t build custom software for guys having the intellectual capacity and budget of a tractor driver unless you enjoy pain.

This whole thing reminds me of the dotcom boom where people who learned a week or 2&#x27;s worth of HTML were suddenly &quot;programmers&quot;, and were being hired because they could put on their own pants more often than not.

&gt; The worst is when a request starts out simple — I help them fix one hallucination — but then that customer wants to build more complex logic, and somehow I’ve set the expectation that I will provide unlimited free support forever. I’ve gotten a number of angry messages from customers who essentially want me to build their whole app for free.<p>Welcome to the life of any consultant&#x2F;support agent&#x2F;open source maintainer ever. AI isn&#x27;t the problem here, managing expectations is.

I wonder if there&#x27;s any consistent pattern to the API hallucinations.

The game might change (a little) when it can actually run the code it generates, examine any errors and potentially fix based on that.<p>And when you use a language that has better checks all around.

As a basic offering have good documentation and a community support forum and the option to report bugs.<p>Make the type of support you are providing paid, this could be in tiers.<p>Outsource support as needed.

We have come full circle: Silicon Valley is finally disrupting the software engineering industry...

I will take all of these customers. Please forward them to me.

AI is going to be like any other tool. If you don&#x27;t know how to use it, you may end up hurting yourself.<p>If you know how to use it, it will make you 100x more efficient.

Set firm boundaries and reject&#x2F;ghost customers that want to build their app for free and are &quot;angry&quot;. Those never lead to anything profitable.

The crypto product in question is for a very young age group.<p><a href="https:&#x2F;&#x2F;pump.fun&#x2F;board" rel="nofollow">https:&#x2F;&#x2F;pump.fun&#x2F;board</a>

Seems like the opportunity here is for the author to sell an AI that is trained on the API and documentation.

This is totally understandable, valid, etc.<p>OTOH, see script kiddies, WYSIWYG Stack Overflow C&amp;P, etc.<p>It&#x27;s just the way things are now.

Could you make a model from your API&#x2F;docs and let them plug that into their AI stuff? That would be funny.

&gt;I&#x27;m tired of fixing customers&#x27; AI generated code<p>Well, that didn&#x27;t take long . . .

and they thought AI would put SWE&#x27;s out of work...<p>If there are more people able to prototype their ideas with AI, sooner or later they will need a real engineer to maintain&#x2F;improve that code.<p>That will lead to more jobs and higher demand for SWE&#x27;s.

is there a way we can make a support bot or a chat-based document that is fine tuned and limit the answer to only what&#x27;s in the API? Getting the users to use it is another issue but one problem at a time.

Charge for engineering support and hire someone to do it for you!

Same, but for academic documents.<p>It used to be that a &#x27;bad&#x27; document had things &#x27;missing&#x27;, which were easy to spot and rephrase &#x2F; suggest improvements.<p>Now a &#x27;bad&#x27; document is 50 pages of confusing waffle instead of 5, you need to get through the headache of reading all 50 to figure out what &#x27;sentence&#x27; the person was trying to express at the place where an autogenerated section has appeared, figure out if the condensed document is something that actually had any merit to begin with, THEN identify what&#x27;s missing, and then ask them to try again.<p>At which point you get a NEW 50-page autogenerated document and start the process again.<p>F this shit.

&gt; Often, though, the customer is envisioning a more complex application, and I just have to tell them, “Sorry, you’re going to have to hire a professional developer for this.”<p>Welcome to enterprise sales, where building custom solutions atop of your sprawling BaaS empire is exactly what you do.<p>Your customers have a need and you’re basically turning down an upsell opportunity that also buys you lock in to your platform.<p>Another way to look at it: are they all trying to build something similar or something with similar components? Maybe it’s time to expand the features of your product to reduce the amount of work they need to do.

So have an AI do it for you!

Welcome to the hell that is being a senior dev nowadays. Every junior with a Copilot license all of the sudden fancies themselves an expert programmer now. It&#x27;s absolutely maddening.

AI, Rust, crypto, Medium, SAAS startups<p>It&#x27;s a fuckin perfect hacker news bingo.<p>Magnificent!

now you probably should build an AI to fix AI generated code based on your documentation.

Do you want to outsource this L1 L2 or even L3+ support? Lets talk

I&#x27;m not! Email in bio.

Why don&#x27;t you use AI to provide support? I&#x27;m serious actually. This sounds like something AI can do really well.

I&#x27;d be tempted to just roll with the AI generated endpoints&#x2F;hallucinations. If it&#x27;s presenting me statistically probable names that make sense after absorbing the world&#x27;s knowledge, I&#x27;m tempted to lean into that, instead of insisting that I have it right. Correct names don&#x27;t succeed as often as useful names do.