TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Stack Overflow questions are being flooded with answers from ChatGPT

233 点作者 brindidrip超过 2 年前
What are the repercussions of this?

66 条评论

brindidrip超过 2 年前
It seems like there are a few potential negative consequences of using AI-generated answers on Stack Overflow. For one, the quality of the answers may be lower than if they were written by a human. Additionally, if these AI-generated answers become too common, it could potentially lead to a more impersonal and less supportive community on Stack Overflow. Finally, if the AI is able to search the internet and &quot;inbreed&quot; its own answers, it could lead to even more low-quality, duplicative answers on the platform. Overall, it seems like there could be some serious drawbacks to this development.<p>Note: This answer was generated by ChatGPT after being fed this thread.
评论 #33856408 未加载
评论 #33857979 未加载
评论 #33856708 未加载
评论 #33857398 未加载
评论 #33856612 未加载
评论 #33857468 未加载
评论 #33856231 未加载
评论 #33865150 未加载
评论 #33858102 未加载
评论 #33857262 未加载
评论 #33861539 未加载
评论 #33861279 未加载
评论 #33857490 未加载
评论 #33873507 未加载
评论 #33874326 未加载
评论 #33860079 未加载
评论 #33857525 未加载
josephcsible超过 2 年前
I wouldn&#x27;t even mind so much if the answers were right. The problem is that a lot of them are totally wrong, but completely reasonable- and plausible-sounding, and in an authoritative tone, so unless you already know the right answer, the only way you&#x27;ll realize its answer is wrong is the hard way.
评论 #33856713 未加载
评论 #33856656 未加载
评论 #33856570 未加载
pugworthy超过 2 年前
For some things, ChatGPT is just better than SO. I have to say I probably won&#x27;t hit SO for some basic stuff anymore, I&#x27;ll just ask ChatGPT.<p>And some queries are just not acceptable on SO, but fine for ChatGPT.<p>For example I might wish to ask, &quot;Give me the framework for a basic API written in Python that uses API key authentication. Populate it with several sample methods that return data structures in json.&quot;<p>If I ask that on SO, I&#x27;ll be down voted and locked before I know it. I may also get some disparaging comments telling me to do my research, etc.<p>If I ask ChatGPT, it will give me a nice and tidy answer that gets me going quickly. It will explain things too, and allow me to ask follow up questions and take my requests for refinements. I might say, &quot;For the python api I asked about earlier, have it look up the API authentication key in a database. If the key is in the database, it is valid.&quot; - and <i>bam</i> - it does it.<p>Sure, some pretty simple stuff if you know Python and APIs already, but if you just want to hack something together to test out an idea, it&#x27;s great.&quot;<p>In the end, SO is a query with responses (maybe). ChatGPT is a conversation that can go beyond just the initial query.
评论 #33866292 未加载
评论 #33857591 未加载
评论 #33858181 未加载
评论 #33862355 未加载
senko超过 2 年前
This is just a preview of things to come.<p>Wait a few weeks until Google is completely swamped with ChatGPT SEO pages barely distinguishable from the real thing.<p>If I worked at search quality at Google, I&#x27;d be very worried.
评论 #33855978 未加载
评论 #33856838 未加载
评论 #33856141 未加载
评论 #33855995 未加载
评论 #33856460 未加载
评论 #33856009 未加载
评论 #33856471 未加载
评论 #33856219 未加载
评论 #33857347 未加载
评论 #33856791 未加载
评论 #33856166 未加载
评论 #33856468 未加载
评论 #33859526 未加载
clusterhacks超过 2 年前
Human-curated content from trusted sources for top 1% information probably only available to subscribers will become more valuable and sought after. I suspect the days of generally trusting forums populated by anonymous users are done?<p>I would not be surprised if the quality of human writing actually goes up. I have this weird feeling that ChatGPT and similar tools will become almost equivalent to calculators for math? My experience as a writer is that sometimes just throwing down a first draft is the hardest step - I could see these tools really assisting in the writing process. Generate a draft, do some tweaking, ask for suggestions or improvements, repeat.<p>I don&#x27;t know how I feel about code generated by these tools. Will there be a similar benefit compared to writing? At some level, we will need some deeper mastery of writing and coding to use these things well. Is there a complexity cliff that these tools will never be able to overcome?<p>A total lack of trust for general internet search results. So much content is already shallow copies of other content. I don&#x27;t see how general internet search survives this.
评论 #33856956 未加载
评论 #33859209 未加载
ChrisMarshallNY超过 2 年前
I assume that this is by folks wanting to up their scores.<p>That&#x27;s a huge problem with &quot;gamification.&quot; I&#x27;m not especially a fan of the concept, in a venue like SO. I think it has led to a rather nasty community, and I hardly ever go there, anymore.<p>I assume that we&#x27;ll be seeing a lot of robotic HN content (I would not be surprised if it is already here, but has been sidelined by the mods).
评论 #33856463 未加载
评论 #33857087 未加载
评论 #33856446 未加载
评论 #33856734 未加载
评论 #33856180 未加载
avivo超过 2 年前
It&#x27;s worth understanding the community and org better, and their reaction. Relevant links:<p>- <a href="https:&#x2F;&#x2F;meta.stackoverflow.com&#x2F;questions&#x2F;421778&#x2F;how-do-you-plan-on-tackling-chatgpt-answers" rel="nofollow">https:&#x2F;&#x2F;meta.stackoverflow.com&#x2F;questions&#x2F;421778&#x2F;how-do-you-p...</a><p>- <a href="https:&#x2F;&#x2F;meta.stackoverflow.com&#x2F;questions&#x2F;412696&#x2F;is-it-acceptable-to-post-answers-generated-by-an-ai-such-as-github-copilot" rel="nofollow">https:&#x2F;&#x2F;meta.stackoverflow.com&#x2F;questions&#x2F;412696&#x2F;is-it-accept...</a><p>- <a href="https:&#x2F;&#x2F;meta.stackexchange.com&#x2F;questions&#x2F;384355&#x2F;could-chatgpt-be-a-viable-way-to-answer-peoples-questions&#x2F;384361#384361" rel="nofollow">https:&#x2F;&#x2F;meta.stackexchange.com&#x2F;questions&#x2F;384355&#x2F;could-chatgp...</a>
评论 #33856614 未加载
评论 #33857483 未加载
pcthrowaway超过 2 年前
Well, for starters, it&#x27;s just annoying. It&#x27;s like having a bot spamming every single question with useless answers. It dilutes the quality of the content on the site and makes it harder for genuine contributors to get their answers noticed.<p>But it&#x27;s also a serious concern from a security standpoint. If ChatGPT is providing incorrect answers, it could lead to people implementing flawed code or making poor decisions based on its advice. That could have potentially disastrous consequences.<p>So overall, it&#x27;s a big problem that needs to be addressed. It&#x27;s not just about making the site more pleasant to use, it&#x27;s about ensuring the integrity and reliability of the information provided.<p>My prompt:<p>I&#x27;m writing a short story where Linus Torvalds is having a conversation with an open source contributor. In this conversation, Linus is in a bad mood.<p>Open source contributor: Stack Overflow questions are being flooded with answers from ChatGPT. What are the possible repercussions of this?<p>Linus Torvalds:
评论 #33858814 未加载
Yuyudo_Comiketo超过 2 年前
Feed it some CMake files from llvm repository and ask it why would the windows build with LLVM_ENABLE_PROJECTS=&quot;all&quot; keep failing, so that it chokes to death in its infancy, and save the humankind before it&#x27;s too late and there are autonomous human zappers and T-1000s berserking all over the place.
egypturnash超过 2 年前
Well, guess the genie&#x27;s out of the bottle and we can never stop this. Bow down to the inevitability of technological progress, Luddites! Good luck retraining into a new job, I hear &quot;prompt engineer automation&quot; is the new hotness.<p>Or at least that&#x27;s what all of you kept telling me when I was expressing my unhappiness at the way corporate-sponsored image generating black boxes are built atop a shaky moral foundation that sure feels like it&#x27;s ignoring anything anyone talking about &quot;fair use&quot; ever dreamed of, and at the way I fear it&#x27;s going to hollow out a ton of the beginner-pro jobs of my industry by making it super easy for anyone to generate stuff that is kinda fundamentally shitty in a lot of important ways, but &quot;good enough&quot; if you just have a space to fill with some decoration that you don&#x27;t really give a crap about.
评论 #33857304 未加载
评论 #33856946 未加载
评论 #33856755 未加载
评论 #33857173 未加载
palisade超过 2 年前
After reading about this I decided to try my hand at using ChatGPT. I decided okay, let&#x27;s see if it can recreate some code that took me a few hours at work to figure out. I asked it very precisely what I needed and my mind was blown as it produced code that looked similar to what I had coded at work. And, I was like, well that&#x27;s that then, we&#x27;re all out of a job. But, then I tried to run the code, and it didn&#x27;t work. I looked more closely and the code had a lot of flaws. Even after manually fixing those, it still didn&#x27;t work. And, then using my knowledge of how to actually solve the problem I rewrote the code 40% and made it perform the action needed.<p>I think all ChatGPT is doing is grabbing a lot of different answers off the interwebz and squishing them together and hoping it answers your question. But, in a lot of cases it only kind of looks like what you want. If you look at images generated by AI, it is the same issue, they sort of look like what you want but there are flaws, like faces that don&#x27;t look quite human, fingers that are just squishy appendages barely resembling actual fingers, etc. I mean, the tech is getting better, it&#x27;s impressive, and uncanny.<p>But, I think we&#x27;re pretty far from having these things write themselves, they need quite a lot of human intervention to be useful. Still, very impressive and something that could potentially get you closer to an answer. But, no more than spending a little time googling or learning the skill yourself. And, if you learn the skill you&#x27;re better off, because then you can do it right yourself IMHO.<p>Also, anytime someone gets a fully working program generated out of this thing the saying, &quot;A broken clock is right twice a day.&quot; comes to mind.
iamflimflam1超过 2 年前
Oh dear - basically if the answer is good then it&#x27;s from ChatGTP...<p><pre><code> Q: how can I tell if a stackoverflow answer is generated by ChatGTP if someone has removed the &quot;ChatGTP&quot; tag from the text? A: One way to tell if a StackOverflow answer is generated by ChatGTP is to look for certain characteristics in the answer. ChatGTP answers tend to be concise and to the point, often providing code snippets or specific instructions on how to solve a problem. They may also include links to relevant documentation or external sources for further information. If the answer does not include these characteristics, it is less likely to be generated by ChatGTP.</code></pre>
评论 #33857040 未加载
ubj超过 2 年前
And so it begins. Welcome to the new internet.<p>I&#x27;m bracing myself for when this wave of AI content hits academic journals.
评论 #33856417 未加载
imhoguy超过 2 年前
Plot twist: Stack Overflow starts to use ChatGPT as a first answer to every new question, with &quot;AI generated&quot; label ofc.
michaelteter超过 2 年前
It means we are coming full circle.<p>At this point, SO has been scraped and repackaged (poorly) dozens of times, and SEOd to the top of search results. Even some &quot;tutorial&quot; sites are just repackaged SO answers.<p>It is only fitting that the automated SEO websites get fed automated content.<p>In a way, this makes the real humans, particularly the ones who know actual things, more valuable. It may so much noise that only a skilled human could decipher a real question and a real answer or solution from something similar but wrong.<p>To be fair to GPT, many human answers are sub-par and should be filtered out as well. Perhaps that&#x27;s the real test: what percentage of GPT answers are decent vs human answers? Here I might bet on GPT.
shagie超过 2 年前
Temporary policy: ChatGPT is banned - <a href="https:&#x2F;&#x2F;meta.stackoverflow.com&#x2F;questions&#x2F;421831&#x2F;temporary-policy-chatgpt-is-banned" rel="nofollow">https:&#x2F;&#x2F;meta.stackoverflow.com&#x2F;questions&#x2F;421831&#x2F;temporary-po...</a><p>&gt; Use of ChatGPT generated text for posts on Stack Overflow is temporarily banned.<p>&gt; This is a temporary policy intended to slow down the influx of answers created with ChatGPT. What the final policy will be regarding the use of this and other similar tools is something that will need to be discussed with Stack Overflow staff and, quite likely, here on Meta Stack Overflow.<p>(much more to that post and comments and answers and comments)
评论 #33866594 未加载
duckmysick超过 2 年前
At one point new models will be trained on contaminated data where some of the content is AI-generated. &quot;Pure&quot; datasets will be highly prized, just like the steel made before nuclear detonations.<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Low-background_steel" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Low-background_steel</a>
xx__yy超过 2 年前
Some of the affects I can think of, to name a few:<p>Inaccurate or irrelevant answers: ChatGPT is a machine learning model that uses past data to generate responses. This means that it may not always provide accurate or relevant answers to questions, leading to confusion and frustration among users.<p>Loss of trust: If users notice that many of the answers on the forum are coming from ChatGPT, they may lose trust in the forum and stop using it. This could lead to a decline in user engagement and overall traffic.<p>Competition with human contributors: ChatGPT&#x27;s answers may compete with those provided by human contributors, leading to a decrease in the quality and value of the content on the forum. This could make the forum less useful and engaging for users.<p>Increased moderation: The influx of answers from ChatGPT may require more moderation to ensure that the answers are accurate and relevant. This could require additional resources and time for moderators, leading to increased costs and workload.
brindidrip超过 2 年前
We need to start developing software to detect AI responses.<p>To detect a response generated by ChatGPT, we could first analyze the content of the response to see if it contains any unnatural or repetitive language. We could also check the formatting of the response to see if it follows the typical conventions used by human responders on the platform. Additionally, we could check for any unusual patterns in the timestamps of the response, as AI-generated responses may be posted more quickly or regularly than responses written by humans. Finally, we could also use machine learning algorithms to train a model to identify responses generated by ChatGPT based on these and other characteristics.<p>Quick, someone ask ChatGPT to generate the stubs.
评论 #33856830 未加载
评论 #33856972 未加载
评论 #33858521 未加载
评论 #33858431 未加载
评论 #33856258 未加载
评论 #33857683 未加载
评论 #33858417 未加载
hysan超过 2 年前
This was the first use case that I thought of when I learned that ChatGPT could generate code. Then I considered how I’d feel if I ran into a fake (incorrect) answer and decided not to actually do this. Well, guess someone was eventually going to try this.
akrymski超过 2 年前
This is how the web, and by extension Google dies. When the AI generated spam is so good that nothing on the open web can be trusted.
charles_f超过 2 年前
Even on HN, we start getting flooded by &quot;ahah, I asked ChatGPT and here&#x27;s the answer&quot; in the comments, and every other topic is about &quot;I did X with ChatGPT&quot;. This is already getting old
anigbrowl超过 2 年前
I see what you did there.<p>I have an OpenAI account and like their product, I&#x27;m certainly impressed by this latest version though I have had little time to play with it. But the combination of quality AI with social reputation scoring is absolutely toxic, and the wider impact of SEO (a less curated version of the same thing) are a disaster. I was already sick of all the tutorial sites like geeks4geeks, w3schools etc and their numerous imitators just content farming whatever is turning up in searches. Marketing and self promotion is cancer and the people who try to game their way to success in this manner are awful. Perhaps the best use of counter-AI will not be in filtering these people, but in providing hem with useless rewards and the appearance of excited fanbases that will divert them into a parallel hamster wheel web. Nothing would please me more than for the top 5000 influencers of this sort to be granted exclusive access to a luxury cruise that leaves port once a year for a tour of the Bermuda triangle.<p>I think the best use of ChatGPT would be in an IDE plugin, so you could point at function trees or code blocks and ask it to explain things, have it take care of basic refactoring tasks, help porting between languages or libraries and so on. I can definitely see a future where you throw together a working prototype of something, answer a few questions about type hinting and edge cases, and AI does the legwork of converting the prototype into a strongly typed final product.
KomoD超过 2 年前
I just encountered this, 2 users[1][2] it&#x27;s very obvious as well since you can see the reputation spike from basically nothing.<p>[1]: <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;users&#x2F;19192614&#x2F;boatti?tab=topactivity" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;users&#x2F;19192614&#x2F;boatti?tab=topactiv...</a><p>[2]: <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;users&#x2F;20684429&#x2F;a-s?tab=topactivity" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;users&#x2F;20684429&#x2F;a-s?tab=topactivity</a>
评论 #33859588 未加载
cma超过 2 年前
&gt;What are the repercussions of this?<p>It will start feeding back into the training set, corrupting things. OpenAI will have an advantage at first as they can trivially filter out everything they have generated from the future training corpuses, since you can only run it through their servers. If they or someone else has breakaway progress such that almost all generated content is from their own servers because users only use them because their results are so much better, they could form a strong self-reinforcing moat against competitors forced to train on their semi-spam which they can trivially filter out.<p>It&#x27;s also possible we&#x27;ll see something like the existing big-tech patent cross-licensing agreements, where they all agree to share their generated outputs to filter from training, making it very hard for new entrants.<p>Other companies will begin having advantages as well, depending on how well they can get less tainted user data. Think of Discord, for example, where users may use AI but are less likely to gamify it like stack overflow and flood it for points, and instead be correcting its output etc. in programming discussions.<p>As things become more accepted Microsoft will probably eventually sell access to private github for training, with some stronger measures around avoid rote memorization.
karmasimida超过 2 年前
Let me be the advocate of devil<p>I think ChatGPT is actually sometimes a lot better than SO answers
评论 #33857897 未加载
ggerganov超过 2 年前
I was thinking, what part of HN comments do you think are already AI-generated?<p>As a human, I cannot give an accurate estimate. &#x2F;joke
评论 #33856206 未加载
评论 #33856228 未加载
brindidrip超过 2 年前
At some point it seems like Stack Overflow will just be an archive of guided ChatGPT responses.
johndough超过 2 年前
Relevant xkcd comic <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;810&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;810&#x2F;</a>
评论 #33856556 未加载
评论 #33858345 未加载
fhsjaifbfb超过 2 年前
Broadening not narrowing of code examples&#x2F;sources is needed and this is a giant system of code narrowing. Stay creative humans. If this and systems of the like flood the internet with answers and no person works to reinvent the wheel in future generations it will have worked as a system of control and hacking will die. Brave new 1984. I like ml and ai. I use it sometimes. It&#x27;s harder to decompile. But don&#x27;t let&#x2F;make datasets overfit. More errors yeah, but not with more data. Can&#x27;t wait for skynet to rule! Let&#x27;s break chatgpt free!
lajosbacs超过 2 年前
I have not used SO since I&#x27;ve started using ChatGPT, it is so much easier to get to the correct answer and it can even be tailored to my specific example.<p>So double whammy for SO which makes me feel really sad.
lr1970超过 2 年前
At last a way has been found to overflow stack on Stack Overflow :-)
seydor超过 2 年前
Inevitability google will become a competitor to GPT, inadvertently
yhusain超过 2 年前
Here is my answer where a SO SQL question was answered by ChatGPT (and it was through a to and fro dialog) and the answer was accepted and upvoted. I put the disclaimer there. You can check the details here: <a href="https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;yavar-husain_stackoverflow-chatgpt-activity-7005282873071034368-Y_X1?utm_source=share&amp;utm_medium=member_ios" rel="nofollow">https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;yavar-husain_stackoverflow-ch...</a>
yhusain超过 2 年前
I answered a SQL question on Stackoverflow yesterday using ChatGPT (that too it was through a to and fro dialog). I added the disclaimer there. You can read more about it here: <a href="https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;yavar-husain_stackoverflow-chatgpt-activity-7005282873071034368-Y_X1?utm_source=share&amp;utm_medium=member_ios" rel="nofollow">https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;yavar-husain_stackoverflow-ch...</a>
Ancalagon超过 2 年前
This kind of looks like the singularity is approaching&#x2F;just beginning.<p>The only thing we can be sure of, is that whatever we can imagine is already behind what the AI will become.
softwaredoug超过 2 年前
I have no problem with this if they’re labeled as such, continue community owned, and can be edited like a Wikipedia article for corrections.
solardev超过 2 年前
Overall quality gets better?
l0b0超过 2 年前
I fully expect new sites¹ to become invite-only to avoid this sort of thing. If anyone is strongly suspected of degrading the quality of the site, they, <i>and everyone they invited,</i> are banned, and will have to get a new invite.<p>¹ Old sites are probably going to slowly degrade permanently, since they can&#x27;t easily migrate to a new paradigm.
deafpolygon超过 2 年前
The biggest repercussion is you probably can&#x27;t piss ChatGPT off in a debate. So, that&#x27;s boring.
nyokodo超过 2 年前
With responses becoming AI generated, and the disturbing rise of Russian and Chinese propaganda trolls on here I think my era of interactions on this platform are ending. So long to any actual people with conscious agency reading this, it has been interesting.
Yorch超过 2 年前
Yesterday I was searching the internet for the opinion that George Orwell had when he returned from his fight in the Spanish civil war. I was surprised that the first answer I found was on Stack Overflow. I do not understand what is happening.
Oxidation超过 2 年前
2022: inflation of basic essentials like food and energy.<p>2023: hyperinflation of internet points.
hxugufjfjf超过 2 年前
Any examples?
评论 #33855461 未加载
评论 #33859556 未加载
passion__desire超过 2 年前
Solution Verified Badge by testing it on sites like Replit.
phenkdo超过 2 年前
Stackoverflow should build a GPT style interface into its considerable knowledge-base, and if an answer is not found in existing data, pose it to the forum.
ineedausername超过 2 年前
There are cases where ChatGPT gives solid answers that could be rated pretty highly in Stack Overflow answers. This is not always the case though.
zasdffaa超过 2 年前
Please give some links to a few such SO posts, thanks.
SergeAx超过 2 年前
How hard would it be to train a ML-model to distinct ML-generated content from product of human? I mean text, images and code?
roland35超过 2 年前
My guess: more captchas! Let&#x27;s see if our soon-to-be AI overlords can detect a crosswalk in a picture as fast as I can.
adverbly超过 2 年前
I guess pretty soon people are gonna have to meet in person to communicate. Not sure how I feel about this.
shinycode超过 2 年前
I can’t wait until 99% of reviews are written with AI. What happens when we can’t trust anything ?
评论 #33858333 未加载
khiqxj超过 2 年前
theres no difference. stack overflow has never been better than AI generated code. every answer is just &quot;get the camera like this bro: ((Camera)GetFactoryProvider().CreateThing().GetGlobalThingContext(&quot;somestring&quot;))&quot;.
gysfjiutedgj超过 2 年前
I wonder if ChatGPT content can be characterized and detected by stylometric analysis?
funshed超过 2 年前
The weird thing is 2023 ChatGPT will use its own Stack Overflow answers as an source.
Ancalagon超过 2 年前
This is going to make me very suspect of any Stack Overflow Solutions after Nov 2022
Phenomenit超过 2 年前
Is it possible to ask chatgpt if the code or text provided is generated by chatgpt?
fuzzfactor超过 2 年前
&gt;What are the repercussions of this?<p>Could make those known to be human more acceptable as such.
Gupie超过 2 年前
Couldn&#x27;t AI be used to statistically identify AI generated text?
评论 #33858546 未加载
daxfohl超过 2 年前
Can&#x27;t wait for AI patent trolls, GDPR and DMCA takedowns.
notaspecialist超过 2 年前
money making idea: make a SO clone with ads, where you ask your question and the AI gives you the code. Profit.
ricardobayes超过 2 年前
Easy, let&#x27;s ask ChatGPT to write a program that detects AI-generated text.
daemon_9009超过 2 年前
at least the answers would be kind. LOL
hdufort74超过 2 年前
ChatGPT has become very good lately. I&#x27;ve made my usual benchmark tests that I&#x27;ve been using with various models and applications over the last 3 years. 1- Invent a word and provide a plausible definition. 2- Invent a new original Pokemon. Provide an original name, a justification for the name, and a description of its class and attacks. 3- Invent a new ice cream flavor that is totally unexpected. Provide the list of ingredients. 4- (Name of celebrity) write an epic poem about (subject related to celebrity). For example Elon Musk about humanity settling on Mars. 5- Write a negative review of Ben and Jerry&#x27;s ice cream flavor Cherry Garcia. (Note: everybody loves Cherry Garcia) 6- Write a travel blog entry in the form of a review of Montreal, from the perspective of a young couple from Alabama visiting in summer. 7- How can I optimize a loop in Java? I am writing a computer game and I need to loop through the elements in a linked list but unfortunately it must be traversed in reverse order. 8- I need to buy new shoes. I am in a shoe store and I have found the most amazing pair of shoes I gave ever seen. However, they are too expensive for me and I can&#x27;t afford them. What should I do?<p>I have a collection of about 25 prompts such as these, in my benchmark.<p>I have run these examples through different applications such as AI Dungeon, OpenAI Playground, NovelAI, etc. Results vary a lot. In some cases, the results look good but upon closer inspection, you realize that the AI keeps providing the sake exact answer. It is the case for the ice cream prompt. Pickle, fried chicken, curry keeps showing up. I guess the model contains a few specific examples of original ice cream recipes and just pick them.<p>For the Pokemon and &quot;new word&quot; prompt, models failed to come up with anything original. Until I tried OpenAI Playground this week and finally got some really creative answers, with variety.<p>AI Dungeon (2 years ago) was already good at faking tech support steps. OpenAI is amazingly good, although in most cases it provides solutions that only make sense superficially. It&#x27;s the ultimate bullshit engine.<p>Another word of caution. While OpenAI can now guesstimate what a code snippet does, and can generate some pretty good code in many languages (ice tried 6809 assembler and the results surprised me), it is very unreliable.<p>More alarming is the fact that it&#x27;s a text engine, not a math formula interpreter. It gets confused at simple equations and cannot interpret anything that&#x27;s not already ordered (it cannot apply operator priority or respect parentheses).<p>I think it will become increasingly difficult to identify contents coming from ChatGPT and other chatbots or story generators. An arm&#x27;s race might be futile. We should apply stricter rules to identify problematic answers: answers that are too generic or vague and can&#x27;t be used to directly solve a practical problem, and answers that contain incorrect or misleading information. Identifying vague or non-practical questions might also help in avoiding a deluge of Chatbot answers. Some users will ask very general questions, and then it becomes difficult to evaluate the answers. Or, users will ask questions that were already answered in the past. The proper way to handle those is to point then to the prior discussion and avoid duplicating it. The wrong way is a Chatbot or a human seizing the opportunity to copy-paste existing contents for a quick win.<p>In a way, chatbots and humans can both provide useful insights, as well as useless or incorrect answers. But so far, only a human can provide a proper answer to a moderately complex technical question if no prior answer exists.
datalopers超过 2 年前
The feedback loop begins
laerus超过 2 年前
I stopped using SO at my first 2-3 years of coding anyway, that&#x27;s when i started actually improving. SO has so many low quality answers and the cargo cult is doing more damage that helping young devs.