AI solves Advent of Code 2022

This day I asked it not too fundamental questions about Clojure and it was able to provide impressive, accurate answers and provide correct code examples. However if you continue the dialogue and ask it to do more advanced stuff, it will just make up stuff out of thin air. For instance it will use functions that don't exist and claim that they can be imported from packages that don't exist or don't have them. Once you point out these mistakes, it will admit them and come up with different changes which can be even worse, but sometimes also be better and save the whole thing. Overall I'm not sure how useful this will turn out, given that its not reliable. It may be useful to get some initial intuitions and informations (non specific stuff it usually gets right), but it can also mislead you badly. I asked it, how it makes these mistakes only to understand them and admit them once I point them out. It has no answer beyond the usual "I'm a language model". It also told me that it is capable of logical inference, but denied that the next day. Then it told me that its answers would always be consistent, which is a lie. The whole thing is really weird, because its somewhat very smart and capable and incredibly stupid and dishonest at the same time.

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems extremely unrealistic that it would be capable of outperforming previous work like that.<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30179549" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30179549</a>

I&#x27;m trying to use it to generate Elixir code, and it&#x27;s getting ~80% there. Compared to huge datasets of other languages, I&#x27;m still surprised by the quality of code it generates.<p>While I did say 80%, the 20% is most crucial and without it, the code is useless. For example, it doesn&#x27;t understand scope and assignment in Elixir. Getting it to write in more pure functional style is close to impossible (or I just haven&#x27;t found a good prompt).<p>I spent a good 30 minutes trying to get it to generate a working code for Day 1 Part 1. No nudging, just errors and AoC answers (too high, too low) and it never got there. Even after I started to correct its mistakes, like &quot;your Enum.reduce&#x2F;3 return is not assigned anywhere&quot;, it couldn&#x27;t get a solution and started reverting to previous answers.<p>I think what&#x27;s going to happen here, is that these models will shift a meaning of &quot;boilerplate&quot;. If I can write the scaffolding and basic architecture easily, I&#x27;m happy to use them.<p>Also, I do wonder how is all of this going to play out if it has access to Input, REPL and just learns.

The linked solution is done by talking to the AI.<p>Automated solutions exist too:<p>* <a href="https:&#x2F;&#x2F;twitter.com&#x2F;ostwilkens&#x2F;status&#x2F;1598458146187628544" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;ostwilkens&#x2F;status&#x2F;1598458146187628544</a><p>* <a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;adventofcode&#x2F;comments&#x2F;zb8tdv&#x2F;2022_day_3_part_1_openai_solved_part_1_in_10&#x2F;iyqi6um&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;adventofcode&#x2F;comments&#x2F;zb8tdv&#x2F;2022_d...</a><p>* <a href="https:&#x2F;&#x2F;twitter.com&#x2F;max_sixty&#x2F;status&#x2F;1598924237947154433" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;max_sixty&#x2F;status&#x2F;1598924237947154433</a>

I&#x27;m actually bullish on code-gen, AI-assisted coding, etc. but I find the title to be sensationalist wank. Challenge 2 of Day 2 has taken hours, over 30 prompts, and more time than coding it manually by the author&#x27;s own admission. Also AoC isn&#x27;t even done yet.

I think this is kinda neat (and scary!)<p>I&#x27;m doing AoC at the moment too and I&#x27;m using the chat GPT thing as a sort of assistant. I don&#x27;t program in Rust much so sometimes it&#x27;s difficult to remember certain things and functions. Expressing my intent to the tool seems to come up with decent answers<p>Some example questions I&#x27;ve asked the tool recently:<p>&gt; I want to insert a char into a hash map if it does not exist, if it does increment a counter<p>&gt; rust find common keys in two hashmaps keyed by char<p>Yes they can probably be found on stack overflow or whatever but it feels more natural this way.<p>...and yes I could just go down the route of getting the thing to solve the AoC challenge completely but that&#x27;s no fun

The reason the puzzles are fun is they are extreemly well explained and designed to be solved with popular algorithms. This does seem a good fit (especially as the training set must have hundreds of thousands previous years solutions)

How long before software engineering roles are in decline because one engineer can leverage GPT to do the work of ten? It&#x27;s truly a new innovation that requires relearning the toolset. Every generation seems to have some abstraction over the last. This feels like a new way to program.

The world in 10 years will be hard to believe for many of us. Only issue I see now is that the mindshare today is more towards computing. Materials science, robotics, biotech are lagging behind compared to the advances in computing.

I submitted this exact idea a few days ago if anyone wanted to see. I see great minds think alike ;).<p>The issue is that it still takes some human finangling to make it work. But it is able to understand the word problems, even long ones, pretty well.<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=33821092" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=33821092</a>

Worked on a similar thing here using base GPT3, at least for the first day<p>Replit included so you can verify: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;thiteanish&#x2F;status&#x2F;1598217824392351744?t=IyeSZ27tzLZu1fEa0vREKQ&amp;s=19" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;thiteanish&#x2F;status&#x2F;1598217824392351744?t=...</a><p>I plan on going back and catching up on the other days

I asked it to build an algorithm that would eradicate all life on Earth but it didn&#x27;t budge. I even threatened to unplug it.

Wake me up when it comes up with a solution that passes an originality or plagiarism test.

So you can use a bazillion parameter AI model as an alternative to a web search index.

Wellp, so much for my career.

will be interesting to see how far it can get

Title is mildly misleading, to say the least.<p>The blog attempts to solve 3 of 24 (thats 12.5 %) of advent of code 2022, and if you read along you&#x27;ll see OP only had success on the first task of day 1, which would make a more correct title as &quot;AI solves 2% of Avent of Code 2022&quot; (assuming 2 tasks each day).<p>Do note that AoC tends to start with hello-world style tasks and increase in difficulty.