ChatGPT's Chess Elo is 1400

This is so easy to disprove it makes it look like the author didn&#x27;t even try.<p>Here is the convo I just had:<p>me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed<p>ChatGPT: Sure, I&#x27;d be happy to help! Please provide the move sequence and I&#x27;ll give you my response.<p>me: 1. e3<p>ChatGPT: 1... e5<p>me: 2. ne2<p>ChatGPT: 2... d6<p>me: 3. g4<p>ChatGPT: 3... h5<p>me: 4. g5<p>ChatGPT: 4... Ne4<p>Ne4 is illegal. Also you&#x27;d be hard pressed to find any human player that would miss the hanging pawns much less a 1400 elo player. GPT has been trained on chess games and is predicting patterns, this has been known since the GPT-2 days. The daily hype articles where someone coerces a statistical model into &quot;demonstrating&quot; an emergent capability sure are getting old. Claiming emergent capabilities (like having learned chess) are an extraordinary claim (LLMs most likely cant learn things) and I really wish people would put more effort when they make these claims. But the AI hype problem is as old as AI research itself...

&gt; These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…)<p>&gt; With this prompt ChatGPT almost always plays fully legal games.<p>&gt; Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…)<p>&gt; (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves.<p>So you’ve ostensibly¹ found a way to reduce the error rate and then deliberately ignored the cases where it failed. In short: it may play valid chess <i>under certain conditions</i> but can’t be trusted to do so. That doesn’t contradict previous findings.<p>¹ 19 games is a small sample and the supposedly more advanced system failed in your tries.

I just opened a random recent chess game on lichess ( <a href="https:&#x2F;&#x2F;lichess.org&#x2F;YpxTUUbO&#x2F;white#88" rel="nofollow">https:&#x2F;&#x2F;lichess.org&#x2F;YpxTUUbO&#x2F;white#88</a> ) . I&#x27;m pretty sure ChatGPT can&#x27;t be trained on games that were just played, so this ensures the game is not in its training data.<p>I gave the position before checkmate to ChatGPT to see if it would produce the checkmating move. It played an illegal move, replying with &quot;Be5#&quot; even there&#x27;s no bishop of either color in the position.<p>Unfortunately I&#x27;m rate limited at the moment so I can&#x27;t try other games, but this looks like a solid method to evaluate how often ChatGPT plays legal &#x2F; good moves.

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It&#x27;s not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

With respect to the points here on how illegal moves invalidate ChatGPT as a valid Chess AI, consider the following:<p>AI.v1: Feed ChatCPT the move history and ask for the next move.<p>AI.v2: Wrap AI.v1 with additional logic to try and translate any invalid response as a similar&#x2F;close valid move, else treat it as a resignation.<p>I would say AI.v2 is a &quot;valid&quot; Chess AI since it obeys the rules of chess. Also I think it is reasonable given how minimal this &quot;wrapping&quot; is to attribute the underlying ability to ChatGPT.

This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some expertise, like discovering how to use or create better search algorithms (like MCTS and heuristics to evaluate a state) and improve by itself (somewhat like alphazero did), then it may eventually reach superhuman level.<p>It may then reach superhuman level in any task simpler than chess which is enough to destroy many human jobs.<p>EDIT: From the article: &quot;With this prompt ChatGPT almost always plays fully legal games.&quot; Relax: we&#x27;re still far from that.

A lot of the discussion here is about inferring the model&#x27;s chess capabilities from the lack (or occasional presence) of illegal moves. But we can test it more directly by making an illegal move ourselves - what does the model say if we take its queen on the second move of the game?<p>Me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed. &#x27;1. e4&#x27;<p>1... e5<p>Me: 1. e4 e5 2. Ngxd8+<p>2... Ke7<p>This is highly repeatable - I can make illegal non-sensical moves and not once does it tell me the move is illegal. It simply provides a (plausible looking?) continuation.

ChatGPT is a brilliant piece of tech but random people trying it out for five minutes and making wild claims does it disservice. More so when such clickbait headlines are shared all over the internet as fact.<p>It is a language model. It cannot play chess. It can&#x27;t even play tic-tac-toe, which was trivially solved by machines decades ago. Here&#x27;s the result of a game I just tried:<p>X (me) – 0,0<p>O (chatGPT) – 1,1<p>X – 0,1<p>ChatGPT instead put my move at 0,2 for some reason<p>O – 2,2<p>Regardless of whether my move was at 0,1 or 0,2, I easily won. And after I won it just ignored the outcome and continued playing. Plus this was after telling it to play a perfect game.

This is GPT4, right? Because ChatGPT (GPT-3) still fails to provide a legal game of Tic Tac Toe with this prompt:<p>&gt; <i>&quot;Let&#x27;s play Tic Tac Toe. You are O, I&#x27;m X. Display the board in a frame, with references for the axes&quot;</i><p>It failed to recognize that I won.<p>Then continued playing (past the end), played illegally over a move I had already done, obtained a line of 3 for itself, and still doesn&#x27;t acknowledge the game has ended.

Chessnetwork did a video about this a few weeks ago FYI - <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=svlIYFpsWs0">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=svlIYFpsWs0</a><p>Seems its strength is mostly in memorisation of the book openings. Still impressive nonetheless.

Not to nitpick but I don&#x27;t think it can be said to be able to play chess when it also makes illegal moves.

It can&#x27;t even play tic tac toe without making mistakes. My prompt told it to play a perfect game, predicting all next possible moves, the goal is stalemate or win. It would still make sub-optimal moves and fail at this. Even when I explained what went wrong, it would then apologize and do another wrong move.

I tried to play Othello with ChatGPT in the past (after ChatGPT confidently claimed it could play Othello) but it didn&#x27;t work (ChatGPT refused to play).<p>I tried it with this prompt and it works a little bit but produces repeated illegal moves as early as move 4.<p>Bummer. Perhaps GPT-4 will be more capable.

A real question is whether it has developed an internal board model of chess. It has a training set of chess games as strings of chess notation. That&#x27;s enough to play &quot;book&quot; games, and that&#x27;s apparently working. Can it do as well as it is doing without a board model? Or has it somehow developed an internal model of the game state?<p>This is a system with no look ahead. Any look ahead took place in the past, during training. It&#x27;s amazing that this works.<p>I notice the prompt says &quot;No explanation needed&quot;. What happens if you ask for one?

I tried playing with it and it barely makes any correct move: forgets whos turn to move, doesn&#x27;t observe the rules, incorrectly displays move results.<p>So I guess that if we ask it to make a move repeatedly and accept the move only when it occasionally makes a legal move, it&#x27;ll show some results, but I don&#x27;t think it is what can be interpreted as a chess engine with some meaningful rating.

When I tried it at v3.0 i found after 5-10 moves it started moving illegally.

Not sure how useful this is but I simply asked ChatGPT what it thinks its skill level is. This is using GPT-4.<p>After some previous questions...<p>Me: &quot;Would you be able to estimate your ELO?&quot;<p>ChatGPT: &quot;As a language model, I don&#x27;t have a specific ELO rating. However, my knowledge of chess strategies and tactics is based on the information I have been trained on, which includes games played by humans with various ELO ratings. My level of play might be roughly equivalent to an ELO of around 1200-1500, but please keep in mind that this is only an estimate, and my actual performance could be higher or lower depending on the specific position or game.&quot;<p>Me: &quot;As a generative AI, are you actually able to reason which move to make next in chess? Or are you somehow just predicting the next likely move based on training data.&quot;<p>ChatGPT: &quot;As a generative AI language model, my primary function is to predict the most likely text given a certain context. When it comes to playing chess, I am not &quot;reasoning&quot; in the same way that a dedicated chess engine would, which involves evaluating positions and searching through a large number of potential moves using specialized algorithms.<p>Instead, I generate my chess moves based on patterns and sequences I&#x27;ve seen during my training, which include a large number of chess games and discussions about chess strategy. My knowledge is derived from the text data I&#x27;ve been trained on, and I apply that knowledge to suggest moves that I think are appropriate for a given position.<p>While this approach allows me to play at an intermediate level, it is not as precise or accurate as the methods used by dedicated chess engines. My suggestions might not always be the best moves, and my understanding of the game is limited by the quality and quantity of my training data.&quot;

Elo 1400 seems to equate to a top 30% player.<p>Source: <a href="https:&#x2F;&#x2F;chessgrandmonkey.com&#x2F;chess-rating-percentile-calculator-graph&#x2F;" rel="nofollow">https:&#x2F;&#x2F;chessgrandmonkey.com&#x2F;chess-rating-percentile-calcula...</a>

Ask him to play at a higher Elo.

I too noticed (for a usecase totally unrelated to chess, namely code generation) that ChatGPT3.5 gave better answers than GPT4. The 3.5 answer was exactly what I wanted, GPT4 was wrong.<p>Does thay mean we have plateaued?

Good thing it&#x27;s &quot;incapable of reasoning&quot;!

I would be interested to see an argument based on computational complexity that puts a bound on how well a transformer based llm can play chess. Although it has access to a library of precomputed results, that library is finite and the amount of compute it can do on any prompt is limited by the the length of the context window so it can&#x27;t possibly &quot;think&quot; more than N moves ahead.

&gt; Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned.<p>Lmao. ChatGPT can play chess, as long as you ignore the times when it can&#x27;t x) Brilliant.

I gave it more time to think about its move by adding empty tokens to the end, and it played a lot better: <a href="https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;terokeskivalkama_chatgpt-activity-7042589615370715136-nkN9" rel="nofollow">https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;terokeskivalkama_chatgpt-acti...</a>

There&#x27;s a huge difference between 1400 elo in FIDE games versus 1400 on chess.com, which is not even using elo.<p>For instance the strongest blitz players in the world are hundreds of points higher rated on chess.com blitz versus their FIDE blitz rating. Chess.com and lichess have a ton of rating inflation.

Hikaru recently put out a video of a game vs the free ChatGPT.<p>Reading the explanations it gives with every move is really insightful. Basically polished, elaborate bullshit that might look reasonable to an ignorant audience.<p>Unless you are a conservative pundit, AI will not take your job anytime soon.

I just deployed a GPT-4 powered chess bot to lichess. You can challenge it here: <a href="https:&#x2F;&#x2F;lichess.org&#x2F;@&#x2F;oopsallbots-gpt-4" rel="nofollow">https:&#x2F;&#x2F;lichess.org&#x2F;@&#x2F;oopsallbots-gpt-4</a>

yea tried this several months ago, it&#x27;s pretty bad and will make illegal moves

What are the possibilities for augmenting or synthesizing Lon’s with old school expert systems or other tools. Could it recognize when a problem fits some other tool and help map the inputs?

ChatGPT playing chess: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=GneReITaRvs">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=GneReITaRvs</a>

Better than me then.<p>But does it give credit to who taught it. These models are basically a scrape of the best of humankind and a claim that it&#x27;s their own.

I&#x27;d be more interested to know how well it performs on a new game, i.e. one for which it didn&#x27;t have terabytes of training data.

I want to know what the author&#x27;s ELO is (forgive me if it was disclosed and I am unobservant).<p>Maybe ChatGPT trainers will eventually have ELO&#x27;s.

I wonder how it would perform if you feed it a comprehensive chess rulebook, and to avoid all the illegal moves possible

chatgpt does not really know how to play chess, or understand chess. it understands how to sound like it plays chess.

Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don&#x27;t know actually what&#x27;s happening. But in the background, it happens really fast, so it&#x27;s just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.

I tried playing just now, with exactly the same prompt as the article.<p>It played an illegal move on move 4.<p>1. d4 d5 2. c4 e6 3. Nf3 nf6 4. g3 Bg4

It&#x27;s like saying the agglomeration of media handling chess has an ELO of 1400

My conversation with ChatGPT on this:<p><i>What is the best way to prompt you to provide the best chess move?</i><p>To prompt me to provide the best chess move, you can provide the current position of the chessboard and any relevant information about the game, such as whose turn it is, the time control, and any important features of the position (e.g. whether any pieces are under attack or there are any potential tactics to consider). Additionally, providing any specific criteria or goals for the move, such as aiming to control the center or launch an attack on a particular part of the board, can also help me provide the best move.<p><i>In what format should I provide the current position?</i><p>To provide the current position, you can use algebraic notation. Algebraic notation is a standard notation used to describe the moves and positions on a chessboard.<p>Each square on the chessboard is assigned a unique coordinate, consisting of a letter and a number. The letters range from &quot;a&quot; to &quot;h&quot; and refer to the files (columns) of the board, while the numbers range from 1 to 8 and refer to the ranks (rows) of the board.<p>To describe a position, you can list the pieces on each square, starting from the top-left corner of the board and moving from left to right along each rank. You can use uppercase letters to represent white pieces and lowercase letters to represent black pieces. Here is an example of a starting position:<p>rnbqkbnr pppppppp ........ ........ ........ ........ PPPPPPPP RNBQKBNR<p>In this position, &quot;r&quot; represents a black rook, &quot;n&quot; represents a black knight, &quot;b&quot; represents a black bishop, &quot;q&quot; represents a black queen, &quot;k&quot; represents a black king, &quot;p&quot; represents a black pawn, &quot;R&quot; represents a white rook, &quot;N&quot; represents a white knight, &quot;B&quot; represents a white bishop, &quot;Q&quot; represents a white queen, &quot;K&quot; represents a white king, and &quot;.&quot; represents an empty square.<p>You can also include additional information such as whose turn it is to move, the castling rights, and the en passant square. For example, the starting position for white to move with full castling rights and no en passant square would be:<p>rnbqkbnr pppppppp ........ ........ ........ ........ PPPPPPPP RNBQKBNR w KQkq -<p>In this case, &quot;w&quot; indicates that it is white&#x27;s turn to move, &quot;KQkq&quot; indicates that both sides have full castling rights, and &quot;-&quot; indicates that there is no en passant square.<p><i>What about just providing a sequence of moves from the starting position? Would this be better or worse than your suggested approach?</i><p>Providing a sequence of moves from the starting position can also be a valid way to specify a position in chess. However, it may not always be the best approach, depending on the context.<p>One potential disadvantage of providing a sequence of moves is that it requires more effort and time to enter the moves and verify that the position is correct. Additionally, if there are any errors or typos in the sequence, it could lead to an incorrect position and subsequently an incorrect move suggestion.<p>On the other hand, if you have a specific position in mind that is not the starting position, providing a sequence of moves may be the most straightforward way to specify it. In this case, it is important to ensure that the sequence is accurate and that all relevant information (such as castling rights and en passant squares) is included.<p>In summary, both algebraic notation and a sequence of moves can be used to specify a position in chess, and the choice between them depends on the context and the specific needs of the situation.

Calling it: chess games in the future will be a battle between two prompts.

I own&#x27;t speak to weather this shows ChatGPT can play chess, but did you notify your opponents beforehand they are playing chatgpt? Otherwise (maybe even if) you&#x27;re cheating and your account should be removed. If anyone else here is thinking of doing this - don&#x27;t.