LLMs vs Traditional Chess Engines:
Can Language Models Play Chess?

Google and Kaggle organized the first major LLM chess tournament in August 2025. Eight leading models competed in a single-elimination bracket. Each AI got four attempts per move to produce a legal move; failure meant forfeiture.

Key Takeaway: Even the tournament winner (o3) plays at roughly Class D level (~1200 ELO). Many matches were decided by illegal move forfeiture rather than actual chess skill. GM Hikaru Nakamura noted o3 made "very few mistakes" but GM David Howell observed that Grok "crumbled under pressure."

The most comprehensive academic benchmark, testing 50+ models against both a random opponent and Komodo Dragon 1 chess engine.

Key Results vs Random Opponent (30 games each):

Move Quality (o4-mini vs GPT-4.1-mini):

Critical Finding: 71.9% of non-reasoning model losses were due to instruction-following failures (unable to format valid moves), not chess knowledge. Reasoning models reduced this to 24.4%.

The percentage of illegal moves is perhaps the most telling metric for understanding the LLM-chess gap:

Reasoning models achieve near-perfect legality because they use their "thinking budget" to write out the board state, enumerate candidate moves, verify legality, and self-correct before committing. This is computationally expensive but effective.

For historical context, AlphaZero's matches against Stockfish 8 remain the most famous neural network vs. engine competition:

• Initial 100-game match (2017): AlphaZero won 28, lost 0, drew 72
• Extended 1,000-game match (2018): AlphaZero won 155, lost 6, drew 839
• AlphaZero searched ~60,000 positions/sec vs Stockfish's ~60 million -- 1000x fewer
• AlphaZero learned chess from scratch in 4 hours of self-play
• Its games featured stunning piece sacrifices for long-term strategic advantage

Important distinction: AlphaZero is a purpose-built chess engine with MCTS, not an LLM. It has perfect knowledge of chess rules and an internal board representation. It just uses a neural network for evaluation instead of hand-coded heuristics.

Perhaps the most important result bridging LLMs and chess engines. DeepMind trained a 270M-parameter transformer on ChessBench -- 10 million Lichess games annotated with Stockfish 16 evaluations (15 billion data points).

• Result: 2895 ELO on Lichess blitz (grandmaster level) with zero search at test time
• Puzzle solving: Lichess Puzzle ELO of 2867
• Trained to predict action-values (win percentages) for board positions
• Demonstrated that Stockfish's search-based algorithm can be approximately distilled into a transformer
• Key limitation: "Perfect distillation is still beyond reach" -- gap to actual Stockfish remains

This proves transformers can learn strong chess -- but only when purpose-built and trained on chess-specific annotated data, not as a side effect of general language modeling.

A follow-up that added search back into the transformer equation, achieving even stronger results:

• External search: Transformer guides MCTS rollouts without external engine calls
• Internal search: Model generates linearized trees of future positions in-context
• Both approaches achieved Grandmaster-level performance on a human-like search budget
• Pre-training method "minimizes hallucinations" -- highly accurate state prediction and legal moves
• Tested across Chess, Fischer Random Chess, Connect Four, and Hex

This represents the most promising hybrid approach: a chess-trained transformer augmented with structured search.

A landmark interpretability study showing that even a small (50M parameter) model trained on PGN chess notation develops sophisticated internal representations:

• Linear probes achieved 99.2% accuracy classifying pieces on each square
• The model develops a "my/their" perspective rather than absolute black/white encoding
• 89% accuracy estimating player skill level (above/below certain ELO thresholds) from move patterns
• Model learns rules including check, checkmate, castling, en passant, and pinned pieces
• Intervening on internal activations can change the model's play style (e.g., simulating a stronger player)
• 50M parameter model reached ~1300 ELO from 5M training games

Key Insight: LLMs don't just memorize move sequences -- they develop genuine internal representations of the board. But these representations are fragile and break under perturbation.

Published in January 2025, this paper demonstrated that training on complete game sequences (rather than isolated positions) dramatically improves LLM chess ability:

• Achieved 1788 ELO vs Stockfish when permitted 10x sampling
• Long-round data supervision yields a 350 ELO improvement over short-round data
• Trained on 20 billion tokens of complete chess games
• Uses FEN (Forsyth-Edwards Notation) for position representation
• Open-sourced code, model, and dataset

One of the earliest systematic attempts to make LLMs play chess, presented as a Datasets and Benchmarks paper at NeurIPS 2023:

• Collected large-scale online chess game datasets with annotated Stockfish evaluations
• Leveraged PGN metadata: ELO ratings for player strength, annotated evaluations for value learning
• Introduced both ChessGPT (policy model) and ChessCLIP (state understanding model)
• Open-sourced code, models, and datasets on GitHub

Multiple 2025 papers investigated whether RL post-training can develop genuine strategic reasoning in LLMs through chess:

• Finding: All models plateau far below expert levels despite RL fine-tuning
• The limitation stems from deficits in pretrained models' internal chess understanding
• RL "mainly amplifies existing capabilities" -- it cannot create chess knowledge from scratch
• Models cannot reliably track game states or recognize elementary tactics
• ChessArena benchmark: No model could beat Maia-1100 (human amateur level); some failed to beat a random player
• Pre-trained domain knowledge is essential -- RL alone is insufficient

One of the most curious findings in LLM chess research is that OpenAI's gpt-3.5-turbo-instruct (released September 2023) plays significantly better chess than newer, more capable models:

• ~1750-1800 ELO vs calibrated Stockfish -- strong club player level
• Only 0.3% illegal move rate across 8,205 moves
• GPT-4 (chat) scored only ~1370 ELO with 0.66% illegal moves
• Later chat models (GPT-3.5-turbo chat) had 93% of games containing illegal moves

Why? The instruct model is a text completion model, not a chat model. RLHF training for chat may actually harm chess ability. As a pure next-token predictor, the instruct model more faithfully simulates the chess games in its training data. Chat-tuned models are optimized for helpful conversation, which conflicts with the narrow task of generating valid chess notation.

The most recent development: Google's Gemini 3 Pro and Gemini 3 Flash claimed the top ELO positions on the Kaggle Game Arena chess leaderboard as of February 2026.

• Marked performance increase over the Gemini 2.5 generation
• Gemini 3 topped all three game leaderboards (Chess, Poker, Werewolf)
• Uses pattern recognition and strategic reasoning grounded in chess concepts (piece mobility, pawn structure, king safety)
• Represents rapid model-over-model improvement in chess ability
• Specific ELO scores not publicly disclosed, but surpassed o3's previous top position

Almost certainly not as general-purpose models. The architectural limitations are fundamental:

• No guaranteed legal moves: Every move requires "hoping" the next token is valid notation for a legal move. Engines generate legal moves by definition.
• No systematic search: Without exhaustive lookahead, LLMs cannot find tactical combinations that require seeing 10-20 moves ahead. This is where engines dominate.
• No stable board representation: LLMs must reconstruct board state from text each time. Any error compounds. Engines maintain a perfect internal state.
• Tokenization mismatch: Chess notation was not designed for transformer tokenization. Position encoding artifacts degrade performance.

However, specialized chess transformers with search (like DeepMind's MCTS-MAV) may eventually match engines. The key is combining the transformer's learned evaluation with a proper search algorithm -- essentially reinventing AlphaZero with a larger, pre-trained model.

Chess is increasingly used as a benchmark for genuine reasoning (vs. pattern matching) because it requires:

• State tracking: Maintaining an evolving board state across 40-80 moves
• Planning: Evaluating consequences of moves several turns ahead
• Constraint satisfaction: All moves must be legal within a complex ruleset
• Adversarial reasoning: Predicting and countering an opponent's strategy

The fact that LLMs perform better at math and coding (where chain-of-thought works well) than at chess (where spatial reasoning and lookahead are essential) suggests that current "reasoning" capabilities are more about sequential logic than spatial-strategic intelligence.

The LLM Chess benchmark found a Pearson r = 0.686 correlation between chess and coding performance -- moderately positive but with significant gaps, confirming chess tests a distinct reasoning capability.