Implementing the Royal Game of Ur using Mistral's Codestral

In this post, I test how Mistral's Codestral, the model specifically trained to generate code, compares to the Mistral's Le Chat, the general AI chat. For my experiment, I use the Royal Game of Ur, an ancient Babylonian game that I already implemented using Le Chat. TLDR; At least in my experiment, Codestral's performance is, frankly, disappointing. While it produces code that follows a certain architecture, the code quality is low. In fact, it even generates a statement of the form a = foo() where a has a different type from what foo() returns. Read on for more details. Same Prompt, Different Architecture After the same prompt as with Le Chat, Codestral's response is: To implement the Royal Game of Ur in Python, we need to define several components: Game Board: Represent the board and the special squares. Game Pieces: Represent the pieces for each player. Dice: Implement the dice rolling mechanism. Game Logic: Implement the rules of the game, including piece movement, capturing, and winning conditions. Let's start by defining the game board and the special squares. Codestral then generates the code for all 4 files plus code for main.py to start the game. Compared to Le Chat that outputs a single Python script, this is already some progress in terms of software architecture. The class Game which implements the actual game logic imports classes Board, Piece, and Dice implementing the game equipment. This architecture is sensible as it leads to a cohesive, well-structured code. Code Quality ... Functions that Codestral generated in my experiment lack docstrings. But apart from that the code runs. But wait, the game seems to always terminate after a single move: $ python main.py player2 starts the game. player2 rolled a 1 player1 wins! $ python main.py player1 starts the game. player1 rolled a 2 player1 wins! $ python main.py player2 starts the game. player2 rolled a 3 player1 wins! $ python main.py player1 starts the game. player1 rolled a 3 player1 wins! Attempt to Fix the Code using Codestral I type in the following prompt in the Chat mode: The game implemented using the code in terminates after a single move. This is not correct. Help me debug and explain why the game terminates after a single move Codestral responds: The game terminates after a single move because the play_game method in game.py does not contain a loop to continue playing turns until a winner is determined. The loop in play_game should continue to call play_turn and switch_player until a winner is found. But this is actually not true: def play_game(self): self.determine_starting_player() print(f"{self.current_player} starts the game.") while True: self.play_turn() winner = self.check_winner() if winner: print(f"{winner} wins!") break self.switch_player() Oddly enough, as a fix to the issue, Codestral proposes exactly the code that is already in game.py... So, as a next step, I ask Codestral to explain how the function check_winner() works. It does a good job at explaining the function: The method first checks if all pieces of player1 are off the board. It does this by iterating over all pieces in self.player1_pieces and checking if their position attribute is None. If all pieces have position set to None, it means all pieces of player1 have been moved off the board, and player1 is declared the winner. Here's the code of that function and, as you can see, Codestral's explanation how the code works is correct: def check_winner(self): if all(piece.position is None for piece in self.player1_pieces): return 'player1' if all(piece.position is None for piece in self.player2_pieces): return 'player2' return None Looking at the code in check_winner(), it is immediately clear why the game terminates after just one move and why player 1 always wins: if no pieces where moved so far, the condition all(piece.position is None for piece in self.player1_pieces) is always true and the function returns player1 as the winner. Trying to Fix the Bug using Codestral So, I change into the Edit mode and tell Codestral: The code in game.py terminates after just 1 move because the function check_winner() always returns "player1". This is because the condition "all(piece.position is None for piece in self.player1_pieces)" is true when no pieces were moved. Fix this so that the game end only after all pieces were moved across the board and taken off the board. Unfortunately, Codestral does not really know how to perform the fix based on my prompt. Instead of fixing the code, it suggests to add a third condition to check_winner() that implements a draw: if all(piece.position is None for piece in self.player1_pieces) and all(piece.position is None for piece in self.player2_pieces):

May 6, 2025 - 20:34

Implementing the Royal Game of Ur using Mistral's Codestral

In this post, I test how Mistral's Codestral, the model specifically trained to generate code, compares to the Mistral's Le Chat, the general AI chat. For my experiment, I use the Royal Game of Ur, an ancient Babylonian game that I already implemented using Le Chat.

TLDR;

At least in my experiment, Codestral's performance is, frankly, disappointing. While it produces code that follows a certain architecture, the code quality is low. In fact, it even generates a statement of the form a = foo() where a has a different type from what foo() returns. Read on for more details.

Same Prompt, Different Architecture

After the same prompt as with Le Chat, Codestral's response is:

To implement the Royal Game of Ur in Python, we need to
define several components:

Game Board: Represent the board and the special squares.
Game Pieces: Represent the pieces for each player.
Dice: Implement the dice rolling mechanism.
Game Logic: Implement the rules of the game, including piece movement, capturing, and winning conditions.
Let's start by defining the game board and the special squares.

Codestral then generates the code for all 4 files plus code for main.py to start the game.

Compared to Le Chat that outputs a single Python script, this is already some progress in terms of software architecture.

The class Game which implements the actual game logic imports classes Board, Piece, and Dice implementing the game equipment. This architecture is sensible as it leads to a cohesive, well-structured code.

Code Quality ...

Functions that Codestral generated in my experiment lack docstrings. But apart from that the code runs. But wait, the game seems to always terminate after a single move:

$ python main.py
player2 starts the game.
player2 rolled a 1
player1 wins!
$ python main.py
player1 starts the game.
player1 rolled a 2
player1 wins!
$ python main.py
player2 starts the game.
player2 rolled a 3
player1 wins!
$ python main.py
player1 starts the game.
player1 rolled a 3
player1 wins!

Attempt to Fix the Code using Codestral

I type in the following prompt in the Chat mode:

The game implemented using the code in  terminates after a
single move. This is not correct. Help me debug and explain why
the game terminates after a single move

Codestral responds:

The game terminates after a single move because the
play_game method in game.py does not contain a loop to continue playing turns until a winner is determined. The loop in play_game
should continue to call play_turn and switch_player until a winner is found.

But this is actually not true:

    def play_game(self):
        self.determine_starting_player()
        print(f"{self.current_player} starts the game.")

        while True:
            self.play_turn()
            winner = self.check_winner()
            if winner:
                print(f"{winner} wins!")
                break
            self.switch_player()

Oddly enough, as a fix to the issue, Codestral proposes exactly the code that is already in game.py...

So, as a next step, I ask Codestral to explain how the function check_winner() works. It does a good job at explaining the function:

The method first checks if all pieces of player1 are off the board.
It does this by iterating over all pieces in self.player1_pieces and checking if their position attribute is None.
If all pieces have position set to None, it means all pieces of player1 have been moved off the board, and player1 is declared the winner.

Here's the code of that function and, as you can see, Codestral's explanation how the code works is correct:

    def check_winner(self):
        if all(piece.position is None for piece in self.player1_pieces):
            return 'player1'
        if all(piece.position is None for piece in self.player2_pieces):
            return 'player2'
        return None

Looking at the code in check_winner(), it is immediately clear why the game terminates after just one move and why player 1 always wins: if no pieces where moved so far, the condition all(piece.position is None for piece in self.player1_pieces) is always true and the function returns player1 as the winner.

Trying to Fix the Bug using Codestral

So, I change into the Edit mode and tell Codestral:

The code in game.py terminates after just 1 move because the
function check_winner() always returns "player1". This is because
the condition "all(piece.position is None for piece in
self.player1_pieces)" is true when no pieces were moved. Fix this
so that the game end only after all pieces were moved across
the board and taken off the board.

Unfortunately, Codestral does not really know how to perform the fix based on my prompt. Instead of fixing the code, it suggests to add a third condition to check_winner() that implements a draw:

if all(piece.position is None for piece in self.player1_pieces) and all(piece.position is None for piece in self.player2_pieces):
    return "draw"

Well, then... next try. I ask Codestral to do the following:

The function check_winner always returns "player1" because if
the pieces are not played yet, i.e., if no move has been made at
the start of the game, all pieces are off board. YOUR TASK:
suggest how to adjust  so that the pieces are in a start state
before they enter the game board and in the finish state after
they complete the player's path and the player can take them
off the board.

The response from Codestral is helpful:

"player1" because all pieces are initially off the board,
you need to adjust the initial state of the pieces and the logic
for moving pieces off the board. Specifically, you should:

1. Set the initial positions of the pieces to their starting positions on the board.
2. Adjust the logic for moving pieces off the board to correctly handle the finish state.

Unfortunately, what Codestral proposes is to assign every piece a starting position from a1 to a7 for player 1 and from c1 to c7 for player 2. This is, of course, completely wrong given the rules of the game.

Somewhat frustrated, I give up and start fixing the bug by hand. The fix is actually quite simple:

When a piece is taken off the board, it must enter a final state
The check_winner() function must check whether all pieces are in the final state and, if that is the case, declare the winner

This is one small change in the move_piece function: when the piece is taken off the board, instead of piece.position = None we set it to piece.position = 'final'.

Alarm!

While changing the code in game.py, I discover that the code for the class Game appears twice in game.py.

After deleting the second instance of the Game class, I get the following error when I try to run the code:

File "/Users/paulduplys/Repositories/royal-game-of
ur/codestral/game.py", line 35, in move_piece
  opponent_piece.position = 'a5' if opponent_piece.player 
  == 'player1' else 'c5'
AttributeError: 'str' object has no attribute 'player'

Again, in the Edit mode, Codestral misses where the error occurs and does not help to fix it. So, I ask Le Chat the following question:

You are an experience Python developer with over 15 years of experience. Your task is to help me debug my Python code.

My code contains the following statement: "opponent_piece.position
= 'a5' if opponent_piece.player == 'player1' else 'c5'". When
running the code, I get an error: " opponent_piece.position = 'a5'
if opponent_piece.player == 'player1' else 'c5'   AttributeError:
'str' object has no attribute 'player'". How can I fix the error?

Codestral's answer:

The error message AttributeError: 'str' object has no attribute
'player' indicates that opponent_piece is a string, and you're
trying to access an attribute player on it, which doesn't exist
for string objects.

-- snip --

And, indeed, print(type(opponent_piece)) prints ...

Digging deeper, my attention is first drawn upon the definition of the game board:

class Board:
    def __init__(self):
        self.special_squares = {'a2', 'a8', 'b5', 'c2', 'c8'}
        self.board = {
            'a': ['a1', 'a2', None, 'a5', 'a6', 'a7', 'a8'],
            'b': ['b1', 'b2', 'b3', 'b4', 'b5', 'b6', 'b7', 'b8'],
            'c': ['c1', 'c2', None, 'c5', 'c6', 'c7', 'c8']
        }

It is incorrect because the middle "lane" has more squares than the outer lanes and it does not comply to the specification:

+----+----+         +----+----+----+----+
| a1 | a2 |         | a5 | a6 | a7 | a8 |
+----+----+----+----+----+----+----+----+
| b1 | b2 | b3 | b4 | b5 | b6 | b7 | b8 | 
+----+----+----+----+----+----+----+----+
| c1 | c2 |         | c5 | c6 | c7 | c8 |
+----+----+         +----+----+----+----+

Rather, it has to be:

-- snip --

'a': ['a1', 'a2', None, None, 'a5', 'a6', 'a7', 'a8'],
'b': ['b1', 'b2', 'b3', 'b4', 'b5', 'b6', 'b7', 'b8'],
'c': ['c1', 'c2', None, None, 'c5', 'c6', 'c7', 'c8']

-- snip --

But the code still doesn't work... After some reading and debugging, I discover the following code in function move_piece():

-- snip --
if self.board.get_square(new_position) is not None:
                opponent_piece = self.board.get_square(new_position)
                opponent_piece.position = 'a5' if opponent_piece.player == 'player1' else 'c5'
-- snip --

While there is a dedicated class Piece for representing the game pieces, the function board.get_square() returns board[row][col] which is simply a String. So, the code that Mistral generates is not even type-safe.

Conclusion

Overall, Codestral is better when it comes down to the architecture. But at least in my experiment, the code quality was really low. And I'm not even speaking about logical errors. If the model generates code such as a = foo(), and the return value of foo() is different from a's type, then the code is basically unworkable. If it weren't for an interpreted language like Python, this code wouldn't even compile. So, unfortunately, Codestral does not really offer advantages compared to Le Chat.