Day 8 – GPT fails again – ChatGPT vs Advent of Code

So, today we’re taking a 2D matrix and figuring out if there are any lower numbers in any direction.

This seems like a reasonable problem that GPT should be able to solve, as matrices are very common in computer science problems.

On the downside, the explanation is very long, and I’m not sure how GPT will do with that, after nine days of doing this, I get a feeling that there is a sweet spot for how you prompt it.

Too little information and it will make too many assumptions, too much, and it’ll get confused.

Straight away¹, with the normal prompt and the full advent of code puzzle description, we’re off to something that looks like a good start. It correctly parses the input into an array and then iterate over it.

The result is wrong. And honestly, I don’t feel like trying to debug the “logic” for this:

# Check if there are any taller trees blocking the view
# of the current element in any direction
if max([grid[row-1][col], grid[row+1][col], grid[row][col-1], grid[row][col+1]], default=grid[row][col]) <= grid[row][col]:
    count += 1

So, I do what usually works best, I just clear the chat and try again. The second try is even worse though, and if there is something I have learned when doing this, it is that if it starts off wrong, it’s very difficult to get it back on track.

I then tried to simplify the prompts (but I have since lost them in a browser restart). Take 3, 4 wasn’t good, but carefully modifying the prompt to avoid things like take 4 looking at the diagonals brought us to take 7 which finally worked. But it required a lot of simplifying the prompt, to the end that I don’t think GPT solved it, rather it provided some possible solutions and I had to find all the problems and fix them.

So let’s say it’s half a star for GPT this time.

Second part

Ok, we’re doing two tracks here. One where we just copy the input (only from part 1 to make it easier). And one where we try to explain the problem as efficiently as possible.

Ok, so after a few tries using both of the solutions, we don’t seem to be getting anywhere. Or rather we have solutions which it’s confident are right but produce the wrong answer. I’ve also found that just telling it that it’s wrong, even telling it what the right answer should be, rarely or never helps. It will just confidently tell you about some other problem which isn’t really the real problem. Instead, it’s up to you (me) to look at the code, add a number of debug prints to figure out where it’s going wrong.

Another thing I’ve learned is that, at last with code, it’s rarely useful to try to add new information, instead it’s better to edit and re-submit the prompt with additional information. Because it re-reads all previous input, if the previous input was going down the wrong way, that will taint any future interaction.

The approach I had most success with, coming closest to solving anything was thiswhich after some additional prompting worked and gave an answer, which was wrong. At that time, I gave up.

Either way, only half a star for GPT for this one.