Reinforcement learning & parallel systems
Connect Four
Your move.
A Connect Four agent that learns from solved positions, then borrows a little foresight from search.
Take a seatBrowser opponent: alpha-beta minimax. The trained DQN’s results are below; this is the search family it was evaluated against.
01 / The question
How far can a little learning and search go?
02 / What I built
I pretrained a five-layer convolutional backbone on 67,557 solved board positions, froze it, and added a trainable branch for reinforcement learning. The resulting DQN has 26.2 million parameters.
A staged curriculum runs across 1,024 parallel environments, with batched self-play and a C alpha-beta engine. At inference, a shallow search filters the moves before the learned policy chooses.
03 / What happened
The best DQN hybrid checkpoint won 57 of 100 games against depth-6 minimax, with 8 draws. The bare DQN won 40 under the same benchmark: a 17-percentage-point lift from the search filter. A separate depth sweep recorded 62 wins at depth 7.
04 / Where it stops
Each matchup alternated first player. The headline checkpoint came from iterative development; the documented 10-hour, roughly 3.1-million-episode curriculum is a separate run. The playable opponent here is browser minimax, not the trained network. The related comparison project also implements PPO.
05 / Keep exploring
One more? / 04