Carra Wu
@carra
On Saturday morning I met some cracked student engineers on the sidewalk and offered them a place to crash in NYC. 24 hrs later, we won the biggest robot arm hackathon ever. Meet Raju: an autonomous chess-playing robot (ELO >2500) inspired by GM Gukesh Dommaraju, who earned his title at age 12. Like Gukesh, Raju is poised for greatness at a young age, having just learned to move pieces yesterday.
1 reply
0 recast
3 reactions
Carra Wu
@carra
Raju interprets board states using a fine-tuned YOLO object detection model, selects optimal moves via Stockfish, and then performs physical piece movements using an ACT policy trained to generally move a chess piece from a start square to a target square with an SO-ARM100 from LeRobot
1 reply
0 recast
0 reaction
Carra Wu
@carra
This process involves masking the input frames (and the video frames used for training) with an overlay that indicates where the piece should be picked and placed.
1 reply
0 recast
0 reaction
Carra Wu
@carra
Building Raju came with a lot of challenges. First was interpreting the board state. We were unable find an out-of-the-box VLM or LLM capable of reliably parsing board state from the camera view(s) we'd chosen for training. Initially, we built an OpenCV system that identified pre/post move image diffs, but it struggled with the lack of contrast between pieces on their respective colored squares. In the end, we took a few dozen photos of various board states, labeled them, and trained a YOLO model to recognize the pieces. This solution was accurate once integrated with an algorithm that accounted for the previous board state.
1 reply
0 recast
0 reaction
Carra Wu
@carra
The other big challenge we encountered was getting Raju to transport pieces from one tile to another. Mechanically, the base SO-ARM100 gripper struggled to hold the rounded chess pieces consistently. Redesigned gripper mandibles with centering indents and rubber solved this issue.
1 reply
0 recast
0 reaction
Carra Wu
@carra
On the controls side, we knew Raju would struggle with identifying the target piece amongst 32 options on the board. After brainstorming, we decided to train a model with masked input frames that indicated the target tiles, leaving the transitions to be learned along with the pickup and placement routines.
1 reply
0 recast
0 reaction
Carra Wu
@carra
This was a big risk, as we didn't have a precedent to guarantee success or a general model to tune from. Seeing Raju target pieces was a big payoff. Both of our custom solutions worked better than any alternative on the internet for the timeframe we were given.
1 reply
0 recast
0 reaction
Carra Wu
@carra
We're very proud of Raju, his abilities, and how much we've been able to learn during this project. We went from knowing nothing about controlling robots with ML to being fully capable of modifying, initializing, training, and using them for specific functions. Devpost: https://devpost.com/software/raju-sassy-and-prodigal-chess-playing-robot
1 reply
0 recast
0 reaction
