On Saturday morning I met some cracked student engineers on the sidewalk and offered them a place to crash in NYC. 24 hrs later, we won the biggest robot arm hackathon ever.

Meet Raju: an autonomous chess-playing robot (ELO >2500) inspired by GM Gukesh Dommaraju, who earned his title at age 12. Like Gukesh, Raju is poised for greatness at a young age, having just learned to move pieces yesterday.

Raju interprets board states using a fine-tuned YOLO object detection model, selects optimal moves via Stockfish, and then performs physical piece movements using an ACT policy trained to generally move a chess piece from a start square to a target square with an SO-ARM100 from LeRobot

Building Raju came with a lot of challenges. First was interpreting the board state. We were unable find an out-of-the-box VLM or LLM capable of reliably parsing board state from the camera view(s) we'd chosen for training. Initially, we built an OpenCV system that identified pre/post move image diffs, but it struggled with the lack of contrast between pieces on their respective colored squares. In the end, we took a few dozen photos of various board states, labeled them, and trained a YOLO model to recognize the pieces. This solution was accurate once integrated with an algorithm that accounted for the previous board state.

The other big challenge we encountered was getting Raju to transport pieces from one tile to another. Mechanically, the base SO-ARM100 gripper struggled to hold the rounded chess pieces consistently. Redesigned gripper mandibles with centering indents and rubber solved this issue.

On the controls side, we knew Raju would struggle with identifying the target piece amongst 32 options on the board. After brainstorming, we decided to train a model with masked input frames that indicated the target tiles, leaving the transitions to be learned along with the pickup and placement routines.

This was a big risk, as we didn't have a precedent to guarantee success or a general model to tune from. Seeing Raju target pieces was a big payoff. Both of our custom solutions worked better than any alternative on the internet for the timeframe we were given.

We're very proud of Raju, his abilities, and how much we've been able to learn during this project. We went from knowing nothing about controlling robots with ML to being fully capable of modifying, initializing, training, and using them for specific functions.

Devpost:

We're very proud of Raju, his abilities, and how much we've been able to learn during this project. We went from knowing nothing about controlling robots with ML to being fully capable of modifying, initializing, training, and using them for specific functions.

Devpost: https://devpost.com/software/raju-sassy-and-prodigal-chess-playing-robot

As an easter egg, Raju also "trash talks" opponents as the game odds shift, prompting Gemini for an appropriately confident quip, and outputting audibly. He's entering his teenage era :)

I hadn't done a hackathon in >5 years, and it was jaw dropping to see how powerful student engineers have become in the age of AI. It's humbling.

Many thanks to the organizers for facilitating such a fun weekend, to for the last minute H100s, and to this awesome team! Now back to my day job...

This process involves masking the input frames (and the video frames used for training) with an overlay that indicates where the piece should be picked and placed.