Jenga Bot pt2: Pez for robots
Are you a scientist?
"Are you a scientist? Like, is that what you do for your job? What do I tell my friends you do?"
"Well yes, I suppose I am something of a scientist, a COMPUTER scientist. It's just that, well, no one's ever actually called me that before."
I'm not crying, you're crying.
Even though my eldest daughter is gleefully uninterested in the Jenga robot project thus far, we did have the above conversation on the drive home from school this week. I'm gonna chalk that up as a win in my effort to bridge the digital divide of the AI era. My youngest daughter is very interested in the robot project but I'll touch on that more later.
Also, a quick shout out to people who write for a living. This is legit hard AF. I'm not that great at it but I’m here slogging away at my keyboard because AI is a lot worse than me. I hesitate to say that it won’t ever be good at it, because I’ve been proven wrong too many times. But, for the time being, it ain’t good so we shall keep on keeping on.
I do think there is something noteworthy here to call out. Software doesn't have a personality: either it works or it doesn't. Certainly there is taste involved but nothing that captures the personhood of the author, with the exception of an occasional comment or humorously named function. Software sketches an individual's approach to problem solving but tells us little about the author themselves. Actual writing requires an opinion and a personality, which, for now, AI has very little of.
Ok, enough deep thoughts for now - back to robots!
Where we left off
If you haven't read my first installment, give it a quick read here. I'll sit here quietly while you work through it.
Ok done?
Great! We had just finished our first successful pick and place. After many false starts this seemed to almost magically work once we let the LLM take charge and removed ourselves from the process. The side effect of success is confidence … whether that confidence is warranted or not is another question.
New goals and workspace updates
For our next phase, the plan was to have the robot pick from a source of Jenga blocks and attempt to place them on or near the tower build area.
Before starting the next phase we first needed to solve a few problems from our first week. The main problem was the butcher block workbench. On camera it looked nearly identical to the Jenga block and made the blocks difficult to spot.
I solved this in the last phase by using colored tape on the blocks. This provided the necessary contrast but isn’t feasible for the full tower build. So I set about changing the workspace. I taped a green backdrop from a photo lightbox onto the work area and built a simple cardboard frame to act as the tower build support. On the next run, Claude immediately called out the improved contrast.
Basket case
We again needed to find a way to let the LLM train on its own with minimal involvement from yours truly. So I started simply. I took our felt basket of Jenga blocks from our game cabinet and put it directly on the workbench, flush against the wall so the robot arm would be less likely to knock it over.
It was getting late again and so I set about with my previous approach - let the LLM train overnight and by morning we’d have the pick phase sorted.
The next morning I expected big things. We got almost nada. I found 2-3 blocks next to the basket in the morning and a lot of feedback in the session that the blocks were too close together (they weren’t).
The same problem we were having with the workspace was recurring in the basket - the blocks looked too similar and the contrast or lack thereof made it hard for the LLM to find a good candidate block to pull.
It was clear the basket approach wasn’t going to work so I decided to make the problem easier. We didn’t need to pull from a basket, we just needed to pull from somewhere.
Pez for robots
I fancy myself something of a cardboard smith. And cardboard smithery is a great skill to have, especially if you haven’t yet pulled the trigger on a 3D printer. I had an idea: an upside-down Pez dispenser for Jenga blocks. Gravity would feed a new block after each pull and make the process repeatable. I quickly set to knocking out a prototype.
In about five minutes I had a pretty solid first attempt at a small Jenga dispenser. The main problem was that the blocks fit too snugly and gravity alone did not provide enough force to drop the blocks smoothly through the dispenser.
Another five-minute prototype later, I had a perfectly sized Jenga dispenser which held about 20 blocks.
I decided after the first attempt not to tape the dispenser directly to the table, because loading it was a challenge once it was taped down. Instead, the body (the part that holds the blocks) stands free and slots into a separate cardboard base taped to the table. If things went smoothly, I could have a few refills ready to swap in. At the bottom, a cut-out plays the part of the Pez head: that's where the gripper takes the next block.
A new virtual leader attempt
It’s here that I realized that I would need to help the LLM a bit more this time around. And as usual, I was willing to expend a great deal of effort to make this as easy as possible on myself.
One thing I haven't gone into detail on yet, is the SO-101 "leader" arm that also comes with the kit I purchased. It looks very similar to the "follower" arm we've been highlighting so far but it has a handle with a scissor-like control for manual manipulation of the jaw grips. The basic idea is that someone (me, I guess?) is supposed to control the follower arm with the leader until you have enough training data for a VLA (vision-language-action model) to act on its own. The problem with this approach is that the leader arm is clunky and hard to control. I knew immediately I would want to minimize the amount of time spent actually driving that arm.
I had a wild idea. I would make a new virtual leader application. This time I would skip the mouse controls from my first virtual leader dashboard and I would use a webcam to track my actual arm motion to provide control to the robot.
I set to work and had Claude build a camera-tracking app that captured my arm in frame and drove a simulated robot arm alongside it, so I could see how the robot would track compared with my arm.
The first cut of software delivered a generic robot arm that had a similar number of joints as the SO-101 arm. The camera tracking was impressive right off the bat.
The motion tracking was overly sensitive so I dialed that down to keep the simulated arm from flailing wildly. The simulated robot arm wasn’t cutting it so I had Claude download the schematics for the SO-101 arm and in a few minutes I had a faithful virtual clone of the device sitting across the room. Despite my regular use of this technology this was a moment that still amazed me. A reminder that what we can build is limited only by what we can imagine.
As impressive as it was, I realized that holding my arm in awkward positions for long periods was going to be a non-starter. I decided to pause on the virtual leader approach in favor of another option.
Y.A.A. (Yet Another Approach)
I had two realizations that led me to change my approach. The first was that the LLM was mostly able to complete the action (pulling from the dispenser) - but was getting hung up while gripping the block in the last stage. What if, instead of using the leader to control the arm for the full operation, I only jumped in during the stages it was having difficulty completing? My only hangup with this plan was that once the follower arm was in position, it would be up to me to position the leader arm in the exact same pose to hand off control. Otherwise, the motions would not track cleanly between the two arms and this would be a deal breaker.
The next insight was that the leader arm wasn't limited to sending control instructions, it was also capable of receiving them. The leader arm could be controlled by the LeRobot software. That meant we could have the LLM move the follower into position, ask it to pause, and then have the leader arm assume the follower pose. At which point I'd guide the arm through the steps that were challenging for the LLM on its own. Did you get all that?
The above dance required some careful coordination. To further complicate matters, the leader arm has less power and torque than the follower arm. That means the leader's servos can only hold their positions briefly before collapsing into a pathetic pile on the workbench. It's quite a sad sight to see, poor little robot.
Stream Deck FTW
My Stream Deck used to add joy to my daily workflow, allowing me to mute and unmute my audio, toggle video on and off and skip any Taylor Swift song that wasn't a complete banger (lol kidding they are all bangers).
However, in the last year, Microsoft nerfed the Teams integration for the Stream Deck and made those functions useless. Spotify did something similar, and its integration suffered as well. With that, my Stream Deck became unused real estate on my desk and it hasn't seen action in months.
Given the new need to coordinate a delicate dance with the LLM, I wondered how easy it would be to utilize my Stream Deck for the task. I plugged it into the robot controller and asked Claude to assign the new functions:
At this time the Raspberry Pi USB bus was getting overloaded with the newly attached devices (leader and Stream Deck) and it started to reboot intermittently during the training runs. I decided to repurpose the laptop running the 45-degree camera view to run the robot controller and host the USB devices.
The big dance
After working through the gripper challenges a few times (I needed quite a bit of training myself) - I managed to get the LLM the data points it required to complete the grip and pull operation successfully. There was some back and forth and a few minor adjustments but those were all directed via prompting.
The final test was to see if we could clear the full Jenga dispenser without any further intervention from me. I set it to work, and by the time I was back from lunch it had cleared 15 of the 20 blocks. My chat session showed it tried the next block, couldn't grip it, and decided the dispenser was empty. I assured it that it was not empty and it proceeded to clear the dispenser fully into the drop area. Pez dispenser emptied!
Success!
We're calling this a win for now. And next week we're going to attempt to place the first row of blocks onto the tower build area. The bigger win for me this week was that my youngest daughter has taken a strong interest in the robot project and she was thrilled when I let her control the robot using the leader arm. A huge smile and a few follow-up "could we do some robot stuff" made all the week's effort worth it and next I'm going to coax her sister to try to control the robot as well. Wish me luck!