Hardwareby Zeller, MA · HacksterHardware · build
Teaching a Balance Bot to Stay Up With RL
Instead of hand-tuning a PID loop, Make's Shawn Hymel trains a neural-net policy with PPO to keep a two-wheeled robot upright, starting in simulation.
Build sheet
- Method
- PPO reinforcement learning
- Training
- Simulation on a computer
- Sensors
- IMU and two motor encoders
- Hardware kit
- M5Stack Bala2-Fire (optional)
Can I build this?Intermediate · AI 4/5 · $100–150
- Difficulty
- Intermediate
- AI-buildability
- 4/5
- Time
- A weekend
- Cost
- $100–150
Most of the work is Python simulation, reward design and training plus deploying a small policy to firmware on an off-the-shelf kit, with only light hardware handling.
Tools
computersoldering iron
Starter prompt
Based on "Teaching a Balance Bot to Stay Up With RL" on Afterware: https://afterware.xyz/p/teaching-a-balance-bot-to-stay-up-with-rl
Set up a PPO training environment in Python for a two-wheeled balance robot in simulation. The observation is IMU pitch, pitch rate, and wheel encoder velocity, and the action is left and right motor speed. Reward staying alive and upright, and penalize lean angle, abrupt action changes, position drift, and yaw rate. Export the trained actor network so it can run on an ESP32 or Raspberry Pi.
Estimates based on the original write-up. How AI-buildability is scored. Building with an AI assistant? Connect it to Afterware (free MCP server).
Story
Classical control has balanced two-wheeled robots for decades. This guide swaps the hand-tuned loop for a reward function. The agent gets points for staying alive and stays upright, and it loses points for leaning, jerky motor commands, drifting off its starting spot, and spinning in place. Trial and error does the rest.
Most of the work happens on your computer. The training runs in simulation with PPO, using the CleanRL implementation and a repo with the environment code. The policy only needs the actor network to run on the robot, so the critic gets dropped at deploy time. If you want real hardware, the guide points to the M5Stack Bala2-Fire self-balancing kit.
Worth stealing: the reward design. Each term is a plain penalty you can tune. Once you have one working balance bot, the same loop scales to harder robots, including Pollen Robotics and Hugging Face's Microduck bipedal.
More like this
More like this
