# Teaching a Balance Bot to Stay Up With RL

Instead of hand-tuning a PID loop, Make's Shawn Hymel trains a neural-net policy with PPO to keep a two-wheeled robot upright, starting in simulation.

- Category: Hardware (build)
- Made by: [Shawn Hymel](https://makezine.com/article/technology/robotics/train-a-balance-bot-with-reinforcement-learning/) on Make:
- Original: http://github.com/ShawnHymel/make-reinforcement-learning-for-robotics
- On Afterware: https://afterware.xyz/p/teaching-a-balance-bot-to-stay-up-with-rl

## Can I build this?

- Difficulty: Intermediate
- AI-buildability: 4/5 (Most of the work is Python simulation, reward design and training plus deploying a small policy to firmware on an off-the-shelf kit, with only light hardware handling.)
- Time: A weekend
- Cost: $100–150
- Parts: IMU, M5Stack Bala2-Fire self-balancing robot kit
- Tools: computer, soldering iron

Estimates based on the original write-up. How the AI-buildability score works: https://afterware.xyz/ai-buildability

## Build sheet

- Method: PPO reinforcement learning
- Training: Simulation on a computer
- Sensors: IMU and two motor encoders
- Hardware kit: M5Stack Bala2-Fire (optional)

## Starter prompt

```
Based on "Teaching a Balance Bot to Stay Up With RL" on Afterware: https://afterware.xyz/p/teaching-a-balance-bot-to-stay-up-with-rl

Set up a PPO training environment in Python for a two-wheeled balance robot in simulation. The observation is IMU pitch, pitch rate, and wheel encoder velocity, and the action is left and right motor speed. Reward staying alive and upright, and penalize lean angle, abrupt action changes, position drift, and yaw rate. Export the trained actor network so it can run on an ESP32 or Raspberry Pi.
```

## Story

Classical control has balanced two-wheeled robots for decades. This guide swaps the hand-tuned loop for a reward function. The agent gets points for staying alive and stays upright, and it loses points for leaning, jerky motor commands, drifting off its starting spot, and spinning in place. Trial and error does the rest.

Most of the work happens on your computer. The training runs in simulation with PPO, using the CleanRL implementation and a repo with the environment code. The policy only needs the actor network to run on the robot, so the critic gets dropped at deploy time. If you want real hardware, the guide points to the M5Stack Bala2-Fire self-balancing kit.

Worth stealing: the reward design. Each term is a plain penalty you can tune. Once you have one working balance bot, the same loop scales to harder robots, including Pollen Robotics and Hugging Face's Microduck bipedal.

## Resources

- [Project repo](http://github.com/ShawnHymel/make-reinforcement-learning-for-robotics)
- [CleanRL (PPO implementation)](http://github.com/vwxyzjn/cleanrl)
- [Netron model viewer](https://netron.app/)

Tags: reinforcement learning, ppo, balance bot, two-wheeled robot, imu, simulation, m5stack

AI assistants can search all Afterware builds through the MCP server at https://afterware.xyz/mcp (setup: https://afterware.xyz/agents).