# rl-reach: train a robot arm to reach with reinforcement learning

Companion project for the TechniaHQRobot guide
"How to Train a Robot with Reinforcement Learning: A Complete Beginner's Guide"
(https://techniahqrobot.com/guides/train-robot-reinforcement-learning/).

A three-joint MuJoCo arm learns, with PPO from Stable-Baselines3, to move its tip within 2 cm
of a random target and stay there. Everything runs on a CPU, in simulation.

## Files

| File | Purpose |
| --- | --- |
| requirements.txt | Pinned package versions (Python 3.12 or 3.13) |
| check_setup.py | Prints Python, CPU, disk and package versions |
| assets/reach_arm.xml | MuJoCo model of the arm, servos and target |
| reach_env.py | Gymnasium environment |
| test_env.py | Environment checks; run before training |
| train_ppo.py | PPO training with logs, checkpoints and evaluation |
| plot_training.py | Training curves from the logs |
| evaluate.py | Success rate and distance for PPO, a random policy and a Jacobian controller |
| view_policy.py | Watch a trained policy in the MuJoCo viewer |
| executed/ | Outputs recorded when this project was checked (see below) |

## Commands

```bash
python3.12 -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu   # Linux only
pip install -r requirements.txt
python check_setup.py
python test_env.py
python train_ppo.py --timesteps 500000 --seed 0
python plot_training.py --run-dir runs/ppo_seed0
python evaluate.py --model runs/ppo_seed0/best_model/best_model.zip --episodes 200
python view_policy.py --model runs/ppo_seed0/best_model/best_model.zip   # macOS: mjpython
```

## What was executed

See executed/execution.json for the exact environment. In short: the model and environment
were run with MuJoCo 3.16.0 and Gymnasium 1.4.0, test_env.py passed, and evaluate.py was run
for the two baselines. plot_training.py was run only on synthetic log files to test its parsing.
train_ppo.py, view_policy.py and the PPO part of evaluate.py were not executed, because PyTorch
and Stable-Baselines3 could not be installed in that environment. No PPO result is claimed
anywhere in this project.