A robot policy is the neural network that turns what the robot senses into motor commands. Teams spend weeks training one, usually in a simulator built for training, and then measure it on the same kind of scenes it was trained on. It looks finished. Then it meets a wet floor, a parked bus or a ramp with a railing, and something goes wrong that no training metric predicted.
Our view is simple: a trained policy should be tested on purpose, in scenes it has never seen, before it moves real hardware. That is cheaper than a damaged robot, and far cheaper than an incident with people around.
Training and testing are different jobs
Training simulators are tuned for speed. They run thousands of copies of the robot at once on simple ground, with randomised friction and pushes, so the policy learns to stay upright. That is the right tool for learning, but it is not a test. A test asks a different question: in the places this robot will actually work, where does it fail, and why?
So at SIS we keep the two apart. Policies can be trained in a fast training simulator such as NVIDIA Isaac Lab. Testing happens in MuJoCo, with real-time physics, realistic scenes and the robot's own sensors, with the policy used exactly as it will run on the robot. We do not retrain it.
What a stress test looks like
We take a policy as it is and run it through scenes built to find its limits:
- A forest trail with rocks dropped in front of the robot while it walks
- A city street with kerbs, bins, benches and traffic that does not stop for the robot
- An office building with doors, stairs, a ramp and furniture that can be moved during the run
- Wet ground: patches of mud, slick floor and ice that look the same as dry ground to the sensors
Every run is logged: falls, slips, collisions, distance slid, time to goal, and how often the planner had to change its path. The result is a report and videos that show exactly where the policy breaks.
In the video above, a Go2 quadruped is given a flag on the far side of a busy road. Its 360° lidar maps the bins and benches, it treats low steps as climbable, and it tracks every vehicle, including cars that could be hidden behind a bus. It waits at the zebra crossing until no vehicle could reach the crossing while it is on the road, then crosses. The walking policy is by its original authors; we built the sensing, mapping, planning and the test scene.
What our tests found
These are real findings from our own simulation runs with publicly available walking policies:
- Lidar cannot see a wet floor. On patches of mud, slick floor and ice, the robot's lidar map marked nothing at all: to the sensor, ice is just flat ground. A quadruped that never slid more than a metre on dry ground slid up to 20 m while turning on ice. It never fell, but a humanoid on the same patches did fall. A camera, or a policy that feels slipping through its feet, is needed.
- Sensors have blind spots. A small rock dropped one metre in front of the walking robot sat below the lowest lidar beam, and the robot brushed it. Larger rocks were seen and avoided every time.
- Planners can trap themselves. In a U-shaped trap made of three benches, two popular local planners (artificial potential fields and the dynamic window approach) got stuck, while a global A* planner found the way out. Choosing the planner is part of the test.
- The building matters as much as the robot. A humanoid's legs rubbed the inner railing of an office ramp and caught on a short landing. The fix was in the layout and the look-ahead distance, not the policy, but nobody would have known without running it.
- Traffic is unforgiving. In early city runs, a humanoid crossed safely in only one of three attempts and was hit once. That is exactly the kind of result you want to see on a screen, not on a street.
None of these failures shows up in a training curve. All of them showed up within minutes of putting the robot in a realistic scene.
Why this saves money
A real robot costs tens of thousands of dollars, and a single fall can mean weeks of repairs. A day of simulation testing can show a team which situations to fix before any hardware moves, which situations need more training data, and which need a different sensor. It also gives clear evidence for a customer or a safety review: here is what the robot does on ice, in traffic, on a ramp.
How we work with robot teams
We test a team's already-trained policy, as delivered, without retraining it. The team sends the policy and the robot model; we build or adapt the scenes that match where the robot will work, run the tests, and deliver a report with videos and the failure cases ranked by how often and how badly they happen. If the team wants, we also help fix what we find, from sensor setup to planning.
If you are building a legged robot, a humanoid or a mobile manipulator and want to know where your policy breaks before your customers find out, see our robotics software and robot simulation work, or contact us.