You can go a long distance on pavement with a belt drive, but gravel roads are trouble. When a stone gets between the belt and pulley the belt cords break because there's no slack and the belt resists stretching quite strongly. Hopefully the belt just gets a hole or tears a bit but they can and do completely snap. A rock in a chain isn't recommended but they generally pass through with little or no damage because of the normal slack in a chain, and only small rocks can balance on a 1/2" wide chain. The wide belt can catch pretty big stones. I have seen a picture of a stone sweeper mounted on a Harley swingarm that pushes any foreign objects off the belt.
You would probably have trouble with the Honda swingarm geometry because belts have very little tolerance for changing tension. The belt will skip when slightly slack. Harley has obviously designed their factory frames for belts, and a hardtail chopper won't have any problem.