The bearing opposite the retainer is free to slide as far into the hub as it likes, the bore is not stepped. It will eventually fall off the end of that bore inside the hub if you want to push it that far, but that would be a bad idea. Without the retainer, the bearing that should have it can move out of its bore while the opposite one moved into its bore.
You can make up the spacer-bearing-inner spacer-bearing-speedo drive sandwich without the hub if you want, tighten the axle, and mount it on the forks. The rear is a bit different because of the sprocket carrier but again you can make it up without the hub if you feel like it.
The inner spacer just makes a solid stack of the assembly, so the axle nut is not breaking the bearings apart by putting axial load on them: this type of ball bearings are not designed for axial load. Automotive axles have tapered roller bearings that are.
Without the retainer the hub will slide side-to-side relatively freely, limited by the tightness of the hub bores on the outer races. Some will know my ideas on the bores, that the beareings should be a snug sliding fit and no require a press or major force to install... whatever, the hub will be free to move axially without the retainer. I don't know if it would move enough for one of the bearings to come out of its bore bur otherwise it would move until two parts not intended to touch, do.