The spacer stack is holding the bearing in place by its inner race. So it ain't goin nowhere. But it was designed to be held in by its outer race primarily via the retainer. Still if you notice the bearing on the other side doesn't have a retainer. Hmm. It was designed so the outer race could float. Why can't both sides float?
I can't answer that other than to repeat what BobbyR said: Honda put a retainer there for a reason.
So you could probably have continued riding it without mishap. But now that you know, I could never ride it a mile again.
Removing the retainer takes a trick. Before i knew the trick, I destroyed a retainer and a hub getting one out. That's probably what happened to yours.