Yes, you need them, that spacer is what keeps the bearing races centered in the hub to keep it from binding. If they weren't there, when you either tightened the axle, or a side load is placed on the bearings (like when you corner), there would be problems. At the least, it would force the inner bearing races toward each other causing excessive drag, or at worst the bearings might come apart.