In future, the way I take a grip off that I want to re-use is get a long thin screw driver under the rubber and give a short spritz of WD-40 alongside the screw driver. Rotating the screw driver as if using a pepper grater further gets the WD-40 around and the grip slides right off.
I'm gussing you have bar-end mirros? The rubber is now rubboing too hard against that.
Or, youpshed the rubber far enogh away to get the gizmo on, but rubber memory got it back too original shape and is now rubbing too hard agaisnt that,