I dont know for sure, but isnt there a coating of some sort on the cam to prevent wear? or is it just on the rocker?
If so, i'd be worried the rust was into/through the layer. Removing it might create a pitted area increasing wear on the rocker.
Got a picture?