It's pretty hard to totally clean the wheels once you've applied a certain color of rouge because the cutting particles are so tiny - you can always go coarser, but never the other way around. They sell rakes to clean off the wheels (remove excess wax, metal & polish etc), but these won't really get rid of enough emory that you could re-apply red or white on top. Perhaps they make some sort of chemical that does it, but I've never seen it.
My guess is that you're probably getting the swirls because of the drill (I assume you're holding it in your hand?). If you could find a good way to mount the drill to a table, or better yet, spend the $50 to get a bench grinder motor (you don't need a great one to start off with), you'll find everything is alot easier & comes out more consistant. It's also better to have a stationary wheel so that you can use 2 hands to hold the piece (safety and more control over pressure & angle). I have no idea about how fast your drill is spinning, but the electric motors spin pretty darn fast, so that might also have something to do with it as well.