I personally have pulled many, many sets out of many bikes and they do not move around quite as much as you think even after over 85,000 miles.
Just what I have seen many, many times WITH MY OWN TWO EYES. Not what I was "told" by people that haven't actually experienced this.
Eric