Deceleration wobble beginning between 50-40 mph is very, very common. It can happen with any style of bike. Even a bike with everything in correct adjustment and appropriate tire pressure may do it, especially as a front tire wears. You need to fix those swingarm bearings, but even after you do, you may have the wobble. Every bike I've had from dirt bikes, dual sports, sport bikes and standard bikes have all done it to some extent. Ask a Gold Wing owner of any vintage (bike, not rider) about deceleration wobble.
The advice the Hee Haw doc gave is valid. If it wobbles when you take your hands off the bars, then don't do that.