I couldn't feel any movement until I removed the wheel and shocks and had the arm swinging free.
This is typical. When the shocks are fully extended and locked the swingarm becomes a set of 4 triangles, so it rigidly locks and many people then don't "see" the looseness. Also, the phenolic bushings, once soaked with grease, are spongy in terms of the several-thousand-pound forces that are applied sideways to it when recovering from a turn, even a small lean-over angle.
When I roadraced in the 1970s, this stuff was CRITICAL to making the bikes handle at all: the stock bikes would always wobble (and fall behind the pack for insecure handling traits) when coming out of turns, while those we modified with solid bushings (some even with interference fits) could be observed to come straight up out of a turn.
I might add that we also had the tapered steering head bearings added, but the steering heads were only an issue after the bikes had more than about 8k miles on them. Many of the tracks were go-kart racing tracks, so the speeds were low, like 60-70 MPH tops, which gave lot of going-into and coming-out-of turns to look for these things. Before I switched to the bronze rear bushings in my 750 I could not even stay in a line when coming "up". Here in the Rockies, where speeds are nearly triple-digit when coming down the steeper hills in heavy traffic, rigid swingarm bushings make all the difference.