Ok, I think this may be resolved. Still need to really get it up to speed, but I just got it over 50 and it had smooth road feel the whole way. I think that work I did on the back wheel did help, it was somewhat out, but I also upped my tire pressure. I had around 32 in the rear, so I upped to the high end, 38 (cause I'm a big heavy mf, so I consider myself to be a rider + passenger) and I found out that the pressure in my front, which seemed fine, was fairly low, so I took that up to about 35.
I'm a little surprised at how much difference a few PSI makes, but I know how much it means in a car, so I guess I shouldn't be to surprised.