Thanks for the reply.
I guess it's possible it has been worked on before, but I did have to drill out the punch marks on the bearing retainer, and the threads didn't looks like someone had previously removed it and forced it back in or anything.
Some of the rotor bolts were pretty rusty, I had to give them a few whacks with a brass punch before they came out. 2 out of 5 were covered in rust on the surface. I took the rust off with a die grinder, but that may indicate some moisture found its way into the hub.
Like you said, I'll let it soak a couple of days, I have to wait until the new bearings get here anyway. I just wanted to make sure I wasn't missing something before I start wailing on it. I'll try using something harder than the rubber mallet, with the nut on to protect the threads. Man, that was a bear to get off, thank god for air impacts.