Following and adding to srust:
The wheel should always move free as it is on the outer races of the bearings, while the inner races of the bearings are a slip fit over the axle and may oxi-weld themselves to the axle over time.
Only if the bearings had locked up, a very serious and rare condition would you expect to see the axle turn with the wheel.
So at any exposed seem, you should try to get some penetrant into the interface between the axle and the components it slides thru. PB Blaster would be the most convenient aerosol, but acetone/ATF 50/50 in a spray bottle would be a stronger penetrant. Even if the seam seems impenetrable, spray it. WD-40 is most popular but the weakest.
Ultimately you are correct. If everything was fresh and clean, the axle should tap out with little resistance. Wailing on it probably wouldn't hurt much, but is not the most elegant of approaches.
As srust says, the aluminum of the caliper holder could oxi-dize to the axle, but some penetrant will break that bond easily as its not a chemical fusion, just crud.