I decided I had nothing to lose and gave the sand paper a try last night.
I bought a $3 pack of self adhesive paper that had various grits, and stuck a strip of the paper on each brake shoe.
Put it back on the bike, and ran it on the center stand in first gear, and slowly applied the brake. I could feel the sand paper start to work down the two high spots that were in the drum liner.
I added more pressure to the brake as the spots wore down, until there were no more pulses in the brake pedal.
I repeated this with a finer grit paper to smooth out the drum surface after the heavy grit, and in less than 10 minutes I was left with a smooth drum with no high spots or pulsing in the brake pedal. The brake works like it should, smooth and no issues.
No doubt that the liner will likely move more in the future, if the rust in-between the liner and the hub is expanding. But it took 40 years to get to where it was, so I think I have some time to just ride it as is for now. If it gets to be an issue, I can either sand it again, or replace the hub with a spare that I have on hand.
For now, I'll call it a win.

I took the bike out for it's first real ride since the 80's, and it was awesome.
Pete