I'd cut it out at this point. Drill most of the way through the retainer carefully in two or three places, using a bit that just smaller than the retainer is thick. Then use a chisel to remove the retainer in chunks. If you're careful you can avoid dinging the threads.
mystic_1