I just did this yesterday.
1. I removed the bolt (obviously)
2. Soaked both sides with WD40
3. Took the handle of my floor jack and beat the baffle from the inside, and knocked it out.
The thing was in their pretty good , and put up quite a good fight. So dont be bashfull, and give it a few good hits, and it will eventually pop out...