I just did the same thing last weekend. I spilled oil all over the header/muffler connection, so I figured I should pull the whole thing off and get it clean, lest it burn.
The collars that go around the ends of the header pipes (and under the flanges that bolt onto the head) have flared ends that point away from the head. The flared end of the collar needs to seat firmly in the recessed space that’s on the inner surface of its flange. The straight side of a well-seated collar is what pushes on the flared end of the header pipe, firmly onto the crush washer.
The seal comes from the crush washers. The collar just pushes on the header pipe. Those collars are pretty flimsy, though. They get bent out of round. You have to be sure they sit in the notches inside the flanges.
I found that it helps to bolt up the headers loosely, then rest the muffler end on a stack of wood or something, so that it doesn’t keep sagging and pulling the headers away from the head. The headers don’t line up perfectly square to all four holes. They need to be coaxed and pushed and pulled into place. So you have to bolt it all up loosely, then tighten here and there, then wiggle the header pipe, then tighten some more, then wiggle some more, etc.
If you tighten one flange fully in one go, you end up with another one that doesn’t seat properly.