I'm in exactly the same position, mine has jumped out of 4th 3 times while on full yee haa, I have mine due to go back together in a few days, I have replaced all the trans bearings including the two needle rollers in the primary hub because my trans had a pretty decent whine. I don't reckon there was a thing wrong with any of the ball bearings that came out but the two needle rollers weren't great.
Like you I can't find a single thing, I even bought a used set of forks with the drum n shaft but mine measure up a tad better, so I'm just hoping new bearings will help with the gear jump??? Maybe, maybe not but I can't see anything else that could be a problem.