Well for what it's worth I think I've finally run down why the shifting on this bike is so sloppy.
The drum shifter - that is the device which physically rotates the shift drum has a very loose fit.
The drum shifter is connected to the shift drum with 2 spring loaded pawls - those pawls are what contact the shift drum and rotate it. See the image - they're parts 10, 11, and 12.
The connection between the pawls and the shift drum is simply very sloppy - even with brand new pawls. The shift drum isn't worn out, the pawls are too thin.
But here's what I think was actually happening for me to have missed all those shifts. If the shifter shaft is pulled out even a very small amount the drum shifter will no longer contact the shift pedal and it simply won't shift. Basically the gears will no longer be in contact and you can pull up on that shifter all you want - the actual drum shifter won't rotate. It's what I think was happening in my case. So my real solution here is to find a way to get that shift shaft as solid in there as it can get - no in and out motion.