The rectifier is connected to the battery all the time, it has nothing to do with the key switch.
There's a healthy discharge as soon as you turn the key on, but removing the rectifier ground should not make any difference... that's the odd thing about what you report. No power should be passing through the rectifier except when the engine is running. Check your main ground wire - a large wire from battery "-" to an engine mounting bolt. If the entire bike is grounding through the rectifier small wire, I would expect that to melt when the starter motor is used. You must have a very good connection direct from battery "-" to the frame and engine case.
The alternator field coil and ignition take a lot of current whenever the key is on, through the MAIN fuse. You can remove the HEAD and TAIL fuses and reduce load but the field coil and ignition will always be on unless you detach the black wire from the regulator and put the kill switch to OFF (STOP?).
Try removing the HEAD and TAIL fuses and that black wire. If you switch the kill switch to OFF the remaining load should be very light: maybe an indicator light or similar? I can't check the wiring diagram to say for sure right now.
The very early 350-Four bikes only had one fuse, I believe. Could be wrong - but if that's your bike then switching kill to OFF and removing the black wire from the regulator would leave only the lighting load.