The brake arm would never swing that far anyway unless something was broke, cause the caliper would hold things in place. Some guys run without the adjustment screw entirely.
When you let go of the brake lever, the only thing that retracts the pad into the caliper is the caliper piston seal. This deforms when you squeeze the brake lever and the piston moves out, then it retracts when you let go and retracts the piston.
The amount of "spring back" from the piston seal is very small. Call this distance X. The idea is to set the adjustment screw so that the inner (dead) pad moves X/2 (half of X) distance before contacting the rotor. That way, when you let go of the lever, BOTH pads retract from the rotor, the live pad because of seal retraction, the dead pad due to spring pressure. End result: neither pad touches the rotor when the brakes are released.
mystic_1