Yes, the further away the end of the brake rod is from the pivot point of the swingarm, the more effect swingarm deflection will have on brake action.
My original post should have said "... more than the stock setup".
In extreme cases, hitting a bump or dip in the road could result in the rear brake applying itself. So, try to keep the two pivots as close to each other as possible.
In stock configuration, the brake rod end pivot (where it meets the pivot arm that goes through the frame) is quite close to the swingarm pivot point, like within in inch or so, and in a plane connecting the rear axle to the swingarm pivot. This position gives minimal deflection on the brake arm when the swingarm moves.
As Ron says, just something to pay attention to. Remove your rear shocks and raise the swingarm to it's highest deflection point to see what effect it might have on your brake linkages.
mystic_1