Something is amiss... There is something not right with the way your back wheel is set up, or somehow the brake stay is too short. the stay should be at an almost right angle to your drum mounting ear, and the arm that is pulled by the brake rod should angle back towards the rear, at a slight angle, so that when the pedal is depressed, the brake actuator arm is pulled at a right angle also. I would pull the rear wheel, remove the drum baking plate, and check your brake shoes for uneven wear front to rear. Check with some chalk on the shoes, mark then around the circumference, and then insert them back into the drum, and rotate the wheel applying the brake, to check the contact area of your shoes. If there is any type of uneven contact between the shoes and drum, it will show up doing this. Erne, do you know if the rear brake stay is the stock one? Or, has it maybe been swapped out at sometime? Sometimes, the PO may have taken the brake arm off the splined shaft, and re-installed it one spline off, that could give you a poor angle of pull with the pedal and rod. as has been stated, you need a 90 degree angle or as close as you can, when the brake pedal actuates the shoes against the drum. This is for good leverage.
Charlie