You may need to remove the rear wheel again to inspect the brake parts inside the drum. The brake shoes pivot at one end, and expand outward at the other end when the lever arm is rotated. It sounds like one or both of the shoes may not be positioned correctly, and it is requiring excess rotation of the lever arm shaft to cause the brake shoes to engage the drum. It is also possible that the pivots are damaged, broken, etc. I would agree with you about not riding it in this condition, if anything is out of position or damaged such that it could become loose, it could cause the wheel to lock up while spinning, and that could ruin your day.