Yeah, stop lugging your motor!
4k rpms in 5th gear is on the low end, so whacking the throttle will produce those results. These are not fuel injected, electronically controlled engines; they're mechanical carbs and the motor needs high revs to perform. Downshift, rev up above 6k and see how it goes.
Too many think because these bikes are 40 years old they need to be nursed along. Couldn't be further from the truth. If you want it to last and run well, run it up the tach. 6k-9.5k is the sweet spot for the motor. Riding around at 2.5k-4k will make for an unhappy motor.