I have two things for you on this:
#1.) I was working on an old Kawasaki once ('80(ish) KZ550). The only way I could #4 cylinder to run was if the bike was leaned over at a substantial angle. Otherwise the running condition was very similar to what you describe. The problem turned out to be an issue with that carb itself. When building the carb, there were several passages that were drilled into the body at the factory and then were blocked off. One of the blocks had come out causing a HUGE vacuum issue within that carb. Hence - no gas was going into the cylinder unless choke was pulled or bike was tipped over enough to cover that hole. Most motorcycle carbs I have seen are built the same way. Might not be a bad plan to just give it a close once over, right next to another carb, and look for any holes seem out of place.
#2.) This also sounds a lot like the problem I had with my current CB. As TwoTired says, any change in the configuration throws things out of whack - Your exhaust might be part of the issue. With MY CB, I have a VERY altered set up (big bore, ported, exhaust, pod-like filters). For over a year and a half I chased a hanging/racing idle very similar to your issue. What the final answer to the problem was turned out to be the idles jets were too small. Basically, I had everything else turned up so much that the gas for my "Idle" was coming right out of my main jets. Hence when I revved up to the 4-5000 rpm range, it would just stick there. I COULD get it back down to a normal-ish idle, but as soon I opened the throttle the revving came right back.
Take the specs, set your carbs to stock, and start from there. Like TT ssaid, you may need to open your slow circuit to get more gas from your idle.