Gonna take a couple tacks with this one.
-Physically hardest? All of the parts swapping I've done on my CB750. You name it, I've had it apart, several times. Put it this way- I'm on my third frame, and the fourth is currently in the barn awaiting cleanup. The upside? I've done this so many times I can frame swap this entire bike in my sleep.
-Mentally hardest? The first time I pulled apart my GL1000 carbs. Long story short, they are standard 70's automotive carbs, which means they are MASSIVELY overcomplicated compared to regular motorcycle carbs. Two main jets, everything is press-fit (with o-rings on everything), the jets are held in place with a clip that only compresses once the bowls are screwed in, and a plenum that has all the air and fuel passages drilled internally. Now that I know how it goes together they aren't too bad, but the first time was ROUGH.
-Annoyingly hardest? Doing anything on my Velosolex. Mostly because everything is a pain despite being a dirt-simple design. Essentially? There is a single order that bike comes apart. You have to do something on step 3? You do step 1 and 2 first. Need to do something on step 30? You start at step 1 and do all 29 steps first. There's no shortcuts, everything is inconvenient, and the resulting bike is the most gutless thing I've ever ridden.