You'll probably have to disassemble the starter clutch and look at the rollers inside. If they have flat spots on them, you'll have to replace them. Maybe also polish up the crankshaft a little to remove the roughness? If you don't then the new rollers will wear quickly and you will be right back in there, replacing them again.
If the engine has lots of spark advance (like a fixed-advance unit), maybe consider changing [something] to a less-advanced startup setting, like 0 to 5 degrees advance. If the engine is very high compression or has forged pistons, this will be needed. Forged pistons will tend to bind in the engine if it is run briefly (like 5 minutes) and then shut off for about 5-10 minutes, until the thermal equilibriums balance out again. This is why those pistons usually require more clearance than cast pistons. When it binds, the starter clutch can take a real beating!