Just make sure the bearings aren't compressed. Usually tighten it right up then back it off 1/4 turn and your set.
I run my KTM enduro bike steering loose so it's quick to turn, downside is the odd huge tank slappers. Got used to it now and prefer the steering like that, it's never spat me off either, just a wild ride.
That's cool on soft dirt, maybe less so on Tarmac!