I wear a hat when I deem it necessary. If I am dealing with a youngster or someone who's likely to rear and paddle at me.
I have been known to put my hat on and use it back at a particularly bloshy horse with no awareness of personal space. He nutted me, so I nutted him back...and a game of Chinese conkers began.
Otherwise I put the risk of every day delaing with the guys I have, who are already trained and laid back, as very low indeed, and I as a free thinking adult don't wear a hat for everyday messing about.
Do you make your vet or farrier wear a hat round your horses? Do you stop anyone in the street making a fuss of your horses if you meet them?
Come on, accidents happen, but lets use a bit of common sense and have a bit of faith in your own ability to asses risk.