I agree, why anyone wouldn't wear a hat while riding is beyond me. In fact, I had this very conversation last night with an American friend: wearing a hat is not compulsory in rodeos, nor barrel racing I believe, and it absolutely bewilders me. This is a high risk sport, and the risk of injury is huge, so why not take such a simple step? They all wear cowboy hats, so if they're wearing hats anyway, why they wouldn't wear one that could potentially save their life is beyond me.
Anyway, back to body protectors: I wear one every time I ride. I started learning without one, but decided to get one after my sister had a fall and was seriously hurt, and the doctors said without it the damage would likely have been life - limiting. I was bucked off twice a couple of weeks ago but I was able to roll and get right back up on my feet and back on the horse, so I don't find them restrictive in that sense, nor do I find riding in one restrictive, but that might be because I've ridden in one for so long.
I'm easily the oldest rider that wears one at my yard, as it's mostly kids who wear them, but I don't care!