I have a four year old! He does pretty much everything...but hes not ridden much more than a couple of times a week at the moment.
I had him since he was 7 monthsld - I used to walk him out like a dog on roads and accross moorland. When he was 2 1/2 I long reined and lunged him for about 10 mins at a time. I backed him as a 3 year old - he was nearly 16hh and had 10 inch bone, so he was strong enough to sit on, although I am aware bones arent fully developed until around 6 years. He was pretty much left for the winter, and I rode him out occasionally - Im lucky hes the sort of horse who I can leave for weeks without riding, and then just jump up and go! He turned 4 July just gone, and has been ridden around 1x a week (sometimes more, sometimes less) over the summer. Its all roads and train tracks around here though, so its mainly a steady walk and then the odd canter/gallop when we get to a farm....even then its just the length of the field.
Hes a lazy boy - and very trusting, so I took advantage of it, and I used to sit/climb all over him when he was lying in the field. It got him used to having me on his back, so when it came to backing him, I pretty much just hopped straight up (I wore a body protector and hat though) because it wasnt really new to him.
Ever since I first got him, I would put tack him, bags, and even bounce balls off him, so hes used to pretty much everything. Hes still a bit green to ride, just because I havent taught him all the "universal" commands, but I can mostly get him to do what I want, which is fine by me! Ive also moved my horses to a field right next to a construction site - yay! They get to be right up next to tracters and lorries and all sorts of things 24/7! Its desenstising them whilst Im not even there - bonus! the horses are separated from the work site by a metal fence...but its almost like theres nothing there - I thoughtheyd be scared, but they stick their little noses through to be stroked by the builders, and are the first on the scene when something comes crashing down - they are sooo nosey! Builders have my number incase anything does go wrong, but they are pretty safecant see how they can get into site unless a tracter takes the fence down!
Anywho, I believe that as long as you arent over-working or over-loading the horse with weight before they have developed enough (also depends on horse and breed), you can get started on most things from a foal! All my horses are tied on their own (without hay) whilst I school/groom or muck out. I think its good practice for them (I dont leave them out of site though - just incase). The all get a saddle chucked on at an early age, they all get desensitised to bags/flying coats/mad dogs/kids, they all get exercise/footballs bounced off them, long reined/lunged/free schooled,and walked out in traffic (alone and in company) from as early as possible.
I make sure I dont throw them in at the deep end though - I let them take things slowly, and work them up to where I want them - how long it takes depends on the individual horse. I dont over work them - my 15 month old filly gets free schooled/lunged 1x a week, 2x each way. Its mainly for bonding, and getting her to listen, rather than building strength. she also gets walked out (when Im free), and has been desensitised to flying balls etc...you can bounce an exercise ball off her head, and she doesnt react at all! I also hang T-shirts etc off her ears so she gets used to things flapping in her face (there are lots of flags in the village, and some youths around here tend to like pulling them off, and they sometimes fly around, so I dont want her to freak if one does get her in the face, lol).They also get a full body wash with a hose pipe - which they love!
Ive been fairly busy though, so most days I only get an hour or so in the field to do the horses - so they arent over worked incase anyone out there thinks they are, lol. I just believe that a horse should be prepared for the big bad world as young as poss!