I don't think you're too big for him for light work - I agree with others, you want to build him up a bit. I'd also consider a lightweight saddle. I always had synthetic with my old pony, they're just so much lighter, but of course the fit makes all the difference. See how you go on with him, and if he lets you know he's uncomfortable, you may have to reconsider.