Sorry disagree here- athletic fitness benefits your horse. Being rigid is not the same as being strong and stable. A rider should be able to maintain a neutral spine and pelvic position and move with control in the saddle- this helps the horse with balance too.
Off horse training has been shown to improve peoples riding in research. Yes lunge can help too - but an unbalanced rider on a horse on tight circle repeatedly can be considered a welfare issue now given we know that can cause back pain in horses.
Its finding a riding fitness/exercise program that works for you- your current level, what you want to achieve, previous injuries and medical conditions etc