I only ride and spend time at yard when they are at school.I chose to give up horses when I had young children,and only got back into it again when the youngest started full time school,so was never an issue for me when they were babies,toddlers etc.
Weekends and school holidays I only do flying visits to the yard to do the necessary,or OH does it for me.Is a bit of a killer,especially this loooooong summer holiday,but that's the deal I made with myself before getting back into them.None of my kids are interested really in horses,unfortunately,and always said I wouldn't force it on them,or make them fit in around my hobby.OH takes pity on me sometimes and tells me to go ride
,but I rarely do when kids are at home,that's their time IMO.
Maybe you could rope in some help from relatives if you are finding it a struggle??