Interesting question and I think it reflects shifting priorities in your life. I decided to pursue a career as a golf pro in 1999 at the age of 36 with the idea of getting a job as an Assistant Pro, practicing hard, and preparing myself to play on the Senior PGA Tour when I turned fifty.
While I had some success, I won my PAT Tournament, finished second and third in a few others, I discovered that in order to compete at that level, I would have to make sacrifices that would impact the quality of my life, and frankly, I just wasn't willing to make those sacrifices. I have no regrets.
I had my first child in 2003 and of course, that changed everything. Since I went back to teaching in 2004, I've played maybe a dozen times. I do miss it, but I'd rather hang out with both of my kids (I had a son in 2006).
My daughter has shown little interest in golf, but my son has, and so we go hit balls, chip and putt, have some ice cream, and talk about everything. I suspect I will get him on the golf course this year, and then all bets are off if he likes it. I can easily see us playing 2-3 times per week.
So my love for the game never went away, it just dropped on my list of priorities. I still have fun if I shoot ninety, and once in a while I'll break eighty, but I still love it.
I suspect your enjoyment of body building will be similar. And, I suspect that with the base you have, you can maintain your level of fitness without spending hours in the gym. Of course, when mini Brick arrives, you can get your cardio in with a good quality stroller, body weight plus kid squats, kid curls and OHP's, etcetera.
You're a pretty reflective dude, so I know you'll get it figured out. I frequently get up at 3:30 AM so I can lift before school (my first class is at 7:21). That way, I can be home with my kids at 3:00 PM. Of course, I hit the rack early, but my wife is home by then and she gets to have her time with them.
I bet you can maintain on three days a week for an hour, and a few half hour LISS sessions. You'll find your groove.
It will work out, and it will all be worth it.