I think its definately inactivity up to a certain point. Professional athletes can perform at a peak into their late 30's, maybe later. I think someone with a consistent activity level and less wear and tear than the pro's go through can stay at a relatively consistent level into their 40's fairly easily.
I think activity is a huge key to recovery, which is the main complaint people always seem to give about aging. In college, I ran track. 6 days out of every week consisted of biking miles around campus, a running workout, and a weighttraining workout. That was basically the equivalent of three of my current workouts all rolled into one day. I wouldn't stay sore for more than a day after a hard workout because I was doing something the very next day.
Pretty much immediately after college, I noticed that if I didn't do something active the day after a hard workout, I would be sore for 2-3 days. I don't think I aged significantly between my graduation age of 21 and my first year of the working world at 22.
I also noticed that if I play a sport once a week, like soccer or basketball, my legs feel like crap for a while after each game. But if I just upped that number to playing twice a week, I felt much better after each game. Just the more regular conditioning garnered from playing twice a week, in addition to my workouts, dramatically sped up my recovery.
That recovery phenomenom has pretty much stayed the same for me for my almost-10 years after college. Sure, some of you may be thinking I'm still young at 30, but I'm surrounded by all my friends who started slowing down and complaining about their aging, aching muscles and joints years ago. I always point out to them "hey, remember how you used to play basketball 4 times a week in college? And notice now how you play once every two weeks? Think that may have something to do with it?" This is all why I belive that the main problem is inactivity. I definately think that after a certain time, your response time, fast twitch musculature and connective tissue pliability will inevitably decrease, but I think that point can be well beyond where most people start experiencing it due to their inactivity.
On a related note, if you ever want to see how our predominantly lazy lifestyle (drive to work, sit at a desk, drive home, lay on the couch, eat, rest, repeat) affects our health, go live in a city in Europe or the Northeast and give up your car. It's amazing how much just simply walking to everything everyday can make a difference in your fitness.