I think it's a good form of exercise as long as you don't do it for 10 hours at a time. My friend's dad has the distance record on the Appalachian Trail, and my friend dropped 20 pounds hiking that way - minimal rations (junk food at that), 10 hour/20-30 mile days, and so on.
I think if you do day hikes with plenty of food and for fun maybe even work your way up to heavier packs it could be a good way to stay lean. I think the key here (if you are concentrating more on adding mass) is to try to limit the hikes to under 4 hours and maybe do them once or twice a week.
It would be ideal to do heavy pack hikes for 2 hours once or twice a week, but you can decide for yourself based on your respective interest in hiking and mass-building.
I've seen buff looking hikers out on the trail, and caveman says it's a positive thing in his lineup of physical activity, so don't worry too much in any case.