a 4lb fluctuation really doesn't mean shit. It's average to fluctuate + or - 5 lbs just from water weight. Sounds like you need to pack on some muscle, though. I would strongly suggest stocking up on whole food, i.e. chicken, tuna, Tilapia, turkey (for protein), Oatmeal, greek yogurt, sweet potatoes, banana's, other fruit (carbs), almonds, olive oil, walnuts (fats). Toss some whole milk in there too. Go for at least 1g protein per pound of body weight, and you'll be on the right track.
I don't think there is anything wrong with your metabolism, you just haven't been giving your body the right nutrients. I would feel like shit too if I ate "whatever" and worked 60-70 hours a week =p.