Anyone clueless enough to claim he is fat just because he is in the off season is too dumb for me to care about anyway as far as their opinion. Why would I train to please people who don't even understand what I am doing and why?
Bodybuilding isn't about fitting in. It never has been. if you want to simply fit in, go drop 20lbs, blame it on a fad diet and call it a day.