To each his own. I used to feel the same way you did. When I was in the military there were a few times when I went without food for a couple of days. It was horrible, but that was because we were constantly moving and weren't allowed to eat. It is much different when you make the decision to with hold food from yourself.
I also used to make sure that I ate a huge protein meal every 2-3 hours and ate big. Was I big? Yeah. Was I strong? yeah. Was I bloated, out of shape and unhealthy? Yep.
I became obsessive about eating all the time. My digestion suffered and as a result so too did my overall health.
Interestingly it wasn't until I read a book by Ori Hofmekler called the "warrior diet" that I began to put things together. The body functions somewhat as an "on" and "off" switch. The more food you put into it (relatively speaking) the less efficient it becomes at utilizing it. Much the same way that bodybuilders are able to pack on incredible amounts of muscle after their contest. The further into a catabolic state you put the body, the greater the anabolic rebound will be.
I've seen guys go into Ranger school at 180lbs, come out at 135lbs, and within a month be at 190bs and noticably more muscular than when they went in.
The body needs a break from the constant eating, training, and compulsive idea of getting bigger and stronger. At least, that is just how I feel.