All women have different preferences, those girls may actually think you look disgusting if you get any bigger...it's possible.
That being said, in my experience women really have no idea, and are probably just imagining that "any bigger" is Jay Cutler sized, which we know is not going to happen. I particularly remember one girl I "met" my freshman year of college, who said she liked my body but that I shouldn't get any bigger. I ran in to her again a year later, and she said "ok, I really like this, but if you got any bigger it would just be gross." Ask Waylander if he has a problem getting laid.