Move to your own private island. To be more serious, the question you asked is a good one on a grand scale, not in reference to your neighbor's weed killer, but in reference to the use of plastics and other possible enhancers of estrogenic effects in our society.
The rate of gynecomastia occurance, the last I read, was in excess of 60% of all males in this country. That means more than half of the men in this country have some type of it although most may never notice it because most people in the country are also overweight.
I will try to find some references for more info, but I am about to leave the house right now. This is a good topic, however, and hopefully it can evolve into something substantial.