Yes, they can affect you...especially over time.
Here's a quote from Environment Health Perspective: "Recently it has been found that a variety of exogenous modern chemical compounds have hormone-like effects on both humans and wildlife. Their interference with the synthesis, secretion, transport, binding, action, or elimination of natural hormones in the body can change the homeostasis, reproduction, development, and/or behavior the same as endogenous produced hormones."
In simple terms, over time it can affect your own body's hormone production. Also, rBGH that is often found in milk products have been shown to cause cancer. That's why you'll see the healthier proteins on the market say "rBGH free."
I was thinking about the "spill-over" effect today driving home from the gym. We all can get away with ingesting somewhat unhealthy things through the course of our lives...like basic meats (with hormones in them) and foods with artificial sweeteners, colors, flavorings, etc. But then there comes a time when the cup becomes full and you get spill-over. When this happens things begin to ache, you get more inflammation in the body, organs, adrenals, and hormones begin to break down. That's when disease happens. For some people it may be in their 30's, 40's or 50's...but the truth is that it didn't happen overnight.
I totally understand about cost! You've got to pick and choose your battles. Maybe stick with cage free chickens and eggs, eat non-organic veggies that aren't on the "dirty dozen" list and choose only organic for the important ones. Stuff like that.
It's all about finding that balance and what works for you and your budget.