I assume your doctor (and the others described in this thread) are endocrinologists?
Most endocrinologists are focusing on three things: diabetes, keeping bone mass, and fertility. They really are not concerned with such things as sex drive and bodybuilding. If you look at the guy responsible for studying steroid hormones here at Stanford you can get a pretty good idea of that:
I know her -- oops, sorry, HIM -- and he's a nice guy, but we definitely do NOT share many interests.
You might leverage their obsession with fertility by getting a sperm count and making sure it's low by the time you "produce the sample". There are several techniques for that, including alcohol and Speedos, but the side effect is that they lower T too, of course.
On a related note, my wife also has some pretty fun stories about ObGyns. Apparently most of them can't believe she's had kids.
I think most doctors don't actually realize the benefits of a healthy diet, a good sex life and exercise. Maybe they should be trying it out too!
Most people somehow expect doctors to be more competent than the general population, but fail to realize that doctors are not an exception the rule that 90% of people suck at their jobs. Always have and always will.