You can have idiopathic (no known cause) secondary hypogonadism, like many other men, like me, a relative, and several friends of mine. I believe most men who have hypogonadism have it from no known cause. Go to a competent doctor to rule out a cause, such as damage to the pituitary or something else.
45 or so is the upper range for estradiol, though many believe it should be in the 20's, which I believe is not necessary. I've had E2 in the 30s and low 40's and have felt perfectly fine. Same for other guys I know. Not all need an AI.
It's highly unlikely that you are abusing substances in such excess that you are hypogonadal.
Actually, if you have secondary hypogonadism, your HPTA is not working optimally, and clomid will hopefully make it work, which means it will be working dependent on medicine, not without.
And when you say, "get ahold of...", does this mean you are simply going to get it by whatever means or are you going to have a doctor skilled in this area prescribe it to you? If you're hypogonadal, you can't just stop using medication with no ill effects.