Okay I'll explain, LH and FSH are hormones secreted by the pituitary gland to signal the testes to make testosterone and other androgens. When these hormones are not being produced, testes don't get the signal and they don't produce T. There are two types of hypogonadism, one where these signalling hormones are not made by the brain, and other where the signalling hormones are made but the testes don't respond to these hormones.
Clomid can be used when the brain is not making LH and FSH as clomid tricks the pituitary to make them. If ones LH and FSH were okay in the first place, that means testes were not fully receptive to the signal. Using clomid then, increases LH and FSH to where they're not sustainable and will drop back down after discontinuation of the drug.
Clomid is half estrogen. 25mg of the 50mg is an estrogen in itself, it does not "increase" estrogen it is one itself. The increased LH and T increases E2 in the blood. Now when T is not being produced efficiently, and we administer an estrogen through a drug, SHBG increases in some men. This is a protein secreted by the liver and testes in response to estrogens and external thyroid hormones. Higher shbg "artificially" inflates TT measurement. Testosterone bound to shbg carries no biological significance. FT is what we are looking to increase. Shbg is also increased due to testes inability to produce adequate T in response to LH and estrogen load. T decreases shbg and estrogens increase it.
With your high total T but modest FT, it is my suspicion that clomid has soared your shbg very high and when you get off clomid, shbg will remain elevated and your LH FSH will drop from the aforementioned unsustainable position. This will lead to some drop in total testosterone and a bigger drop in free testosterone. Essentially, even with high TT numbers, it wouldn't be biologically active and we face hypogonadism symptoms. Now a doctor doesn't know your history and will wave his hand when you ask for treatment at a high total T.
This is my theory of your problems! You're free to disregard. I'm trying to help. If you want to confirm or rule out my theory, you can get blood tests for LH fsh, shbg among others to see if that's a problem.