Your doc is misinformed, and you are correct. Its use as an AI stems from an old study where it was shown to have AI activity in vitro (lab test.) However, in vivo (your body) it's been repeatedly demonstrated to have zero effect on estrogen in humans or animals. It just plain doesn't work for the stated purpose.
50 is really high. My E2 was in the 40's and I insisted on an AI. You need to do the same! Starting dosage is .25mg every other day. Retest after 6 weeks. You want E2 in the 20's.