Personally I feel that DHT gets the blame for these sort of issues, while estrogen is the main culprit.
Teenagers and young men are chock full of DHT, yet rarely have prostate and libido issues, and usually have a full head of hair. It is only when they get older (and more estrogen dominant) that issues like BPH and hair loss start to manifest themselves. Do you really look at an overweight, unathletic, bald guy and think "man that guy looks like he has way too much testosterone (DHT)"?
Finasteride can absolutely kill libido and cause issues down the road, some of which seem irreversible without extreme medical intervention. A quick google search will cue you in to all the problems have with it and other 5AR inhibitors.
I would tread cautiously with it. Go get a bloodtest for DHT and Estradiol (E2) and treat accordingly.
In the interim, I would definitely make use of the on-hand adex.