Your doc is not very comfortable with hrt. Reading the stickie guides and excellent explanations will show you what is missing. If he/she were comfortable then more hormone levels would of been requested.
Your asking for thoughts so here goes. The whole idea of hrt for guys is a mystery. There is not a clear cut number that establishes abnormal vs normal. Reading medical articles on this the authors readily admit that the tests themselves are imperfect, some advocate the inclusion of SBHG into a calculation to further establish whether a given level is really low establishing the diagnosis of hypogonadism(www.issam.ch is website that gives the calculation tool). I mention this because there is a lot of subtlety that many doctors are not comfortable with and when faced with such things doctors(and like so many others) look for the easy path. "It's normal you are fine". Endocrinologists, Internists, Urologists etc. you can read the forums with many different shared experiences on the negative side.
Let's say you do find a doc whom you can work with, open minded, a person whom you can share legit articles for reputable journals that support your case. Are you ready for the baggage that goes with testosterone replacement? It's not cheap, you need a reliable source for the prescriptions whether you choose local pharmacy or whether you are daring enough to look for an online source.
Now to back up a little, the symptoms you describe could be due to many different things some physical, some emotional, some diet/lifestyle. Your doctor will probably want to explore this stuff well before jumping into something he/she does not understand or even want to undertake.
You may be absolutely convinced test is the problem. Do your self a favor and read the excellent stickies, spend some time reading the many posts before you stick a needle in your ass.