Your current E2 values would be of interest. Many have such problems with elevated E2 and anastrozole can then have great benefits. As you are using Arimidex now, that component is weak or non existent.
Are you getting enough or are you sexually frustrated?
Most are going to agree that ativan should be your last resort.
I and others have worked with guys who are a mess after using this class of drugs for years.
There are many other possible contributors to your current state. Some would be health related, some dietary/supplements, some could be evident in comprehensive lab work. Other drugs, Rx or OTC could be causative or contributory.
How many years ago did the mood-family issues start? If these are well worn pathways in your brain, it make take time and effort to change that, even if other causes are identifiable and managed. I think that you may be frustrated by some aspects of your family life. Understanding what such irritations/resentments are would be very helpful. Next time you feel a negative emotion coming on, do nothing, keep quiet and ask yourself "why am I feeling this way, what am I feeling".