Some lab results would be helpful.
Research on mammals shows that it increases LH and T. If LH is pulsatile it may be safe. It is known that constant high LH does not work out well. hCG has a long half life and that seems to make T OK, but there are concerns of LH receptor degradation if levels are high. High hCG doses can can lead to a lot of E2 generation inside the testes. One can extrapolate that high LH levels might have effects similar to high hCG doses, but I have never seen any data about this ying or yang. One can thus also be concerned about high levels of LH from SERMs as well. High LH or hCG should lead to high E2 levels.
In your case, if d-aspatric acid was used for a long period of time, I would be concerned that when you stopped using it your T levels might be below baseline. So there is some potential for risk.
Any other dietary changes?