This is not true, atleast in my case. I had to attempt my HPTA restart twice to get me going. The first time I tried I used Nolvadex 10mg ED, and response was very good. Nearly all symptoms were gone and I was very happy. A few months later I started to feel a little slump, and got myself retested, and my levels came out to be 415. I investigated all of the contributing factors again, made some lifestyle changes using Igs's fatloss advice and CT's recovery tips and again attempted a SERM restart, resultingly I'm good to go to date. No symptoms at all, feel energetic, good performance in gym as well as bed and I'm in best shape of my life today.
The elemental thing in my case was that my body responded very well to SERM, a low dose of 10mg for 5 weeks took my levels from 202 to 753 on 241-827 range. I also tried to address the underlying causes leading to my low T like Vit D deficiency, 5alpha reductase inhibitors usage and dietary factors.
This thread contains the whole story of my T recovery