Building muscle takes years, dropping from 20% to 12% takes a couple of months.
My advice to you: figure out what goal you want more. And find a BF level YOU are comfortable with, that is in line with your goal. And go out and fucking get it.
20% doesn't bother me, other people would freak out and jump off a cliff, disgusted at their fatness. It is going to be individual. Also, a guy 6' 280lbs @ 20% is going to look a hell of a lot different than a guy 6' 180lbs @ 20%.
Point is, you have to be comfortable with yourself, or at least willing to be uncomfortable for long enough to reach your goal, assuming you take this serious.