I have no idea how you are getting into the gym and doing legs and shoulders both within 50 minutes, assuming your volume isn't really low or your rest times are short.
Would taking the rest day out of the center stop growth? It is possible and no one knows your bodys recovery ability or work capacity. You will have to see for yourself. Probably not, assuming you are eating more then you burn. The human body is a pretty amazing thing, it can handle way more then people seem to think.
"My diet is great consuming 3200 calories". Well that may or not be great. Just because you get 3200 calories doesn't mean anything if you burn 4000 in a day and you are trying to gain.
Rest is important, in the sense you need some to gain. How much you need is individual, which is why some people like 1x a week body part splits and others like more frequency. I squatted 3x a week and deadlifted 1-2x a week for months in a row and made great gains. I was sore all the time, but there's a big difference between soreness and under-recovered.
Is 12 sets too much? Depends on you as an individual, also depends if you are doing straight sets or ramped. 12 sets with only 4 of those being top weight is a lot different then 12 sets done with top weight. I don't think there is anything wrong with 12 though.