Your body grows while at rest. It also does NOT grow linearly.
Maybe you could tell me what you think happens over the long term if you are constantly feeding your body less if it is ready to grow more?
If my body is ready for a growth spurt and is set to pack on 3lbs of muscle, yet I am feeding it less because I falsely believe I can predict exactly how many calories my body needs at all times, over time you will see less progress.
I mean, honestly...like some of us bulked up for no reason?