The assumption here is that you will only add muscle in an effort to progress strength, which is ignoring the process of hypertrophy.
Assuming whatever weight you are lifting is stimulating enough of your type II motor unit pool, you will add size.
If it takes time to re-learn the movement, coordinate other muscles, to lift your PR again, this may have nothing to do with stimulating tissue growth.
If, for some crazy reason, you just can't recruit much of your motor unit pool at all for a given movement, and only end up lifting a very light weight (for that exercise) so not recruiting the type II units, then hypertrophy probably wont' be stimulated.
An untrained person is incredibly inefficient at recruiting their entire motor unit pool, hence the whole neural adaptation thing with training and significant delays till they can stimulate hypertrophy.
A trained (well skilled) person is often VERY efficient at performing the movement, which at most sub-maximal levels means they also will not be recruiting the maximal amount of motor units available. This dictates the need for more trained people to lift very heavy, for as much volume as possible.
Someone bouncing around from program to program is probably not developing sufficient skill to maximize motor unit recruitment for any given task, limiting both their ability to develop strength and hypertrophy.
Someone who sticks to a program for a significant period of time, and doesn't just drop an exercise 'cold turkey' but gradually introduces the new exercise movements into their training before progressing entirely, is not so likely to experience problems.