If this is the case and you truly are feeling instability at the knee after rolling, you are probably lacking some proper all around strength and stability at the knee joint. The resulting pathology (whatever it may be) in the knee that is leading to the instability is causing the quad to become stiff distally as a protective mechanism. When you foam roll the distal quad, you end up relaxing the muscle and cause it to become less stiff, which results in the protective mechanism to no longer be available.
Foam rolling is great at improving the soft tissue quality of the muscles being worked on. You want your muscles to have proper length, strength, and tissue quality. So to say to not foam roll due to you feeling an "instability" in your knee isn't the best answer IMO. You should try and figure out what the underlying cause of this instability is and address that. In the mean time, I don't think you NEED to stop foam rolling, but if it worries you that much, it isn't going to be that big of a deal if you do stop.
I would look at you hip musculature, along with ankle mobility to see if any of that is causing issues at the knee. It may benefit you to seek out an evaluation by somebody qualified in your area to determine what is causing the instability.
Also, when you say "knee instability", do you mean at the patella (knee cap) or at the knee joint itself?