They would stay in you essentially forever, unless cut out. Whether there would be adverse effect from it or not, I cannot say. It might even vary depending on the material of the stopper.
If it's already happened, well, monitor the area from now on. If there will be trouble from it, there might be a long onset before any sign appears.
If it hasn't happened and you're just wondering if you need to throw out a vial that has such particles in it, either filter out with a submicron syringe filter, or throw out. If it's merely a syringe with it, throwing out would be my advice.