I'm not a bodybuilder either, but I started using a stim-free pre workout and like it. The only caffeine I am getting now is from a cup of coffee in the morning, so pretty minimal.
The reason I stopped using a PWO with stimulants is because I have a hell of a time relaxing at night-- otherwise I would still be using them. My personal experience has been that I can get that extra rep or two with stimulants on a set. So, it isn't a huge difference when we are talking about training, but for competition, I would most definitely use caffeine for an increased performance edge. So, for me, it is a precedent placed on sleep, rather than getting an extra rep in the gym.