I think your decision to wear a belt should be based on what you plan to do in each workout. If you are working up to a max attempt then I say you should use the belt for protecting the back. In your warm up sets you don’t necessarily need the belt, unless you are someone who has had previous back problems.
If you are doing light weight, higher reps on the DL, then I would say go without the belt. I only wear a belt when I am squatting or DLing heavy and mostly during my max attempts. Most warm-up sets I do not use the belt.
As for how much does it take away from your development, I would say very little unless you are wearing it ALL the time. I think of belts as more of protection and support when doing heavy weights more than anything.