Like the two gentlemen above hinted, I'm not sure I can tell you what YOUR rest periods should look like, but I can tell you how I've approached mine over the years.
As a beginning lifter I ran a 5x5-ish type of fullbody program. I took long rests usually. I never measured it, but I'd put 3-5 minutes in between most working sets. This got me plenty stronger.
I presently run 5/3/1. I rest only long enough to load the plates for the next set until I get to my top set. I take a 2-3 minute breather before that, but I'm usually still breathing heavy when I start. This is also getting me stronger. Assistance work I keep rest pretty short and incorporate a fair bit of supersets with no rest at all.
There's really no long-term drawback that I can measure from condensing my work. I'm still a big tubby dude, but I'm in pretty good shape for a big tubby dude. My work capacity is insanely higher than it was 3 years ago.
This comes in handy during real life chores. I'm a lot better in bed now too. The only thing I feel like I lose is getting the best possible numbers on my top set for that day, but I don't think that even matters that much in the long term. I'm still getting stronger over the long term with dramatically shortened rest periods. Who cares if I only get 6 reps instead of 7 if I'm still getting stronger?
That said, the competitive powerlifters I train with take some LONG rest periods. I will bust out entire workouts in the time it takes these guys to rotate through a few sets of squats. A few of these guys are elite-level, so they're obviously taking those long rests for some kind of reason.
I hope this helps!