It depends a bit on how your program is laid out, especially the volume and intensity. But generally, if the goal is to build muscle, then yep, completely resting a full three minutes won't be as effective as other methods.
To maximize tension without allowing "too much" inter-set recovery, a rule of thumb is resting a minute or so between moderate rep sets (8-12ish reps per set). Resting longer than that might allow for better performance, but the total work done in a given time period would be reduced and would impact progress. And again, this is for moderate rep work. Once you start talking about heavier/lower rep stuff, longer rest periods can certainly improve performance.
In any case, you can always find "stuff" to do between sets that makes the most of rest periods. Active recovery methods like mobility work or training a non-related bodypart (basically a traditional superset) can be a way to sneak in extra training without interfering with the main work.