Yes, the smallest MU's do have an incredible resistance to fatigue. But they are also incapable of lifting anything other than an extremely light resistance (like say a pencil). The intermediate fibers on the other hand have much more force potential and can in fact lift decent amounts of resistance. They do have more endurance than the largest fibers, but would still fatigue during the later reps of a moderate rep set.
Also, it's not like all small fibers, all intermediate fibers, and all large fibers are identical in terms of size, endurance, or force capacity. You're going to get a range in each one of those categories. Some FT fibers are going to have just slightly more force capacity and just slightly less fatigue resistance than some intermediate fibers.
In other words while you may not recruit your largest FT fibers during the first reps of a moderate load, moderate rep set to failure, that doesn't mean that you're not going to recruit any FT fibers. And, although your intermediate fibers have more fatigue resistance than your FT fibers, that doesn't mean that you won't fatigue any of them during the duration of the set.
The more fibers you fatigue, the lower your capacity to produce force. Even though you may be calling into play fibers that are on an individual basis capable of producing more force than those which you have fatigued, they still may not be capable of producing more force than the combined efforts of those that have already fatigued.
One other thing to consider is that, during the last couple reps in a set to failure, effort is greatly increased, as is the attempt to move the bar with maximal velocity. Vroom actually brought this point to my attention in a previous thread.
In other words, what is important is not so much the load that is being lifted (although that does play a role), but rather the attempt to produce maximal force. And anyone who has ever done a set to failure will tell you that they are most certainly attempting to produce maximal force during the last couple reps.
This whole concept is pretty much the basis for CW's "maximal recruitment" methodologies. I know that speed seems to be his favorite application of this. But, what is attempting to lift with maximal speed other than one example of attempting to apply maximal force/effort against the bar?
Think about it, if attempting to produce maximal force (speed) works at the beginning of a set, or during a 1RM, then why wouldn't it work at the end of a traditional RE set? Especially since up to that point the largest fibers haven't been working, thus they haven't been fatigued.