We indeed did discuss this to death a couple months ago.
Biologically speaking growing muscle is a much more resource intensive process not even counting the actual external work. It isn't even close.
Barring actual pathology, losing fat is relatively easy for anybody who really wants to. Getting VERY lean is a different story and is more difficult and tricky.
By any accepted scientific standard, again assuming no pathological conditions, synthesizing lean mass is much more difficult than burning off adipose stores.
How this presents itself subjectively to the individual can be as varied as the individuals. In other words the fact that some people find it personally more difficult to lose fat, which is much rarer in reality than many believe, has nothing to do with the actual physiology. Most people who will declare how much easier it is to build muscle have a very miniaturized view of what building muscle means.