I think you can state that about pretty much any muscle group, being genetically predetermined as far as potential. Certainly calves are not fun to train, don't give up results easily, nor give you this great pump that lets your strut around the gym feeling like a badass. Calf work is uncomfortable, takes a great deal of effort, shows progress seemingly slower than any other muscle group, and most of the year the calves get hidden by full length pants.
Like any skill, some people may start further along the genetic spectrum of development naturally, while others appear to start at a disadvantage. That doesn't always correlate with what is ultimately possible though. My calves were not great for a long time, but when I finally stumbled upon what worked, coupled with my smaller joints, suddenly I was the guy with the calves at my gym.