I grew up on a dairy farm, my whole family drinks milk right out of the tank and I still do when I'm home. My dad's dad consistently drank the stuff, still eats ice cream out of the bucket and is still lucid and mobile at 82 with zero heart problems.
I understand the fat helps the protein absorb and is in an ideal ratio in whole cow's milk for pretty much all mammals.(Of course, I may be predisposed to paying attention to biased research given my background) It shouldn't be hard with a consistent schedule with a bout of cardio or two a week to offset the cholesterol. If you have a history of heart problems in the family, of course, all bets are off.