In my case, I'm eating a pound of ground beef, half a pound of pork, canned chicken, canned tuna, salad with hardboiled eggs and 4 protein shakes a day just to gain anything at all. Any less than that and I can't sleep - toss & turn endlessly. Don't know if that's excessive, but that's what I need.
You mentioned yogurt, milk and cheese. You're not sensitive to dairy, are you?
I may not be the best person to advise you on losing weight, since that seems much easier for me than other guys.