Try and replace your calories from milk with something else.
What are your macros and cals with that?
If your worried about fat gain, which you are, you have to start tracking that shit.
If you aren't worried about fat gain and just looking to get swole up, then you don't have to track what you eat too much (but don't be foolish, find Artem's thread in the TCell, very very very good read. Lots of good info in there.)
Are you on a budget? Because ground beef, tuna and chicken are cheap, and IMO better than milk. Milk is a "fuck this shit, I'm gonna get big and strong, fuck fat gain, fuck what anyone thinks, I'm gonna bust my ass and take shit down with me" type foodstuff. If your nervous about fat gain, I would replace my milk cals with solid foods first.
Again, this is my opinion.