I think dairy consumption, in regards to muscle building, needs to be treated in the same manner as any supplement. Start at a small to average dose (such as 2-3 cups of milk a day) and increase or decrease consumption based on results.
The average person will respond reasonably well to dairy. I recently introduced 2% milk into my diet - 3 cups a day - for two weeks for the same reason as you (I’ve pretty much avoided dairy products until then, for no specific reason). Results? I’m guessing that I’m a bit lactose intolerant. I had gas everyday. My body fat shot up 2%, even though my calories were calculated at the same (maintenance) level. 2% in 2 weeks!
As you can guess, I’ve gone back to a dairyless lifestyle (except for that once a week pizza).
Definitely try it if you haven’t had a problem with dairy in the past.