A lot of people have low level lactose intolerance, so consuming dairy can cause a cortisol spike, among other things. For a cut i.e. an extreme diet to get in contest shape, this can cause problems. For normal weight loss, dairy probably won't cause a problem unless you're very sensitive to lactose. However, if you stall out on your progress, you may want to remove dairy and see if it helps.