Its all gonna depend on your goals.
What are you trying to do? Are you lactose intolerant? Are you cutting? Do you have any personal, negative reactions to lactose?
Its not gonna be a law to avoid or not to avoid lactose. You are going to have to base that decision based on your goals and your own experiences with foods which contain lactose.