I only add about a tablespoon to my protein shakes. A tablespoon only contains just over 12 grams of fat most of which come from mono and polyunsaturated fat. there shouldn't be any drawbacks to this considering 12 grams is nothing compared to some of the other foods that are undoubtedly in your diet.
the only downsides would only be side effects from the increase in testosterone due to the fat intake, i.e. possible acne.
so buy some neutrogena face wash and enjoy the muscle increases.