From a scientific standpoint, alcohol inhibits fat metabolism by decreasing lipolytic enzyme function.
The physiological effect would be twofold:
1. Decreased fat breakdown, leading to increased levels of bodyfat
2. Increased protein breakdown, due to decreased fat metabolism while not changing total metabolic rate.
In terms of the effect alcohol has on your physique, I would imagine it would be dependent upon the amount you drink and the frequency with which you drink. Personally, I find that having a few drinks once a week has had an imperceptible effect on my physique. Like any other decision, the decision to drink should probably just be based on perceived costs to benefits - it may be good for someone who wants to train and have a nice physique, but at the same time live a somewhat normal life. It may be bad for someone who wants to compete in bodybuilding.
In my opinion, unless you are so focused on your physique goals that you aren't willing to compromise them for social, recreational, or personal means, then don't worry too much about infrequently consuming moderate amounts of alcohol.