No one's questioning his use of drugs as an athlete. It's his use of drugs as a public figure.
If he wants to smoke some pot at a party, that's fine. Just don't get caught green-handed during your fifteen minutes. Phelps has always struck me as a sort of Forrest Gump. He ain't that smart, but boy can he run!