Well, I play racqetball all winter so I've got a little practice at this. I wear contacts when I play so I don't have to worry about the whole goggle thing.
However, my friend doesn't wear contacts and he does wear the big type. The place we ended up buying ours was a safety supply store, they have a selection of about 200 different types of safety glasses so we got to pick what we wanted.
I've been drilled in the head a few times and my glasses have saved my butt a few times too so you'll never get me in there without them. In terms of the issues people mention:
They only fog up when they get cold so if you never take them off it's not an issue. Plus make certain you get goggles designed not to fog.
Buy big wrap-around lenses, that way the frame doesn't interfere with your peripheral vision.
I've never had a problem with the sweat but I'd rather sweat on my glasses then lose an eye.
Get the ones with a strap across the back if they fall off on you. I personally don't use the strap and they've never come off my face but I have the strap in case I want to use it.