See, I'm conflicted about this. I'll be the first to say that fighting is a terrible idea and always ends badly, but I've also been in my share.
Your stuff about the glasses is an excuse to yourself. You could've taken them off had you really wanted to bang. Don't try and fool yourself.
Basically, once someone lays a hand on me, it's gonna be rough. Should you have gone outside to fight him? Maybe, maybe not. Honestly depends how old you are- if you're under 18 then I would have gone outside.
But you should have stood up for yourself in one way or another... even a threat would have been better than nothing, like saying, "I'm not going outside, but if you come back here and do this again, I will split your fucking head open and leave you on the curb."
You know, give him something to think about.