Dude, if you think of yourself as a pussy, you’ll be a pussy.
Get in his face, and tell him to back the fuck off and never fucking touch/bother you again.
If option 1 doesn’t work, swing on him.
It’s all in the mind, brother. In college, one of my great friends was this diminutive little Indonesian dude. I mean dude was short, thin, and some what nerdy. Except he possessed the heart of a mother fucking LION. When this guy lost his shit, he fucking lost his SHIT. It was a site to be seen and it literally struck fear into the heart of people around him.
Anyway, he was on a prestigious business summer internship in London, with a bunch of other students from other schools. A D1 college wrestler, got drunk and stole his roommates vodka. My buddy went up to him and tried to get the vodka back. When that didn’t work, he fought him. A short, little Indonesian fuck, fought a college wrestler. My boy didn’t give a FUCK. He didn’t get his ass beat (although he did get sucker fish-hooked by the guy later on) and he didn’t get kicked out or expelled from this internship abroad program. This guy gained so much respect from everyone around him, and the story became stuff of folklore around the business school.
This motherfucker didn’t have the physical goods to back shit up, but mentally the guy was so powerful. Never took shit from anybody, not men, not women and not even prick professors.