Do they matter, yes. Do they matter to the extent that you should spend every free second thinking about it and posting selfies on a website filled with dudes who pump themselves with Test - no.
You're putting more value on something than women do.
True story: When I was 14 I was super insecure about how I looked, because of a comment someone made. (something to the extent of, "Man the (my last name's) are really attractive people.. except for chris, what happened to him"). Any ways, I resolved to the fact that I couldn't control my face, but I can control my body, intelligence, confidence and ambition. Since then, I've had several supermodel hot girlfriends (discovered that wasn't for me and married girl next door), and the fat fuck who made that comment about me is.. well, a fat fuck.
Staaahhhhp worrying about things you can't control, fix the things you can. Someone, somewhere once said "If you have the resources to solve a problem, it isn't a problem". You got two legs, two arms, go fucking do something of value.