Go for it.
A lot of people think that cause I know a lot about the sport I musta been watching it for years, but I didn't even watch my first fight until 2005 when I was 21 (Poor Arturo getting slaughterd by Money) and I didn't watch another one until 2007 (Floyd and DLH).
Although I do have to say I always enjoyed the Rocky movies and that's what got me lifting in the first place and made me like boxing.
I've been training off and on for the past four years, but only in the last five months have I gotten truly serious about boxing and started training four days a week, running, etc.
It's always worth it. Even if you never fight, you begin to learn a sport so intricate that everyone thinks they know- but they really don't know shit about. And it will make you athletic as hell, at anything else.
Tony Thompson didn't get into a boxing gym as I recall until he was 27, and he fought for the heavyweight title in the last HW fight I forced myself to watch.
Several others, like Sergio Martinez, didn't start until later in life either- so if you want to be a pro, it's plenty possible.