Allow me to share a little personal experience.
Last fall I found myself in the beginner's yoga class at my local Gold's Gym. Looking back, it's not really all that clear why I decided to give yoga a shot, but I was there nonetheless. Upon entering the gym, I promptly scanned the free-weight area to make sure no one that I trained with would spot me following the line of dainty, spandex-clad females into the separate aerobics/spin area. It's worth noting that while I was not the only man in the class, I was certainly the only T-man.
The two other guys in the class looked as though they were making a point to try and hide the fact that they possessed a pair of testicles between their emaciated, chicken legs. Other than those two fruit cups, I found myself surrounded by generally fit, attractive women. Nice! As the session began, I came to another sudden realization... Yoga. Is. Tough.
Go ahead, laugh, but after the 45 min. "beginner's" class was over, I was sweating like a whore in church. The instructor was pretty nutty, and would often try to slip extremely left-wing political propaganda into her instruction, saying things like "Lean until you feel a nice stretch in your obliques. No war for oil...Just relax and focus on your brething..." She then had us wrap up the session with 5 minutes of meditation in the dark. I was so over this bullshit by the time we were sent on our way.
But when I got up to put away my little mat, I realized that I felt incredibly high. The sort of high like I had just smoked a huge fatty of kind herb after having sex with a perfect 10 model. I felt so good that I waited around the gym for 15 minutes before driving home...I felt so good, I literally did not think it safe to drive. Anyways, I ended up going back 2 or 3 more times, but eventually quit to pursue other interests such as squatting, deadlifting, and benching heavy pieces of iron. (That, and the fact that the guys I trained with had caught on and gave me a ration of shit.) But I still remember how good I felt after that first session of yoga. I would say to the OP to go for it. Give it a shot. If nothing else, the experience will be worth it.