Dilligence and discipline is key.
They only true drawback I experienced was a little lower energy levels as well as burping up fish oil all day long. If you don't like the lingering vapor of fish in your mouth, be warned.
As for the Polar Bear plunge:
I did one in Northern Illinois, at Chain O' Lakes, in Antioch.
It was accidental, though.
A buddy and I were ice fishing for stripers but also had tip-ups placed in closer to shore for pike, bass, and musky. I got up from my bucket to go check on them and realized I needed to take a piss. Well, I moved in toward shore, un-did all of the arctic garb necessary to gain access to my wang, and subsequently fell through the ice up to my shoulders as my boots sank into the muck. My core stayed relatively warm, I remember, as crunched and crashed toward shore and climbed onto more solid ice, but my wang was exposed the entire time, and the near freezing water poured in through the ol' barn door.
Didn't really get cold, until I was all the way out and began the trek back to my car and the adrenaline wore off. Uncontrolable shivering really hampers one's ability to do anything that requires motor skill, like getting out of freezing wet clothes.
Envigorating? Awesome? Hilarious? Well, hilarious maybe. However I didn't think so as I sat in my car wrapped in a wack ass itchy wool blanket with the heater maxed out, fearing a hypothermic Illini death, and or permanent loss of penile function while waiting for my buddy to gather up the gear and get me home.
Water at that temp stings, like a trillion little tattoo needles puncturing your body.
Oh, and I think what you're doing for the Special Olympics is commendable and awesome.