A couple moments that come to mind are:
-the first time I boxed full contact with a good boxer and realized that all that Traditional Martial Arts and most of the JKD stuff that is out there wasn't worth crap (there is some legit TMA and JKD stuff, but most is not IME). If you want effective punching skills, boxing is your surest bet (again, there are some others out there, but boxing is the most consistent and easily accessible).
-my first belt test, which completely changed my perspective of what it meant to be "in shape". Consequently, this same realization happened with each subsequent belt up to black.
-the moment I was awarded my black belt (after a 24 hour long test) and realized that I indeed could do anything I put my mind to and that I indeed had developed an indomitable fighting spirit, but that I could not rest on my laurels and still had much development, learning, and growth ahead of me as a martial artist and person
Honestly, I'm lucky enough to be training with world class instructors on a regular basis, so I'm having "ah ha!" moments fairly regularly (even though I have come a long, long way from where I started and have already learned a fair amount), so these are just a few note able ones.