Hm, I won't be able to help too much, but here's my experience:
Before I started properly training, T-Nation style, I was skinny and had a low bodyfat %, but still had "love handles" (i.e. little folds of fat that hung over my pants). I also had a visible six pack. I was about 5'8", 145 to 150 lbs or so.
Now I am slightly less skinny, and just have obliques, not love handles. I honestly don't remember when the love handles vanished, since I wasn't really paying attention to them. My diet hasn't changed much, but I started doing heavy deadlifts, squats (front, back, and overhead), and Olympic lifts (power clean and power snatch) regularly.
I also took up some basic gymnastic exercises (front and back lever and planche practice a la Coach Sommer's article at dragondoor). I also started eating small meals at multiple times through the day.
So which one thing did it? I don't know, but any one of them is worth a try.
One thing I did notice during my love-handle era was that whenever I played a sport that involved a lot of twisting and running (Ultimate Frisbee, for me), the love handles would magically disappear/become less flabby for a while after the game. Are you sure that it's fat, and not just flabby obliques? (Is that even possible?)