Ultimately do what works for you.
Pay attention to your body. (which you are)
Everyone's body is different and we cannot all eat the same thing and get the same result.
A calorie is not a calorie.
If your waist has gotten bigger and your are gaining weight, and the only thing that has changed is you are now eating 5 pieces of fruit a day, that should tell you something.