I'm actually not a fast eater. I have had to train myself to eat faster because I can get more food down. I just don't like extensive conversation when I am eating, mostly because, unlike most people, I am not just eating for pleasure.
I know I need to get more food down to grow so someone holding a conversation taken me away from that goal bothers me. Of course, if I was on a date or something, I would have a different mindset.
However, if it is lunch at work, I would rather be watching tv or something else so I don't have to give a lecture in between bites.