Age gap in relationship?

I have been seeing a man for coming onto 4 months now, and we are really happy. The only thing is, I am 19 and he is 34. We don't mind at all, we know a lot of the same people and all my friends were a bit shocked but pleased for me. His friends however have been distinctly unwelcoming towards me, asking my age repeatedly and guessing younger every time, ignoring me at social gatherings, patronising me, etc. He has even been called a pedophile by a friend of a friend, in our local. I am hurt because these are all people I was aquainted with before and thought I got on well with- but now suddenly I am being singled out? Is there any way to get them to leave us alone, or will we just have to wait until they grow bored of it?