I think you should do something about this and at the very least tell the person how you feel. It's a better alternative than keeping it in for who knows how long and wonder about it down the road.