No need to apologise, at all. It's not you that is making the situation and I honestly wouldn't bother apologising for those that are.

Problem is; it is only a minority (IMO anyway) and a fairly small one. But they tend to be vocal/obvious, and it is one of those things that doesn't take much to make you feel uncomfortable.

I don't think there is anywhere much that I like better than Manhattan! :)