It really annoys me when I'm walking down the street and some random guy (always non-Jewish) walking the other way will say "Shalom!" to me and then continue on their way. They don't mean it maliciously as far as I can tell, but it's offensive in the same way as walking up to a Native American and saying "How!" would be.
Wearing a yarmulkah doesn't make me an effing mascot, jerks.