The physical act of using a word isn't inherently bad. Whether or not someone may view you negatively for using certain words that they find bad is what matters most. You don't necessarily have to care, you're a person that can say what they want, but if people are calling you such a label frequently because you use that term, and you don't like the label, maybe you should consider what image you project to others?