An insult is an insult. Constructive criticism is constructive, and you should receive it with polite receptiveness. But there is a lot in between.
How you will know which is which will depend on all the nonverbal communication going on that many of us on this site have trouble with. But the "I don't want to be rude" comment may be an additional cue that you should lean toward the constructive criticism part of the spectrum.
That's my answer to the question, but I feel compelled to write a little more.
Some clues that it was an insult or at least at your expense:
- It was stated loudly so that others could clearly hear it.
- Others laugh as a result (perhaps even an inside joke).
- No discernible benefit to you or a third party.
- If you spoke this to your pet dog, he would cower?
Some clues that it was meant as constructive criticism:
- The person makes an effort to pull you aside so you two are (mostly) alone, and they speak to you in a low tone.
- If you take the person's advice, someone benefits or good results will ensue.
- If you spoke this to your pet dog, would he would wag his tail or lick you?