I think that it has to do with the culture you live in. In the area I lived in the states, it was considered absolutely taboo to comment on someone's weight to their face. Whether it be the person was under weight or over weight!
Usually, you'll find that the person saying the rude comments to you has some sort of insecurity and are just lashing out to you.
I don't think a person has any right to say those things to you - even a spouse. I can happily say that my spouse has never made such a rude comment and would lose respect for anyone else who would say something to that nature to me. Yes, I've had people say comments to me and yes, I don't regard them highly anymore. I'm hard core!
But going back to the culture, only other thing you can sort of give slack with is a language barrier. Maybe they meant something nice, but it came out all wacky! :)