I've had the same instances - Never used the N word but once I called a good buddy of mine 'boy' and after a short yelling match, I realized that is just as offensive.
On the flip side, the same guy consistently calls me white devil, honky, red neck, etc etc - I hate to say it but I find it...