It is all in how you word it, so to speak.
Augusta National, for instance, is a private golf club where they hold The Masters every year. It wasn't long ago that they did not let minorities or females in.
Like twostroke says, they caved to public pressure, but were not subject to the civil rights act. Why? Because they were not open to the public. In other words, they retained a much stronger right to exclusion than Walmart, for example.
Walmart opens it's doors wide and does not place a limit upon entry. Augusta Nationals said to come in you have to join a private association.
Because of this, a place like Sams Club or Costco could stand a better chance of being able to exclude a specific group.
In your private residence and properties you have absolute right of exclusion. "Whites Only" is ok there, if that is what you want. But if you are "open to the public" you cannot exclude anyone but known thieves.