If 500 people are stuck on an island deciding how to get off and take a vote that comes out 499-1, you being within the 499, would you honestly make the same argument for the 1 vote against? If you believed your survival depended on 500-0, and that 1 was holding it up, would you honestly, really be arguing the same thing?
No, I don't have the right to force a person to interact with me if they prefer not to. But when some person chooses to use my roads and all the other things I offer members of my society without paying for them, I consider that theft, as I'm sure you would too. So what are we to do, put a wall up like Donald Trump wants to between your society that want's to be 100% self sufficient and mine? Tell me how you can even realistically ensure people who do not contribute to the collective society don't benefit from it without some kind of physical barrier between us?
No, I'd say my arguments stem more from pragmatism and reality. How do you plan on arranging a society in the US where those that recognize the rule of law and those that don't can possibly coexist peacefully?
Do you have the right to steal? Somebody owned that land before you did, right?