My understanding is, if they have space and there's nothing going on with your roots, they're not a problem. They eat decaying matter and provide aeration and beneficial waste. If you have a large infestation in a small pot and root rot or something, they can turn their attention to the dying...