It is best to let it sit out for 24 hours to dissipate most of the chlorine. Not sure if the amount in tap water will actually kill the soils beneficial bacteria, but better safe than sorry.
As to the soil, you are fine as long as you don't bury the plant

.
Edit: Raven beat me to it.