Man, if I were you, I would not only change my number, but also my phone. To answer your question plainly, yes, you should be worried. It might be nothing, but if it is something you'd better act quickly. And a piece of advice: The newer telephone models are much easier to tap than the older...