My IP resolves to a location ~20 miles away. I don't see why having a Skype contact and knowing a 20 mile radius where they live is anything to worry about?
Most residential internet connections don't have any sort of DDOS protection, so privacy issues aside, at the very least you are open to a simple denial-of-service attack. This was a huge problem for the popular progamer "Destiny" in the Starcraft 2 community.