Storing IP addresses in logs means that you are now responsible for them, yes. Drop them out of your logs, and you're perfectly fine.