A Personal Relationship Manager feels like something that would be used against you in a jury trial. Not any specific information entered into it, just the fact alone that you were using it.
I think it's the "taking it way too far"-aspect of it, compared to a regular calendar[0], that makes it so creepy. It's the fedora and trenchcoat of calendars.
[0]Which my casual duckduckgoing indicates a filofax is.[1]
[1]On re-reading the above, I fear it might have sounded a little condescending, which was not my intention. I've just never heard of filofax before. In Denmark we just call those calendars or occasionally "Mayland", after a popular brand that makes those.</tangent>
I guess I don't understand what about it is taking it too far. With a filofax, you're taking all kinds of information down about people. Calendaring is one aspect of it, but it's really an analog personal information manager. People keep yearly archives of the information. So is it that a digital PIM automates some of this process that you object to?