It needs to be safe and clear by design, not confusing to the point where systematic PA announcements are needed to clarify the fact that "seat belt sign is off" actually means "you should wear your seatbelt".
Nowadays what they announce on European and Europe-Asian flights (the ones I have any recent experience with) is that, when the seat belt sign is turned off there's always an immediate announcement which says something like "The captain has now switched off the seat belt sign, however it is recommended to use the seat belt whenever seated".
And maybe on new planes they'll change the iconography (which would probably require FAA rules to be rewritten), but let's not downplay the role of just having common sense.