[tyingq]

I do something similar with Perl, since I know the syntax a bit better. It allows me to (from memory) scrub out non-unique things like timestamps.

So, without the scrubbing:

tail -f somefile | perl -ne '!$SEEN{$_}++ and print'

Scrubbing off leading timestamps:

tail -f somefile | perl -ne 's/^[0-9:]//;!$SEEN{$_}++ and print'