[_paulc]

Absolutely right - Perl 4 had a clear niche (text processing, UNIX system scripting etc) and did this very well. Perl 5 added so much cruft and ugly syntax on top of this in an attempt to try to become more 'general purpose' that it lost its focus. At this point it became easier to switch to Python for 'real' programming.

There are still a lot of problems in the Perl 4 space and to be honest I wish that distributions shipped with a supported version of 4.036.

[b2gills]

It would be rather pointless to ship Perl4, as Perl5 is almost completely backwards compatible with Perl4. Most of the parts where backwards compatibility was removed weren't actually used to a large extent. Also Perl5 didn't add that much syntax, and a big reason it was ugly is that Perl4 didn't leave a lot of space for the extra features. Which is also why Perl6 breaks backwards compatibility.

There are also some features that were eventually removed from Perl5, but not really. For example setting $[ was removed (it was a pretty stupid idea), but if you try to use it Perl loads the module arybase which re-implements that feature. (removing it simplified the internals)

[duskwuff]

If you seriously think that Perl5 is less suitable for text processing than Perl4 was, I suspect you haven't actually used the language in a while... it's still perfectly good for that. Perhaps even better, since 5.8 added extensive support for Unicode (without requiring explicit conversions and breaking everything like Python 3 did).

[ptx]

From a quick skimming through the docs[0], Perl 5.8's Unicode support sounds a lot like Python 2's except that the default encoding is latin1 instead of ASCII - i.e. unlike the Python 3 way of explicitly decoding binary data into Unicode text at the point where it's read (where presumably the encoding of the data is known), it will defer the decoding until the data is used (at some completely unrelated point in the program) and decode it with some assumed globally agreed upon encoding. Since there is no globally agreed encoding (Windows even has different legacy encodings between Win32 and the command prompt!) this will appear to work as long as the data is ASCII but later, when you least expect it (and someone inputs a non-ASCII string), give UnicodeDecodeError in Python 2 or garbage data in Perl 5.8.

Ned Batchelder gave an excellent talk[1] that explains how the Python 3 approach to Unicode works. I think it makes a lot more sense once you understand it; the Python 2 way was clearly broken, and it looks like Perl 5 has the same problem but hides it better.

[0] http://search.cpan.org/~jhi/perl-5.8.0/pod/perluniintro.pod

[1] http://m.youtube.com/watch?v=sgHbC6udIqc

[b2gills]

Why are you pointing to the unicode entry doc in an ancient version of Perl?

Also according to perlunifaq the minumum version you should be using is 5.8.1 which the documents for 5.8.0 would of course not mention.

Really if you want good unicode handling you should probably use 5.16.0 or later. If you want the latest version of Unicode there are ways of changing which version of Unicode Perl is compiled with, but it is easier to just use the latest version of Perl, which is 5.20.

http://perldoc.perl.org/perluniintro.html http://perldoc.perl.org/perlunitut.html http://perldoc.perl.org/perlunifaq.html http://perldoc.perl.org/perlunicode.html

p.s. I noticed in the Python talk you linked that no one knew that the pile of poo symbol is in there because the japanese characters for luck and poo are very similar. ( I am unable to find a link to where I first read this ) The Japanese are also responsible for why we call them emoji (e means image, and moji means character. ) http://www.fastcompany.com/3037803/the-oral-history-of-the-p...

[duskwuff]

The documentation you're looking at is quite old; I'd recommend looking at the current version for a better view. (The implementation is largely the same; the documentation has just improved quite a bit since then.)

http://perldoc.perl.org/perluniintro.html

Anyways, Perl5's Unicode support is quite different from Python's.

Specifically, it doesn't have distinct "Unicode string" and "byte string" types; instead, it has a single unified string type. These strings may be internally stored as Latin1 or UTF-8, depending on how they were created, but they behave identically for almost[1] all purposes, and there are easy ways to force Perl to convert between the formats. It's still possible to create a nonsensical string if you do something silly like append Unicode characters to a string containing raw UTF-8 data, but that's not something the language can entirely protect you from.

[1]: The only exceptions I'm aware of are functions which explicitly operate on the utf8 status, like utf8::is_utf8(), and bitwise operations like &|^ and vec().