[Drup]

Creating strings that match a regex can be very useful for testing too, as it allows you to fuzz a regex engine without using another regex engine as oracle.

It is indeed quite subtle to be "productive" (that's how we say we will explore both "+"). I wrote a paper about how to do it well a few years ago, with quite a few tricks: https://regex-generate.github.io/regenerate/

[eru]

You can use the regular expression engine itself as an oracle to some extent.

Eg you can say that if string s match expression e, than s should also match e|e'. (Where e' is some other expression.)

[Drup]

That is very limited in practice. In my experience, it doesn't scale to more complex features.

[eru]

Well, regular languages are a specific, well defined subset of all formal languages.

If you want arbitrary parsing, use a parsing library?

[Drup]

I disagree with the sibling comments: regular expressions should still be about regular languages. It's not because Larry Wall committed yet another atrocity that we should all follow him off the cliff. Perl can keep its monster.

However, that doesn't mean we shouldn't strive for richer feature (within the confine of regularity), or that a parsing library automatically does the job better.

For instance: intersection and complement are regular operators that works very well in regular expressions. lookahead is a regular operator (lookbehind and backreferences are not), etc. Those operators do not necessarily work at all in richer parsing libraries. When added to regular expressions, they should be well handled and optimized by regex engines (which is is currently not the case). There's still work to do, features to add, which require appropriate testing techniques.

[eru]

Indeed, that's my opinion as well.

Regular languages are closed under a host of more interesting operations than those that have 'grown with the capabilities of our pattern matching engines'.

[slimsag]

Regex engines don't just parse regular languages - that ship sailed a long time ago. As I understand it, any regex engine supporting backtracking is no longer working with regular languages.

As the author of Perl's regex engine Larry Wall puts it[0]:

> “Regular expressions” […] are only marginally related to real regular expressions. Nevertheless, the term has grown with the capabilities of our pattern matching engines, so I’m not going to try to fight linguistic necessity here. I will, however, generally call them “regexes” (or “regexen”, when I’m in an Anglo-Saxon mood).

[0] https://raku.org/archive/doc/design/apo/A05.html