[JdeBP]

The revealing shibboleth is when people call it "ANSI". (-: "ANSI" is what people call it when they are working from paltry and incomplete samizdat doco of how this stuff works, from Microsoft's old ANSI.SYS appendix to its MS-DOS user manual, to innumerable modern WWW sites all repeating received wisdom.

The thing to remember is that the "E" in "ECMA" does not stand for "ANSI".

* https://ecma-international.org/publications-and-standards/st...

* https://ecma-international.org/publications-and-standards/st...

* https://www.itu.int/rec/T-REC-T.416-199303-I

If you read ECMA-35, you'll find that there's actually a whole system to escape sequences and control sequences. As I pointed out last month, it's often the case that people who haven't read ECMA-35 don't realize that parameter characters can be more than digits, don't handle intermediate characters, and don't grasp how DEC's question mark and SCO's equals sign fit into the overall picture. People who haven't read ECMA-48 and traced its history don't realize that there's subtlety to missing parameters in control sequences. And people who haven't read ITU/IEC T.416 do what many of us did years ago and get 24-bit colour wrong. Twice over.

* https://github.com/tattoy-org/tattoy/issues/105#issuecomment...

Other common errors include missing out on all of the other 7-bit aliases for C1 characters. Or not realising that the ECMA-35/ECMA-48 syntax allows for any control sequence to have sub-parameters, not just SGR. Or using regular expressions and pattern matching instead of a state machine. Only a state machine truly handles the fact that in the real world terminals allowed, and enacted, various C0 and C1 control characters in the middle of control sequences, as well as had ways of cancelling or restarting control sequences mid-sequence.

* https://github.com/jdebp/nosh/blob/trunk/source/ECMA48Decode...

But it gets even worse for a real world control sequence decoder.

In the real world, not only do terminals interpret the same control sequences, and their parameters, differently depending from whether the terminal is sending or receiving them; but several terminal emulators like the one in Interix, rxvt, the one built in to Linux, and even XTerm, send control sequences that not only break ECMA-35 but also conflict with received control sequences. So if one wants to be comprehensive and be cabable of decoding real data, one needs a switch to tell the program whether to decode the character stream as if it is being received by the terminal or as if it is being sent by the terminal.

* https://jdebp.uk/Softwares/nosh/guide/commands/console-decod...

Microsoft Terminal tries to do things properly, which many modern terminal emulators and tools do not, and handles this with two distinct entire state machines, one for input and one for output.

* https://github.com/microsoft/terminal/tree/main/src/terminal...

I handled it with a few goto statements and a handful of flags. (-:

* https://github.com/jdebp/nosh/blob/trunk/source/console-deco...

[blueflow]

I think this rant is out-of-place here, type "\x1b[:<=>$t" and check for yourself. It parses correctly. You do learn about the allowed character ranges for CSI sequences from ECMA-48 only, not from the Microsoft docs, so i guess the author did their homework.

[JdeBP]

That tells me that you are writing from ignorance, as for starters that's a truly pathetic test that even misses one of the characters that I explicitly mentioned above, let alone thoroughly tests the full range that the specs define. I had an actual poke around the parser code, in contrast to your superficial experimentation. (-: One can, with knowledge, actually find the point where the only three unusual characters that you in fact tested are special cased.

[blueflow]

They are not special cased:

  https://github.com/webpro/ANSI.tools/blob/main/packages/parser/src/parsers/csi.ts#L12

The comment correctly identifies the 0x30-0x3f range as parameter bytes and the following as intermediate bytes. Both the range and the names for the bytes are matching ECMA-48 Chapter 5.4.

But you seem to think that everyone except yourself is incompetent, are you trying to make up for something?

[webpro]

Thanks. Agreed. The way I see it, ignore the noise and there might be something in there.

[JdeBP]

Of course they are. There's a file with all of the special cased constants in, named constants.ts.

Your superficial test tested all three of the special cases in the PRIVATE_OPENERS array, which is what the parser.ts code actually checks. DEC's question mark, which is special cased yet further off on its own, is in reality another "private opener", too, and it isn't limited to DEC (e.g. XQTMODKEYS), and neither does DEC not use the other non-digit parameter characters (e.g. DECDA3).

(There's a hypothesis that DEC's own state machine didn't care where these marker characters were, as it was a simple state machine that had to fit in ROM and probably just set a bitflag. A mistake that we're probably all still making is assuming that they only take effect when in the very first position.)

STRING_OPENERS is another widespread special casing that people do, treating ESC plus a few characters as special rather than handling all of the 7-bit aliases for the C1 characters as the general case.

You seem to think that people who share what the mistakes are and where they themselves have made these very mistakes over the years, to help other people not make them and so that the world continues to remember this hard-learned stuff, is somehow worthy of ad hominems, straw men, insults, and vilification right off the bat. That's a very poor show and you should be ashamed.

[clucas]

> people who share what the mistakes are and where they themselves have made these very mistakes over the years, to help other people not make them and so that the world continues to remember this hard-learned stuff

But then we have this in your post:

> That tells me that you are writing from ignorance, as for starters that's a truly pathetic test

and

> I had an actual poke around the parser code, in contrast to your superficial experimentation.

Perhaps you really did intend for these lines to be helpful and informative? If so, I encourage you to have a moment of empathy for your interlocutor and ask yourself if talking this way is actually the best way to communicate and pass on this hard-earned knowledge.

> ad hominems, straw men, insults, and vilification

I didn't see this from the other poster. I did see it from you. As a disinterested third party, I'm just telling you, you come off way worse in this exchange. Good luck out there buddy.

[webpro]

That's some interesting feedback, thanks for sharing. I'll see what I can extract and apply from it. Please bear with me, this is only my initial take on the whole concept (and as you point out, it isn't that trivial). Didn't have much examples to be inspired by, but we're on our way anyway.

[webpro]

Some of the issues mentioned in this thread have been improved, including private CSI sequences, default param values, and cancellation + substitution.

[j4_james]

> "ANSI" is what people call it when they are working from paltry and incomplete samizdat doco of how this stuff works

People just use "ANSI" as a shorthand for ANSI X3.64-1979. And that was the standard that DEC used for their VT100+ range of terminals, which in turn became the de facto standard from which most modern terminal emulators are derived. If you read the DEC documentation, you'll find many references to "ANSI standard", "ANSI controls", "ANSI colors", etc. I don't think this is because they were ignorant of the subject matter, considering that they were members of the committee that produced that standard.

And ECMA-48 is essentially just the European equivalent of ANSI X3.64, and was developed in parallel. But obviously an American company like DEC or Microsoft would more likely be working from the American version of the standard rather than the European one.

[ForOldHack]

Ill never forget the comments in termcap: "Brain dead", "Very Brain dead" and "Brain? What brain!" I think most of that was terminals that CTEOL Clear to end of line was just garbage.

We just knew that at some point in time, all the Hazeltine terminals were going to end up in the garbage, which is what they deserved, and no one would rescue them.

https://www.shallowsky.com/linux/noaltscreen.html

The parent post is SOLID GOLD.