[efitz]

Why does the successor to the terminal need to be text oriented at all?

Maybe it is an API. Maybe the kernel implements this API and it can be called locally or remotely. Maybe someone invents an OAuth translation layer to UIDs. The API allows syscalls or process invocation. Output is returned in response payload (ofc we have a stream shape too).

Maybe in the future your “terminal” is an app that wraps this API, authenticates you to the server with OAuth, and can take whatever shape pleases you- REPL, TUI, browser-ish, DOOM- like (shoot the enemy corresponding to the syscall you want to make), whatever floats your boat.

Heresy warning. Maybe the inputs and outputs don’t look anything like CLI or stdio text. Maybe we move on from 1000-different DSLs (each CLI’s unique input parameters and output formats) and make inputs and outputs object shaped. Maybe we make the available set of objects, methods and schemas discoverable in the terminal API.

Terminals aren’t a thing of the 80s; they’re a thing of the early 70s when somebody came up with a clever hack to take a mostly dumb device with a CRT and keyboard and hook it to a serial port on a mainframe.

Nowadays we don’t need that at all; old-timers like me like it because it’s familiar but it’s all legacy invented for a world that is no longer relevant. Even boot environments can do better than terminals today.

[vlovich123]

> Heresy warning. Maybe the inputs and outputs don’t look anything like CLI or stdio text. Maybe we move on from 1000-different DSLs (each CLI’s unique input parameters and output formats) and make inputs and outputs object shaped. Maybe we make the available set of objects, methods and schemas discoverable in the terminal API.

This is Powershell. It’s a cool idea for sure. One thing I’ve noticed though is that it becomes closer to a programming language and further away from scripting (ie you have to memorize the APIs and object shapes). And at that point, why would you write the program in a worse programming language?

By comparison, I’ve noticed even windows-leaning folks do a better job remembering how to delete files and find files than doing so through cmd.exe or powershell. I think that’s because you can run the command to see the output and then you know the text transformation you need to apply for the next step whereas powershell shows you formatted text but objects in the pipe.

Maybe a better terminal that provided completion for commands with AI support and a uniform way to observe the object shapes instead of formatted text might mitigate this weakness but it is real today at least imho.

[fainpul]

> whereas powershell shows you formatted text but objects in the pipe

Somewhat true. However it's easy to explore what methods and properties are available. Just add `| gm` (Get-Member) to the end of your pipeline to see what you're dealing with and what's available.

[rbanffy]

> Why does the successor to the terminal need to be text oriented at all?

Terminals are not "text oriented". They are based on a bidirectional stream of tokens - that can be interpreted as text, or anything else.

That simplicity allows for Unix-style composition. If you make the output something different, then the receiving program will need to be able to parse it. The Amiga OS had some interesting ideas with different data types as system extensions - you'd receive "an image" instead of a JPEG file and you could ask the system to parse it for you. In any case, that's still forcing the receiving program to know what it's receiving.

One way to add some level of complexity is to add JSON output to programs. Then you can push them trough `jq` instead of `grep`, `sed`, or `awk`. Or push it through another tool to make a nice table.

> it’s all legacy invented for a world that is no longer relevant.

I hear that since the Lisa was introduced. Character streams are a pretty common thing today. They are also very useful thanks to their simplicity. Much like Unix, it's an example of the "worse is better" principle. It's simpler, dumber, and, because of that, its uses have evolved over decades with almost no change to the underlying plumbing required - the same tools that worked over serial lines, then multiplexed X.25 channels, then telnet, now work under SSH streams. Apps on both sides only need to know about the token stream.

[naikrovek]

> One way to add some level of complexity is to add JSON output to programs. Then you can push them trough `jq` instead of `grep`, `sed`, or `awk`. Or push it through another tool to make a nice table.

That's still text. Even PowerShell passes objects between commands.

Plan9 did this correctly. A terminal was just a window which could run graphical applications or textual applications. Locally or remotely. It all worked. You create a window, you get a shell with a text prompt. You can do text stuff all day long. But maybe you want that window to be a file manager, now? Launch vdir, and now that same window is home to a graphical file browser. close that and remote into another plan9 machine. launch doom. it runs. it all just works, and it all works smoothly.

And the entire source code for that OS could fit into one person's brain.

It is a very simple OS, appears (to my layman eye) to have sandboxing between all applications by default (via per-process namespaces) making it very easy to keep one application off of your network while allowing others to talk via network as much as they want, for example.

[jcgl]

> Heresy warning. Maybe the inputs and outputs don’t look anything like CLI or stdio text. Maybe we move on from 1000-different DSLs (each CLI’s unique input parameters and output formats) and make inputs and outputs object shaped. Maybe we make the available set of objects, methods and schemas discoverable in the terminal API.

Entirely agree. Stdio text (which is really just stdio bytes) deeply limits how composable your shell programs can be, since data and its representation are tightly coupled (they're exactly the same). I wrote a smidgin here[0] on my blog, but take a look at this unix vs. PowerShell example I have there. Please look beyond PowerShell's incidental verbosity here and focus more deeply on the profoundly superior composition that you can only have once you get self-describing objects over stdio instead of plain bytes.

  $ # the unix way
  $ find . -name '*.go' -not -name '*_test.go' -ctime -4 -exec cat {} \; | wc -l
  7119
  $ # the powershell way
  $ pwsh -c 'gci -recurse | where {($_.name -like "*.go") -and ($_.name -notlike "*_test.go") -and ($_.LastWriteTime -gt (get-date).AddDays(-4))} | gc | measure | select -ExpandProperty count'
  7119

[0] https://www.cgl.sh/blog/posts/sh.html

[rbanffy]

I have a distaste for the verboseness of PowerShell, but I also have concerns with the attempt to bake in complex objects into the pipeline. When you do that, programs up and down the stream need to be aware of that - and that makes it brittle.

One key aspect of the Unix way is that the stream is of bytes (often interpreted as characters) with little to no hint as to what's inside it. This way, tools like `grep` and `awk` can be generic and work on anything while others such as `jq` can specialize and work only on a specific data format, and can do more sophisticated manipulation because of that.

[jcgl]

> I have a distaste for the verboseness of PowerShell, but I also have concerns with the attempt to bake in complex objects into the pipeline. When you do that, programs up and down the stream need to be aware of that - and that makes it brittle.

Yeah, you definitely can't write tools for the unix shell that assume some kind of self-describing message encoding. I mean, you could, but you'd have to do a lot of work to wrap it so that it can work with unix byte streams at the edges. I believe oil shell and nushell have prior art on this. To your point, it should be telling that those are shells of their own, rather than tools for existing unix shells.

> One key aspect of the Unix way is that the stream is of bytes (often interpreted as characters) with little to no hint as to what's inside it. This way, tools like `grep` and `awk` can be generic and work on anything while others such as `jq` can specialize and work only on a specific data format, and can do more sophisticated manipulation because of that.

This seems backwards to me. grep and awk are extremely fragile because they have to look at what's inside. They have to read every byte, and the user of grep and awk must understand entirely what the incoming data will be.

Whereas with PowerShell or any other system with self-describing messages, the user makes some lightweight assertions about the abstract shape of the data--not the concrete shape of that data's representation.

[fainpul]

> When you do that, programs up and down the stream need to be aware of that - and that makes it brittle.

You can go from object- to good old text processing with *nix tools no problem.

Instead of using 100% PowerShell to count all lines in the text files:

  gc *.txt | measure | select -ExpandProperty count

you can switch to `wc` if you like:

  gc *.txt | wc -l

`gc` is Get-Content – basically cat. You can also use awk, sed, jq etc.

[forgotpwd16]

What GP talks about is illustrated with the following modification of your example:

  PS /home/user> gc *.txt | measure | cat | select -ExpandProperty Count
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.
  Select-Object: Property "Count" cannot be found.

Essentially at every step you need to consider whether the preceding command outputs objects or not. This isn't the case with the Unix way. The pipeline always carries a stream of bytes. You only have to consider how to interpret that stream.

[jcgl]

> Essentially at every step you need to consider whether the preceding command outputs objects or not.

This is true, no matter what language, paradigm, or even universe you're in; data that gets passed into a pipeline needs to have the abstract shape that the pipeline expects. This is always true, and it's every bit as true of unix byte streams.

You can of course have pipelines that try to coerce data or assert its structure. The PowerShell example you showed does the latter, and raises an error message that the assertion failed.

Unix byte streams do neither. There's no coercion, no assertion. Just blind trust. When you have IFS set incorrectly, you simply get a wrong answer. When you grab the wrong field number with cut or awk, you get a wrong (or empty) answer. The input data matters every bit as much with unix as it does with every other system of computation. What changes are characteristics like brittleness and enforceability of invariants.

[fainpul]

jcgl already answered this well. My example was merely to show that you can go from PS objects to old fashioned text processing without issues. Any non-PS tool will just receive the text representation of the objects – exactly as you see them in the terminal. I wasn't trying to imply you can go back and forth between them like magic (which should be obvious).

Once you have lost the objects and work with simple text, you have to use the text processing tools of PS, if that's what you want to do. To continue your example:

  gc *.txt | measure | cat | sls count 

sls (Select-String) is like grep.

(Note: this example is nonsense, just to show that it works)

[mongol]

Another question is if such pipelines should act on objects, or more structured text streams. Many programs can output json, is this a pragmatic way forward to extend and improve further?

[jcgl]

Replying out-of-order to your points.

> Many programs can output json, is this a pragmatic way forward to extend and improve further?

Yes, json is a pragmatic way to put a band-aid on this issue. It really is Good Enough™ for many things, and shell tools that optionally emit json are strictly an improvement, imo.

> Another question is if such pipelines should act on objects, or more structured text streams.

"Structured text" is inherently brittle, compared to a self-describing message format. For one, even when you structure your text, you will always end up dealing with escaping problems.

Even worse, imo, is that backwards compatibility becomes a terrible challenge. Let me illustrate with a best-case scenario for structured text: ASCII-only tabular data (nested, binary, unicode data is left as an exercise to the reader, in true unix fashion). The most natural thing to do in unix is to select the relevant fields with awk. More specifically, you select fields by their number. Well, now your pipeline is tightly coupled to column ordering.

In principle you could use awk to select based on field names, those field names are just text and so it becomes incumbent on you, the pipeline author, to make sure they don't get lost in transit. No easy tail or grep for you!

All of this becomes avoidable when you deal with self-describing messages instead of raw bytes.

[naikrovek]

Plan9 shed the textual terminal baggage we all carry today, and it did so in 1995.

The terminal of plan9 was just a window. By default you got a shell with a textual prompt, but you can launch any graphical application in there or any textual application. you can launch a 2nd window manager with its own windows. you can run doom. you can `ls` and `ssh` all you like. it all just works.

this debuted in Plan9 in 1995 or so. 30 years ago we had the terminal of the future and the entire world ignored it for some reason. I'm still a bit mad about it.

[jkl5xx]

Totally agree with all of your comment. To solve for structured data instead of everyone writing parsers, I’ve enjoyed using nushell (not affiliated, just love the idea). https://www.nushell.sh/

It’s like powershell but not ugly and not Microsoft.

[kevinrineer]

I love the idea of nushell. Do you have any worry about the lack of portability of having and becoming so familiar with such a tool that you become reliant on it?

I have this feeling with most things that are not the "default", especially when I think of getting new tools adopted into a conservative workplace.

[fainpul]

If you want to adopt it in your workplace, I would not recommend nushell. They have severe, long standing bugs which don't get fixed. The most annoying to me is output formatting:

https://github.com/nushell/nushell/issues/13601

https://github.com/nushell/nushell/issues/16379

[bawolff]

> Why does the successor to the terminal need to be text oriented at all?

I think because we already have non-text based terminal succesors.

I think there is interest in a succesor to text-bassd because a lot of people both like them but the space has been rather stagnant for a while.

To put it bluntly, what if its nothing like you ever imagined isn't all that interesting as speculation because it doesn't commit to any choices. The proposal has to be imaginable to be interesting.

[rbanffy]

> I think because we already have non-text based terminal succesors.

We've had them for a long time. There have been multiple graphics standards terminals supported - Tektronix, ReGIS, Sixels, up to richer, less successful interfaces (such as AT&T's Blit and its successors - all gorgeous, all failed in the marketplace).

The notebook interface popularized by iPython is an interesting one, but it's not really a replacement for a terminal.

[abnercoimbre]

Self-plug since you're my exact target audience: check out Terminal Click [0]. It's still early days though.

[0] https://terminal.click

[gsinclair]

I looked at that link but didn’t understand what I was looking at and didn’t see a link that would give me an overview. I’d like to know more about it.

[abnercoimbre]

For sure, it's high on my TODO list to overhaul the site over Thanksgiving. If you scroll down a bit and play the 3-minute trailer you'll get the gist of it - pardon the inconvenience!