[boomlinde]

You can absolutely pipe programs together using a shell without a terminal. This also doesn't restric in any way the possibility to "output to some kind of graphical device/interface that uses that semantic information to display the data using nice graphical interface elements".

What is it specifically that you want to do which your favorite shell doesn't allow because it is restricted by terminals?

[em-bee]

yes, i can write a program that takes data and displays it nicely. but in order to use graphics i need a second channel to send the graphic instructions, for x11, wayland or even MacOS or wndows. at that point i have two interfaces, the terminal and that graphics display. i want both to be in one. i want the terminal to be that graphical output. that currently is only possible through in-band escape sequences. that is the restriction i want to get rid of.

[boomlinde]

I don't fully understand the request. On one hand you don't want a second channel to send the graphics instructions, but on the other you don't want to use in-band escape sequences.

Maybe you'd be interested to learn about plan9's graphical terminal. Its window manager runs entirely within it, and all windows just represent multiplexed access to limited areas of the terminal.

[em-bee]

On one hand you don't want a second channel to send the graphics instructions, but on the other you don't want to use in-band escape sequences

correct. the in-band sequences are dangerous and unwieldy. they don't convey enough information. they are a hack to work within the limitations if historical terminals. that's what this whole thread is about.

a separate graphics channel creates a separate window. then you have two windows. not good either. it needs to be one window, and considering that this window should be able to support multiple remote connections it needs to be local otherwise i would get a new window for each server i connect to. that works for some people, but not for me. and it needs to work through a single channel like ssh/mosh or another similar protocol and be forwardable.

so i want a third option. one approach is sending semantic data, letting the terminal interpret it and display it graphically. this is interesting because shells are already exploring semantic data. (elvish, murex, nushell, others...)

plan9 sounds interesting. i see several efforts to port aspects of it to linux. they all seem to have stalled. more work needs to be done here. that's what i am advocating.