[hnlmorg]

There’s also a lot of good design that’s gone into Elvish. And I don’t think it’s fair to call it “dead” when the maintainers for Elvish are active both on Github and here on HN too (probably other places too).

However if you’re looking for an alternative then there’s:

- Murex (disclaimer: I’m one of the maintainers) which does support background processes and has extensive documentation. https://murex.rocks

- Nushell: I’m not personally a fan of its design choices but it has a large following of people who do really enjoy it so it might also appeal to yourself too.

As for Elvish, I do encourage others to give it a go themselves. It’s really well thought out and what might be a deal breaker for some people isn’t for others.

[rendaw]

Elvish had some very cool ideas, which is why I tried it out! Like the built in script checker! But it also has a lot of very basic issues that have been open for years, and TODOs in the documentation as I mentioned. People are going to read your message and put N hours into it and get burned, and I think this is a fair warning.

Nushell also had very minimal background task support, so I rejected that. They explicitly say use some other program for background tasks in their docs.

I actually looked at Murex after seeing it in previous threads, but I bounced for some reason... I just took another look though skipping the tutorial and I see you have `bg` and `fg` support! But does `bg` return the `fid`? Can you use those in scripts, or are they hobbled the same way bg/fg are in bash?

It's been a good 4-5 months since I went down this rabbit hole, but IIRC the basic things I wanted to do and got blocked in multiple shells were:

- System-wide interactive-use config file, I use Nixos and manage my system config using that

- Background task support - I need to start an ssh tcp proxy, pipe a command over it, then kill ssh once the command is done (all in a script).

- Post-command hook, to send a notification when a long command finishes

- Async iteration of command output, i.e. streaming events with swaymsg subscribe and running a command when certain events occur

- Value/call arity safety - i.e. a clear distinction between a single value and multiple values that doesn't rely on stringification hacks. I.e. in `command $x` `command` should always have one argument, regardless of the contents of `x`, and making that plural should be explicit.

And then other standard evaluation criteria, like I looked at xonsh but it seemed like a massive hack, despite handling a lot of the above.

[hnlmorg]

> does `bg` return the `fid`

There's two kinds of process IDs in Murex: FID (function IDs) and PID (process IDs)

Forking is expensive in POSIX and has a number of drawbacks such as the inability to share scoped variables without resorting to environmental variables. So a FID is basically a PID but managed inside the scope of Murex's runtime. You can manage FIDs in much the same way as you can manage PIDs, albeit using Murex builtins rather than coreutils (though PID management tools in Bash are technically builtins rather than coreutils too).

What this means in practice is you can have entire blocks of code pushed into the background, eg

    » GLOBAL.name = "rendaw"
    » bg { sleep 5; echo "Hello $name" }; echo "not bg"
    not bg
    Hello rendaw

You can see the FID as well as the job ID the usual way, via `jobs`

    » jobs
    JobID  FunctionID  State      Background  Process  Parameters
    %1     2109        Executing  true        exec     sleep 5

...and you can kill that entire `bg` block too

    fid-kill 2109

But you'd also see any non-builtins in `ps` too:

    » ps aux | grep sleep
    hnlmorg   72749   0.0  0.0 410743712   1728 s012  S+    4:24p.m.   0:00.00 /usr/bin/grep --color=auto sleep
    hnlmorg   72665   0.0  0.0 410593056    432 s012  S+    4:23p.m.   0:00.00 /bin/sleep 5

> Can you use those in scripts, or are they hobbled the same way bg/fg are in bash?

While the above seems very complicated, the advantage is that `bg` and `fg` become much more script friendly.

> - System-wide config file, I use Nixos and manage my system config using that

This isn't Murex's default behaviour but you could easily alter that with environmental variables: https://murex.rocks/user-guide/profile.html#overriding-the-d...

The latest version of Murex (v7.0.x), which is due to be released in the next few days, makes this even easier with a $MUREX_CONFIG_DIR var that can be used instead of multiple specific ones.

> - Background task support - I need to start an ssh tcp proxy, pipe a command over it, then kill ssh once the command is done (all in a script).

Murex has another layer of support for piping in addition to those defined in POSIX, which are basically channels in the programming language sense. In murex they're called "Murex Named Pipes" but the only reason for that is that they can be used as a glue for traditional POSIX pipes too. This is one area where the documentation could use a little TLC: https://dev.murex.rocks/commands/pipe.html

> - Post-command hook, to send a notification when a long command finishes

There are two different events you can hook into here:

- onPrompt: https://dev.murex.rocks/events/onprompt.html

This is similar to Bash et al prompt hooks

- onCommandCompletion: https://dev.murex.rocks/events/oncommandcompletion.html

This hooks into any command name that's executed. It runs the comment in a new TTY and buffers the command's output. So, for example, if you want a command like `git` to automatically perform a task if `git push` fails with a specific error message, then you can do that with onCommandCompletion.

> - Async iteration of command output, i.e. streaming events with swaymsg subscribe and running a command when certain events occur

I'd need to understand this problem a little more. The channels / Murex Named Pipes above might work here. As might onCommandCompletion.

> - Value/call arity safety - i.e. a clear distinction between a single value and multiple values that doesn't rely on stringification hacks. I.e. in `command $x` `command` should always have one argument, regardless of the contents of `x`, and making that plural should be explicit.

This one is easy: scalars are always $ prefixed whereas arrays are @ prefixed. So take the following example:

    array = %[ a b c ]
    
    » echo $array
    ["a","b","c"]  # a single parameter representation of the array

    » echo @array
    a b c          # the array expanded as values

-----

This is quite a lengthy post but hope it helps answer a few questions

[rendaw]

Thanks for the answer! I think we may be talking past eachother a bit, but it's good to get confirmation on a lot of those!

AFAICT named pipes have nothing to do with ssh tcp proxies, or at least that bit is tangential to the key point - the key point I was making is that ssh is running in the background while I'm running another command (with no relation between them, as far as the shell is concerned).

You didn't answer my question about if `bg` returns a `fid` or not, and the documentation doesn't answer this either, nor did you say if those could be used in scripts... but it sounds like you're saying I have to parse `jobs` to get the `fid` of the command I just launched?

> Async iteration

This isn't about objects but about how operators evaluate. Maybe "streams" or "async streams" would be a better description? Actually, maybe this is where the named pipes you mentioned would be useful?

    while true; do echo hi; sleep 1; done | while read line; do echo $line 2; done

in bash, prints "hi 2" once a second. That is, the body of the while loop is executed asynchronously with the pre-pipe command.

AFAICT elvish doesn't have a `read` command, and `for` waits for the argument to complete before executing the body at all. This is a pretty common pattern, so I was surprised when there's no mention of it in the elvish docs. I don't like `read` in bash since it's yet another completely different syntax, but maybe Murex has something similar?

TBH I just looked at the Murex docs again and I'm lost. Where's the reference? There's the "user guide" which contains the "user guide" (again) and also the "beginners guide" which to me are synonyms, "cheat sheet" and "read more" which both appear to be a bunch of snippets (again synonyms?), "operators and tokens" which are a reference of just a subset of the language, etc etc. I couldn't find anything on loops, which I'd expect to be a section somewhere. Clicking on "read / write a named pipe" in "builtin commands" somehow teleports me to an identically named page in a completely different "operators and tokens" section.

In the end the read/write pipes page only shows examples of reading/writing pipes to other pipes, not using them in a for loop or anything. One example appears to use some unique syntax in `a` to send array elements to the pipe, but I couldn't find anything about that in the `a` section when I finally found the `a` section.

Also are named pipes always global? Why can't they be local variables like other structures?

It's great you have so much documentation, but I think it actually turned me away - it seems bloated with every bit of information split into multiple pieces and scattered around. Simplifying, consolidating, and reorganizing it from the top might help.

> Arity safety

Thanks! Yeah, I wasn't talking about arrays here, but if e.g. `x` in my example contains space-separated "these three words", will `command` receive 3 arguments or 1. In bash, `command` would get 3 arguments, if you don't invoke the variable quote hack. I just tried this out though, and it gets 1 argument, so great!

[hnlmorg]

> You didn't answer my question about if `bg` returns a `fid` or not, and the documentation doesn't answer this either, nor did you say if those could be used in scripts... but it sounds like you're saying I have to parse `jobs` to get the `fid` of the command I just launched?

Everything in the documentation can be used in scripts.

`bg` doesn't write anything to stdout. The documentation does actually answer that if you look at the Usage section. In there it doesn't list <stdout>. eg compare `bg` it to other builtins and you'll see the ones that do write to stdout have `-> <stdout>` in the usage. Also none of the examples show `bg` writing anything to stdout.

You wouldn't want `bg` to write the PID to stdout anyway because then it would be harder to use in scripts, because you'd then need to remember to pipe stdout to null or risk contaminating your script output with lots of random numbers.

However the good news is you can still grab the PID for any forked process using <pid:variable_name>. eg

    » bg { sleep <pid:MOD.sleep_pid> 99999; echo "Sleep finished" }
    » kill $MOD.sleep_pid
    Sleep finished

($MOD is required because bg creates a new scope so any local variables created in there wouldn't be visible outside of `bg` block. MOD sets the scope to module level (also supported are GLOBAL and ENV).

It's also worth reiterating my previous comment that `bg` blocks are not separate UNIX processes but instead just different threads of the existing Murex process. In fact all Murex builtins are executed as threads instead of processes. This is done both for performance reasons (UNIX processes are slow, threads are fast...relatively speaking) and because you cannot easily share data between processes. So if `bg` was a process, it would be impossible to grab the PID of `sleep` and then share it with the main process without then having to write complex and slow IPCs between each UNIX process.

> That is, the body of the while loop is executed asynchronously with the pre-pipe command.

Are you just talking about each command in the pipe executing concurrently? That's the default in Murex. for example the following would only work if each command is working as a stream:

    tail -f example.log | grep something | sed -e s/foo/bar/ | tee -a output.log

The loop example could be achieved via `foreach` too. eg

    while { true } { echo hi; sleep 1 } | foreach line { echo "$line 2" }

The only time `foreach` wouldn't stream is with data-types that aren't streamable, such as JSON arrays. But that's documented in the JSON docs (https://murex.rocks/types/json.html#tips-when-writing-json-i...)

> TBH I just looked at the Murex docs again and I'm lost. Where's the reference? There's the "user guide" which contains the "user guide" (again) and also the "beginners guide" which to me are synonyms, "cheat sheet" and "read more" which both appear to be a bunch of snippets (again synonyms?), "operators and tokens" which are a reference of just a subset of the language, etc etc. I couldn't find anything on loops, which I'd expect to be a section somewhere. Clicking on "read / write a named pipe" in "builtin commands" somehow teleports me to an identically named page in a completely different "operators and tokens" section.

I'd generally refer people to the language tour to begin with https://dev.murex.rocks/tour.html It's pretty visible on the landing page but sounds like we could do more to make it visible when elsewhere on the site. I can take that feedback and work to make to the tour more prominent in the menus too (the pages you described do also link to the language tour, but clearly not in a visible enough way)

> Also are named pipes always global? Why can't they be local variables like other structures?

The only reason for that is because named pipes were one of the original constructs in Murex. They came before modules, scoping, and so forth. It would be possible to make them scopeable too but careful thought needs to be made about how to achieve that in a backwards-compatible yet intuitive way. And there have been other requests raised by users and the other contributors that have taken priority.

> It's great you have so much documentation, but I think it actually turned me away - it seems bloated with every bit of information split into multiple pieces and scattered around. Simplifying, consolidating, and reorganizing it from the top might help.

We'd welcome some recommendations here. Organising documentation is really hard. Even more so when the people who organise it aren't the same people who are expected to depend upon it.

> Thanks! Yeah, I wasn't talking about arrays here, but if e.g. `x` in my example contains space-separated "these three words", will `command` receive 3 arguments or 1. In bash, `command` would get 3 arguments, if you don't invoke the variable quote hack. I just tried this out though, and it gets 1 argument, so great!

I know you weren't talking about arrays, but the next question people ask is "how do I now expand one variable to be multiple parameters?"

[rendaw]

Awesome, thanks for the answers! It looks like Murex will be the first shell I try once I recharge my "rip everything up and start over" batteries again.

> I know you weren't talking about arrays, but the next question people ask is "how do I now expand one variable to be multiple parameters?"

I held back from asking that, but yes, exactly :P

[hnlmorg]

Awesome. Good luck if/when you do. And feel free to pester me on Github with any questions / feedback / complaints you have :)

[em-bee]

could you go into more detail how (and maybe why) murex and elvish differ?

also, i'd like to know more about how the job control works. that's one of the pain points in elvish, but both are written in go, so maybe there are some ideas that elvish could copy.

[hnlmorg]

Murex and Elivish share a lot of similar design goals. The author of Elvish has done a lot of great talks about his approach to Elvish development and you can tell a lot of care has gone into its design.

With Murex, I initially took a "let's just experiment until I find something that works" with no fear of writing ugly proof-of-concept code. This allowed me to build a lot of stuff very quickly and originally it was written in a self-hosted git repository to solve my own problems. But as the project evolved I realised there was some good stuff in there that's worth sharing. The downside to this approach is that there is some ugliness to its design that has lasted even to the latest version due to Murex's compatibility promise. However, the latest version of Murex does provide an internally versioned runtime, which means scripts can now pin to a specific version of Murex and not worry about gradual changes over time (even if "gradual over time" in this context literally means "years of compatibility" even before the versioned runtime.

This means that Murex and Elvish might feel like very different shells despite being conceptually quite similar.

I'm a little reluctant to give specific areas where the two shells diverge because both are under active development and thus moving targets. So what might be true today might not be true tomorrow. However, I will say the syntax for each does vary significantly despite being superficially similar.

As for job control, this was part of Murex's early design because it's a feature I used heavily at the time. So the concept of background and foreground processes are weaved throughout all of the core runtime. Like with Elvish, Murex doesn't create new UNIX processes for builtins. And with commands that are forked processes, Murex doesn't hand over complete ownership of the TTY to them so that Murex can still catch the signals. The reason for the latter is because Murex can then add additional hooks to job control, such as returning a list of open files any stopped processes have opened, and how far through reading those files it is. So Murex has needed to re-implement some of the job control logic that would normally be handled by the POSIX kernel. This does result in a lot of additional code, and thus places for things to go wrong. On balance, I think I made the right tradeoff for Murex. However if I were to write an entirely new shell from the ground up, I'd probably not do it this way again.

[em-bee]

i really like your approach to job control. my hope is that elvish can implement something similar. i am hopeful in that you already managed to overcome the challenges go introduces here, so the elvish devs can potentially take advantage of that.

[hnlmorg]

The limitations here aren’t due to Go. You can define forked processes gpid and ctty, which are the two key pieces you need to define to “correctly” support job control.

And in fact Go actually makes it very easy to both catch job control signals raised by the kernel and set those aforementioned parameters when calling the fork syscall.

The real problem here is that we don’t actually want to POSIX compliant job control because that would mean builtins inside hot paths would perform significantly worse and we lose the ability to easily and efficiently share data between commands, such at type annotations, localised variables, etc.

The lack of type annotations is a particularly hard problem to solve and also the main reason to use an alternative shell like Murex or Elvish. In fact I’d say having type annotations work across commands is more important than job control.

So the end result is having to replicate a lot of what you would normally get for free in POSIX kernels, except this time running inside your shell. In places you’re basically writing kludge after kludge. But whenever I despair about the ugliness of the code I’ve written, I remind myself that this is all running on 40 year old emulation mechanical teletypes. So the whole stack is already one giant hot ball of kludges.

[em-bee]

the challenges in go is a reference to a discussion in the elvish chat where one participant claimed that go's os.StartProces() API makes it impossible to implement unix job control with 100% fidelity.

i don't actually know what the issue there is, and maybe there is a way to avoid using os.StartProcess but the point is that murex is not implementing POSIX compliant job control. and that is one way to get around any issues that may exist.

and now having learned how murex handles job control i am happy that elvish avoided implementating POSIX job control so far because this allows rethinking how to approach this.

this is all running on 40 year old emulation mechanical teletypes

isn't that the real issue right there?

i have been wondering if it is not possible to get rid of that emulation layer and provide for a more rich way for programs to interact with the user.

we'll never be able to get rid of the emulation completely, but i wonder if the position in the stack can be moved.

right now it is:

    GUI
    GUI application that emulates a 40 yr old terminal
    shell
    programs running in the shell

how about:

    GUI
    GUI application for commandline programs that provides a rich interface
    modern shell that runs on that interface
    modern programs running in the shell
    or terminal emulation for legacy programs that need it
       legacy programs running with an emulation layer.

the emulation layer could be started by the shell as needed.

to get that emulation layer we only need to port something like tmux onto that new api. there is also a layer that implements job-control for shells that don't support it: https://github.com/yshui/job-security so this can be done without having to reimplement the emulation yet again

[hnlmorg]

> the challenges in go is a reference to a discussion in the elvish chat where one participant claimed that go's os.StartProcess() API makes it impossible to implement unix job control with 100% fidelity.

That's not true. os.StartProcess() takes a pointer to https://pkg.go.dev/os#ProcAttr which then takes a pointer to https://pkg.go.dev/syscall#SysProcAttr

If I recall correctly, either Setctty or Foreground needs to be set to true (I forget which offhand, possibly Foreground, but browsing the StartProcess()'s source should reveal that.

I don't actually do that in Murex, because I want to add additional hooks to SIGSTSP and I can't do that if I hand ownership of the TTY to the child process.

https://github.com/lmorg/murex/blob/master/lang/exec_unix.go...

But that means that some tools like Helix then break job control in Murex because they don't think Murex supports job control (due to processes being marked non-traditionally). You can see higher up in that file above where I need to force Murex to take ownership of the TTY again as part of the process clean up. (line 28 onwards)

Processes invoked from a shell should also be part of a process group:

https://github.com/lmorg/murex/blob/master/lang/exec_unix.go...

You also need to set the shell to be a process session leader:

https://github.com/lmorg/murex/blob/master/shell/session/ses...

All of this is UNIX (inc Linux) specific so you can see compiler directives at the top of those files to exclude WASM, Windows, and Plan 9 builds. I don't even try to emulate job control on those platforms because it's too much effort to write, test, and maintain.

> i have been wondering if it is not possible to get rid of that emulation layer and provide for a more rich way for programs to interact with the user.

Funny enough, this is something I'm experimenting with my terminal emulator, though it's very alpha at the moment https://github.com/lmorg/Ttyphoon

> to get that emulation layer we only need to port something like tmux onto that new api

My terminal emulator does exactly this. It uses tmux control mode so that tmux manages the TTY sessions and the terminal emulator handles the rendering.

> https://github.com/yshui/job-security so this can be done without having to reimplement the emulation yet again

The problem with a 3rd party tool for job control is that it doesn't work with shell builtins. And the massive value add for shells like Murex and Elvish is their builtins. This is another reason why I didn't want POSIX job control in Murex: I wanted to keep my shell builtins powerful but also allow them to support job control in a way that feels native and transparent to the casual user.

[ghthor]

There is ysh