[em-bee]

that was kind of a rhetoric question. i was already pretty sure it would work. so the real question is what is actually needed.

i found the call documentation. that's a good start, but that is pretty much what every other language also provides. what is still needed is a wrapper that allows me to write

    somestr: some command --with arguments

and have that string be populated with the "some" command output. but then we also need to capture stderr and the exit value, so we actually need to capture three return values. next, support for pipes in some form would also be needed.

i have worked with lisp and smalltalk and also TCL so i am aware of course.

with so many other interesting languages, it's hard to justify investing more time into learning Red in particular

i actually find red in particular one of the more interesting languages. alongside lisp and smalltalk. haskell, erlang and ocaml are also interesting, but for shell use i am specifically looking at languages that use whitespace as an argument and value separator and which allow complex expressions to be written on one line, since that happens often in shell commands. that reduces the list of interesting languages quite a lot. if you know any others that you think are worth learning, please share.

btw, your website seems to be down.

[Izkata]

> and have that string be populated with the "some" command output. but then we also need to capture stderr and the exit value, so we actually need to capture three return values.

I haven't used Red, but assuming this part works like Rebol, this works with different refinements. Create empty strings (which are mutable) and pass them in as extra args to populate them:

  stdout: {}
  stderr: {}
  exitcode: call/output/error {some command --with arguments} stdout stderr

That command string is passed to a shell so pipes/etc work within it. Don't remember how it picks the shell so no idea what would happen if Red/Rebol was set as your shell.

[klibertp]

Ah, yeah, that makes sense, then.

I tried writing a shell in GNU Smalltalk, but had to mess with the parser - otherwise, the need to parenthesize subexpressions makes it really tedious. I implemented |> as a pseudo-operator that implicitly parenthesized everything before it, which was then removed from the expression. It worked for many simple cases, but wasn't pretty and quickly broke down for more complex ones.

I think the only other language worth investigating in this context is Raku. I avoided it for a long time due to its PERL ancestry, but it's actually a pretty well-designed language, and supports multimethods and overloading of pre-/post-/in- and circumfix operators. The operators support precedence and associativity. If you want an embedded DSL for shell one-liners, I think Raku could deliver. It unfortunately uses the comma for argument separation; you could maybe work around that with quotations and macros, but last I checked, these were experimental and not documented. There was a major effort to move to a better-designed parser that would support AST-based macros, but I'm not sure how far along that is today.

Io would work, though you'd also need to mess with a parser a bit (it's runtime-extensible, though, and also allows circumfix operators and passing unevaluated code as Message objects, which is very helpful in this case). Arguments are normally in parens and comma-separated, but you can simply ignore that. This: `some command --with arguments` is actually a valid Io expression (if `--` is a prefix operator), and since you're not evaluating the code, it doesn't matter if it doesn't make sense semantically. But Io is not very actively maintained (a shame!) It's still a very interesting design. It's described in "Seven Languages in Seven Weeks", if you want a quick intro.

Prolog would probably work, but you'd need to write a metainterpreter - otherwise, threading state in/out of predicates through logic variables would be a nightmare in the CLI. There's also the issue of commas as separators.

You know of TCL, so no need to mention it, though it is a pretty nice fit. A more exotic direction could be concatenative languages: maybe Factor, or Joy, or Kitten. All either experimental or unmaintained, currently. They tend to use whitespace for a separator, and I think they all allow for passing around unevaluated code one way or another.

Looking at the list - Red does look like one of the better candidates, but in normal Red code, all functions know their arity. This means you don't need separators for expressions. It doesn't work that way for shell commands, so you'd probably need to write a Parse grammar, include a separator to allow for variadic commands, and then, after splitting into subcommands, interpret the block command by command. I'm sorry, I'm not too knowledgeable about Red, so I can't give you more details, but that should be the general shape: `shell [ var: cmd1 arg --switch arg2 <some_separator> cmd2 <some_symbol>var ]` where `shell` passes the block to parse, gets a list of commands (including ones that bind results, which would appear as `var_name,command` pair; also, I think path expressions can be used in assignment, so `var/stderr: some command` could be included and handled easily), then walks through the list while setting vars and interpolating them in the following commands. Handling piping wouldn't be much harder than extending the Parse grammar and handling pipelines in the interpreting phase. Finally, you could add refinements on shell for simpler cases (just return exit code, capture all output, etc.) Subshells could be just nested blocks. Thinking about it, it could be a pretty fun way to learn more about Red. Ping me if you end up creating something like that, I'd be very interested to read :)

> btw, your website seems to be down.

I know - I broke my homegrown SSG some time ago and never had the time to fix it :( EDIT: restored some version of the site, 90% of the content still missing.

[klibertp]

I had a bit of back-and-forth with ChatGPT about the possible implementations[1] - it shows how little I know about Red, unfortunately, but we came to this representation:

    shell [
        var: [cmd1 arg --switch arg2]
        [cmd2 :var]
    ]

as a general shape of the DSL. The need for a separator goes away (and simplifies the grammar a lot) if you require commands to be nested blocks. :var works for interpolation, because it's loaded as get-word! - trying to cram $var there wouldn't work (that's a syntax for `money!` - only digits are accepted after `$`), but if you want A shell, not BASH specifically, I think you can get away with some syntactic changes. I checked some of the things the model proposed in the REPL, and they mostly seem to work. Subshells and piping would probably require more work in this setup, but it should be doable.

Again: really interesting language and seemingly a good fit for an internal shell DSL. It's also trivial to make files written in this DSL execute as scripts. I might have jumped the gun a little with my "why would you" question at the start :) Personally, I would still seriously consider Io, but that's just a familiarity bias: I spent a year hacking Io internals, while I know little about Red.

[1] https://chatgpt.com/share/6a469cbb-1754-83eb-94b8-14bef67735...

[em-bee]

wow, i guess you got hooked :-)

i looked through that chat log. unfortunately over my slow connection it is painful to read. i had to manually copy it one section at a time because it would not load the whole thing at once and kept discarding and reloading sections that are out of view (that's why the slow connection made it painful, anyways, end rant)

i haven't read all of it in detail yet, but i get the gist, and in particular the conclusion. thank you.

i would go for something a little different, possibly a bit more complex DSL:

basically one key syntax feature i would like to have is the ability to run shell commands or red expressions and use their string output as an argument to another shell command:

like in bash:

    cmd1 arg1 $(cmd2 someargs)  arg3 $(cmd3) ...

in red DSL

    shell [cmd1 arg1 [cmd2 someargs] arg3 [cmd3] ...]

the string output that cmd2 someargs and cmd3 return are passed as arguments for cmd1

in addition to that every field in the DSL should also be replaceable with a red expression:

    mycmd: "command"
    shell [:mycmd arg1 :[1 + 3]]

i put the colon there to mark red expressions, but ideally the DSL should figure out on its own whether something is a shell command or a red expression. the two should not be distinct. something along the lines of: let's try if it is red code first, and if not, then it must be shell (since lookups for shell commands are more expensive)

that means we need a different separator for a sequence of shell commands. either ; or a line break. unless the command is nested in a red expression:

    var: [cmd1 arg --switch arg2] [cmd2 :var]

since we know that var: takes one argument it is clear that only one shell expression can follow. whereas

    ls -l [cmd1] [cmd2]

would pass both cmds as arguments to ls. if i didn't want that maybe i would write:

    [ls -l [cmd1]] [cmd2]

or

    ls -l [cmd1]
    cmd2

or some other separator.

but if we allow any red expression then the first syntax would automatically be supported. we could not get rid of it even if we wanted to. whereas any other separator would require special consideration.

lastly, in an actual shell/repl shell [ ... ] would be wrapped around every line, so i would not have to write it every time.

these are just some thoughts. handling stderr and return values still needs consideration as well as high level data types. one of the limitations of todays shells is that commands can only pass strings, but i would like for commands to be able to return lists, structs, etc. elvish, nushell and others are exploring that space. one approach is that shell commands can return json that can be converted to native objects. some kind of wrapper for that would be interesting.

i am not actively trying to create a new shell, but i am exploring languages for their potential to design a shell language without having to reinvent the whole syntax from scratch.

red's ability to create powerful DSLs without having to parse everything manually is very interesting here. lisp is the only other language with the same ability that i am familiar with. (TCL too, but TCL doesn't have strong datatypes and that disqualifies it for me) . i think even in smalltalk this would be more difficult than red or lisp.

[klibertp]

> wow, i guess you got hooked :-)

Maybe a little :D REBOL line of languages has been on my radar for a long time - I've been a PL nerd for 20 years now - but I always found reasons not to engage with them. So it was a bit of a blind spot: I knew some basics, considered it a pretty elegant and powerful design, but the REBOL3 fiasco, the proprietary nature of REBOL2, and the various facts about Red (pivot to it being a "blockchain native" solution was painfull to watch, dependency on REBOL2 for the toolchain/no self-hosting after almost 15 years in development, etc.) made it hard to dive deeper into the language(s). I had some time yesterday, and your comment touched on something I'm already interested in (interactive CLI/universal REPLs), so I took the opportunity to finally learn more about the language.

> i am not actively trying to create a new shell, but i am exploring languages for their potential to design a shell language without having to reinvent the whole syntax from scratch.

As for me, I tried making shells a few times. All those failed, of course, mostly because I wanted to do something a bit different: a text-based computing environment with network transparency, basically a single-player MUD/MOO that would treat the filesystem as a world. I got quite far with Smalltalk - but GNU Smalltalk that I based it on collapsed under me :( I tried Elixir, Io, and Python, and all would require much more work than I was willing to put in. A pity, because I think that would be the best possible environment for LLMs/agents... I'm still thinking about this idea often, and I started working on a new implementation, using Pharo Smalltalk/GToolkit. Parsing is the easier part, actually: it's the liveness of the whole thing (hot reload, adding functionality from within the environment, mapping the environment back to source code, etc.) is the part I had the most problems with, and Smalltalk seems like the most promising platform for implementing it.

[oldes]

The only fiasco with Rebol3, as I see it, is that everybody was asking for the language to be open-sourced, and when it finally happened, almost nobody was willing to take on the responsibility of maintaining it once Carl left, and almost the entire community disappeared - many to the Red camp.

Btw, Rebol is still being developed, without the masses noticing it.

https://rebol.tech/ https://github.com/Oldes/Rebol3/releases

[em-bee]

i suspect the community disappeared and moved to red already before rebol3 opened its license.

but if rebol is still being developed then someone (evidently that's you) eventually did take on the responsibility of maintaining it.

i have no clue here. i even only know about red because i met nenad in china. otherwise this whole ecosystem would not be on my radar)

so tell me, what motivated you to pick up rebol3, how does it differ from red and from rebol2 (the repo mentions making rebol3 as usable as rebol2 which implies that as released it was less usable. why did that happen? they could not open everything?)

one difference i see right away is that red is written in red/rebol whereas the original rebol is written in C. being selfhosting is a major feature for a language in my opinion.

[em-bee]

your scope of interest seems to have a lot of overlap with mine. though i may be am aiming in a slightly different direction. i am also coming from the MUD world (for my websites i use a platform that has a MUD like architecture underneath :-), and i have a strong interest in a better commandline. i am less focused on exploring languages, that's more of a side quest.

what i really want is a commandline that works more like jupyter notebooks. each command is its own entity, that can be edited and run again. there is an output window/frame separate from the input/command, just like the jupyter. that output can run anything, graphics, an interactive app, like an editor, even a filebrowser. commands can produce structured data that can be interpreted and visualized... (if only i could find funding for this idea :-)

and, yes, i agree, smalltalk has a lot of potential for this. the downside of smalltalk i fear is that it is to bulky for the average user, and this system need to be able to integratable with unix commands written in any language so the interface needs to be agnostic (perhaps json? and another more suitable for binary data, also apps need to be able to run and display output directly to the screen, not depend on pharo to display).

there was a time when i tried using pharo as a desktop. i used a terminal written in smalltalk, and i was looking at finding or building tools a desktop needs (workspace management, a clock, all these small utilities)

if you have something to play around with, i'd be curious to give it a try.