[chubot]

Meh the problem is "Which Lisp?" There are dozens of incompatible Lisps. Even this site is written in a Lisp dialect written by its author (Arc).

In fact I conjecture that this is the reason Unix is more popular than Lisp -- because Lisps don't interoperate well. They haven't built up a big ecosystem of reusable code.

Whereas Python, JavaScript, R, C, C++, and Rust programmers can reuse each others' code via Unix-style coarse-grained composition. (Not just pipes -- think about a web server running behind nginx, or git reusing SSH and HTTP as transports.)

You can also use link time composition. It takes some work but it's better than rewriting your Common Lisp code from scratch in Clojure.

-----

Honest question: how do you communicate between two Lisp processes on two different machines? I know Clojure has EDN (which is sort of like JSON : JavaScript), but I haven't heard of the solutions for other Lisps.

I wrote about this problem here: A Sketch of the Biggest Idea in Software Architecture http://www.oilshell.org/blog/2022/03/backlog-arch.html

> The lowest common denominator between a Common Lisp, Clojure, and Racket program is a Bourne shell script (and eventually an Oil script).

I'll definitely update it if there's something I'm missing.

I would say the design of Unix is "rotting", but the answer is to IMPROVE Unix. Not dream of clean slate designs that will never be deployed. Plus this post doesn't actually propose anything. If you actually start trying to build your Lisp machine, I believe you will run into dozens of reasons why it's not a good idea.

[bitwize]

You are aware that Lisp machines understood several different flavors of Lisp? The Symbolics ones understood Zetalisp and Common Lisp at least. Were they on the market today they could be convinced to run Clojure and Scheme as well. There are a few old-timers developing Common Lisp applications to run on modern hardware, using Symbolics hardware.

In fact, Symbolics shipped compilers for other non-Lisp programming languages, including C and Ada. These interoperated smoothly with Lisp code, much more so than they do under Unix. In this demo, Kalman Reti compiles a JPEG decoder written in C, and replaces the Lisp JPEG decoder that came with the system with it, yielding a performance boost:

https://www.youtube.com/watch?v=o4-YnLpLgtk

[chubot]

OK interesting, I will check out the link.

I still think various Lisps don't interoperate enough today, but I'm not very familiar with the Lisp machines of the past. If it can interoperate with C and Ada that's interesting. But I also wonder about interop with JavaScript :) i.e. not just existing languages but FUTURE languages.

These are the M x N problems and extensibility problems I'm talking about on the blog.

[fiddlerwoaroof]

If you’re fine with ES3, there’s https://marijnhaverbeke.nl/cl-javascript/ (I’ve been intending to use Babel to compile modern JS to ES3 and then load it into CL, but no time)

Or parenscript: https://parenscript.common-lisp.dev/

Or CLOG: https://github.com/rabbibotton/clog

And, there’s always the option of writing an HTTP or websocket server and serving JSON.

I personally use the SLIME repl in emacs for a ton of things I used to use bash one-liners for and you can push it pretty far (this is an experiment, I’ve wrapped it up more nicely into my .sbclrc since):

https://twitter.com/fwoaroof/status/1502881957012668417?s=21...

[bitwize]

I'm not sure what you mean by "The lowest common denominator between a Common Lisp, Clojure, and Racket program is a Bourne shell script (and eventually an Oil script)." You can get two Lisp programs (or two Python programs, or two mixed-language programs, etc.) to intercommunicate without involving the shell at all.

It'd be more accurate to say "The lowest common denominator between a Common Lisp, Clojure, and Racket program is sexpr notation." By using sexprs over a pipe, named pipe, or network socket, you can very easily get any of those Lisps to intercommunicate deeply structured data with any other. This is how SLIME and its SWANK protocol work. I don't even think the shell is involved; Emacs handles spawning the inferior Lisp and setting up the communication channel itself.

The thing the Lisp machines had was a very robust ABI. Lisp has a specific notion about what a function is in memory. This is largely because Lisp is always "on" on a Lisp machine, there is no point at which you cannot take advantage of the entire Lisp runtime, including the compiler, in your own programs. Accordingly the Lisp machine C compiler output Lisp functions containing code compiled from C, that could be called by Lisp functions directly (and vice versa). Presumably a JavaScript runtime for Lisp machines would be able to do the same thing.

By contrast, C has no notion of what a function is; and the C ABI used by many operating systems presents a function as simply a pointer in memory that gets called after some parameters are pushed to the stack or left in registers. (How many parameters, their type, and their size, is unspecified and simply agreed upon by caller and callee.) Nothing about memory management is provided by the runtime either. All that has to be provided by user programs. All this adds friction to interoperation by function call, and makes IPC the most straightforward way to interoperate.

But oh well. Our computers are all slowly turning into JavaScript machines anyway, so maybe those Lisp/Smalltalk happy days can return again soon.

[chubot]

What I mean is "coarse-grained composition with text/bytes" (described in the blog post in my first reply)

If you're thinking that's tautological (how else would Common Lisp and Clojure communicate?), then this subthread might help with the context:

https://news.ycombinator.com/item?id=30814716

I don't think Common Lisp, Clojure, and Racket are compatible with Emacs' s-expression format. Lots of people here are saying they use JSON or the like with Lisp, not s-expressions.

Emacs can use its own format to communicate with itself, because it controls what process is on the other side of the wire.

(In any case, any variety of s-expressions IS TEXT; there are non-trivial issues to get it back in memory -- like what to do about graphs and sharing, node cycles, integer sizes, string encodings, etc.)

But the point of the slogan is that when you have even a small bit of heterogeneity (among Lisps) then what you're left with is "Unix". A Lisp machine isn't a big win for this reason.

It is cool that Lisp machines had a robust ABI. That could solve the IPC problem. But then you have the problem of a Lisp machine communicating with a Unix machine, which I'm sure was a very real thing back in the day. So now you're left with the lowest common denominator of Unix. Again that is coarse-grained composition over wires, which is analogous to shell. The shell doesn't have to be involved strictly speaking, but the syscalls that are made and parsing that is done is pretty much the same.

[foobarian]

> In fact I conjecture that this is the reason Unix is more popular than Lisp -- because Lisps don't interoperate well. They haven't built up a big ecosystem of reusable code.

And why are there so many? IMO the language is too flexible for its own good. It promotes this curious intellectual solo-competition where you try to prove you are worth of the elite who get all the fancy FP stuff and make all this bespoke functionality.

It's almost impossible for Lisp to be popular. To be popular means a lot of people use it, but that means it can't be complicated much above the median. But because it lets individual programmers push intellectual boundaries it self-selects itself out of this pool. Any big collaborative project will attract this type of developer and soon the lesser developers (who don't dare object for fear of appearing dumb) are not able to contribute.

Just my opinion, if a little dramatic.

[zozbot234]

Haskell has its share of fancy FP stuff, and people manage to develop workable things in it. I still think Lisp really is too dynamic to be useful beyond a small scale of development.

[Jtsummers]

> Honest question: how do you communicate between two Lisp processes on two different machines? I know Clojure has EDN (which is sort of like JSON : JavaScript), but I haven't heard of the solutions for other Lisps.

Probably TCP or UDP based protocols like essentially every cross-network communication in every language today.

EDIT: Also, it should be noted that JSON does not, itself, allow you to communicate across a network. It's just a serialization format. You still have to select some protocol for actually doing the communication. If your answer to the question "How do you communicate between two JavaScript processes on two different machines?" is "JSON", you've failed at actually answering the question.

[chubot]

Right, so that is what I'm getting at. If you have two Lisp programs on different machines, or two programs written in different Lisp dialects, the way you compose them is basically "Unix" -- serialize to some common format and send over a pipe / socket.

[gmfawcett]

What alternatives are you possibly allowing for, if you're taking IP and wire protocols off of the table?

[chubot]

I agree there aren't many practical alternatives to serialize -> wire -> parse, because that's how all storage and networking hardware works.

But I keep having this conversation with people who are mad at Unix (and Unix shell) and searching for an alternative. Some of them are just fantasizing, as with this post, but some of them are actively working in that direction.

One possible alternative is if the kernel grew a mechanism to pass linked data structures directly, instead of requiring serialization and parsing (a little like Go channels perhaps). I suppose you can kind of do that with shared memory -- but not really because copying is roughly equivalent to serialization.

This might sound like an unrealistic idea, but people are seriously arguing for it. And of course they always argue in favor of their favorite language's data structures, not realizing it comes up with a boatload of design choices specific a problem domain, and that other problems require different data structures.

This post is arguing for something even more radical: an entire machine in Lisp, with a single address space (really bad idea).

----

The other more practical thing I'm arguing against in the post is a certain design of alternative Unix shell [1] that I've been seeing a lot.

The design is that basically that all the tools like 'ls' and 'rm' and even networking tools are expected to be statically linked into the shell binary.

And many of them pass structured data directly, specifically to get rid of serialization and parsing. There are at least 3 or 4 shells with this design.

My point is that this creates a "two-tiered" shell design and that composition is inhibited. In my mind the point of the shell, and the power of it, is to be situated within or "woven into" the OS. It interacts with everything on the system. It's not a "world" unto its own. You should be able to whip up a script in your favorite language and call it from your shell, and compose it with other tools you didn't write. You shouldn't be forced to "escape" the shell to call outside tools, or write algorithms in shell like you would write algorithms in Python or C. The shell is for coarse-grained composition and coordination.

[1] list here: https://github.com/oilshell/oil/wiki/Alternative-Shells

----

Another possible alternative is something like https://www.unison-lang.org/ where source code is not text; it's structured data. This has big consequences that I hinted at in the prior post, and have in my notes for upcoming posts:

They are writing their own version control system and editor because they don't use text. (I think experiments like this are interesting; it's definitely worth some portion of our efforts.) It is true that they have to use byte streams at the end of the day because they have to use storage and networking hardware, but the user is supposedly insulated from any of that. That is, as long as you stay in this hypothetical universe. Again I will bring up interop as a big problem.

A lot of people seem to imagine "monoglot" worlds where they wrote everything from scratch and no other code exists. Everything done in the past must be thrown out because we know better now. :)

Elm is a more widely used language with similar interop problems.

[armitron]

The canonical Lisps still widely used today are Common Lisp, Scheme and Emacs Lisp. They all belong in the same family, and syntax / semantics are close. Porting code from Scheme to Common Lisp can be a lot easier than going from Python 2 to Python 3.

Clojure is something else entirely which is why a lot of people don't consider it a Lisp.

> Honest question: how do you communicate between two Lisp processes on two different machines?

If you want to use built-in object serialization, there is print and read.

[iak8god]

> Common Lisp, Scheme and Emacs Lisp... all belong in the same family

Could you say more about what you mean by this? Is there another family of Lisps that excludes these three? I've met people who make a big deal about lisp-1 vs lisp-2 (https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2), and which is the right way to be a Lisp, but I think maybe those people just enjoy being pedantic.

[armitron]

I was referring to Clojure which going by syntax/semantics does not belong in the same family as Common Lisp, Scheme, Emacs Lisp.

[iak8god]

Makes sense. I think many people would put Scheme in its own branch as well.

E.g. https://gist.github.com/Aethaeryn/3036597

[zozbot234]

An unappreciated means of code reuse under *nix is the static and dynamic library. This seems to be the go-to whenever you need something more involved than simply reusing a full binary via pipes.

[scj]

C's greatest feature is that it trivially maps onto .so files. Linking to and creating .so files isn't just cheap, it's effectively free.

Most higher level languages I've worked with seem to focus on using .so files rather than producing them.

This means the lowest common denominator for the unix ecosystem is what C can provide. Otherwise stated, unix is marching forward at the pace of C.

[samatman]

This is one of the things I like so much about Lua, LuaJIT in particular: it's designed around being just another .so file, from another clib's perspective Lua is a library for manipulating a calling stack, and the LuaJIT ffi makes it short work to adapt a header file so that structures and functions can be used from within Lua.

[pjmlp]

No wonder, given that C was created to turn UNIX from an originally Assembly written OS into a portable one, UNIX is effectively a C Machine.

[fiddlerwoaroof]

> Honest question: how do you communicate between two Lisp processes on two different machines?

Depends what level of integration you want: custom tcp/udp protocols, HTTP and websockets are all pretty easy. But, you can also use something like this over a trusted network/vpn: https://github.com/brown/swank-client

[spacemanmatt]

> how do you communicate between two Lisp processes on two different machines?

Same as every other language: you pick a protocol and use it on both sides. Many of us already have enough JSON in play that it makes sense to start there.

[convolvatron]

> Honest question: how do you communicate between two Lisp processes on two different machines?

using continuations with sexps is insanely powerful

send (+ 1 3 (lambda (x) `(handle-response, x)))

[openfuture]

It's all going to be datalisp, mark my word :)

Although you have no idea what I am talking about yet just wait a bit more :))

[syrrim]

pretty sure hn was ported away from arc at some point.

[throw10920]

> Meh the problem is "Which Lisp?"

Not a problem - you don't need (or want) a single Lisp. A hypothetical Lisp OS would support a standard runtime and typed IPC system that host programs can use (a la Windows and COM, or dbus), and nothing prevents you from using your own custom communication protocol over pipes/sockets instead.

I don't agree with your implicit assertion that there is a single reason why Unix is more popular than Lisp (machines) - I think that there are a multitude of reasons. Certainly, the most impactful one isn't a lack of interoperability between Lisps - it would be something like the bandwagon effect, or the fact that Bell Labs gave away Unix to universities, or the inefficient implementations of early Lisps.

I'm also fairly confident that the lack of a Lisp ecosystem is not because of a particular lack of interoperability (after all, Python has a massive ecosystem that is built almost exclusively on (a) C FFI and (b) other Python code), but for cultural reasons - Lispers just don't like collaborating together, and enjoy reinventing the wheel. These tendencies have been documented in [1] and many other places, and are highly consistent with my own experience in the Common Lisp community.

> Honest question: how do you communicate between two Lisp processes on two different machines?

Use PRINT to serialize an object on one machine, ferry it over to another one through whatever means you wish, and then READ it on the other. Or: "in the same way that two processes in most other languages on two different machines communicate". Serialize, transport, deserialize. Yes, the ecosystem is far less mature, and you'll have to do more legwork, but fundamentally the process is the same as in, say, Python. (Erlang might be an exception and have language-lever support for this, I'm not sure)

This method works for different implementations of the same Lisp, and even for different Lisps, under some constraints (e.g. while a list with the integers 1, 2, and 3 is represented as the s-expression (1 2 3) in almost every Lisp you can find, CL represents "true" as T, while Scheme represents it as #true, so you'll have to work around that). If you will, you can just use JSON or XML as a serialization format instead - every non-trivial Lisp has libraries for those formats, and some have libraries for ASN.1, too.

>> The lowest common denominator between a Common Lisp, Clojure, and Racket program is a Bourne shell script (and eventually an Oil script).

All of those languages share basic s-expression syntax described above, which is rather higher-level than a Bourne shell script. Why do you say that the latter is the "lowest common denominator"?

For that matter, why do you exclude the idea that JSON or XML aren't "common denominators" between Lisp programs, or even between Lisps, Python, and C++?

----------------

Your article says "Text Is The Only Thing You Can Agree On", but "text" isn't even a single thing. The set of bytes allowed by ASCII vs UTF-8 vs UTF-16 aren't the same. Even if it was, plain text is purely sequential and flat. Program interoperability requires structure. If the structure isn't directly encoded in a given "substrate" (such as text), and you need another layer on top, then that substrate isn't actually the one providing the interoperability. You say "Text is the most structured format they all agree on" but text isn't even structured at all, and nobody completely agrees on it - "bytes" is the only thing that fits into place here (which is equivalent to saying that there's no structured communication method that all programs agree on, which is true).

Put another way - programs do not communicate using plain text. They communicate with either an ad-hoc protocol built on top of text (that still has a set of constraints that make it incompatible with other protocols built on top of text), or they use a standardized format like JSON or XML, or even something like ASN.1 that isn't a subset of text.

Communication using text does not make programs interoperable. Bytes might be a "narrow waist", but text is factually not - if it was, you wouldn't need to use sed/awk/perl one-liners to connect various Unix utilities to each other, because the very fact that they were all using text input/output would make them interoperable.

You say "Tables and documents are essential structures in software, and expressing them in JSON is awkward." but you can express them in JSON. You cannot express those structures in "plain text", because "plain text" is not capable of expressing structure at all, and the best you can do is build a protocol that is a subset of plain text that can express structure (JSON, XML, CSV, etc.)

------------

If anything, Lisps are more interoperable than Unix, because the "lowest common denominator" of Unix utilities is "plain text" (which by definition cannot encode structure), while the lowest common denominator of Lisps is some common subset of their s-expression formats, which can encode structure.

------------

> I would say the design of Unix is "rotting", but the answer is to IMPROVE Unix.

You say this, but you haven't given a reason for why that's the answer.

Here's a reason why improving Unix is not the answer: because "everything is text" is a fundamentally flawed paradigm that introduces needless complexity and fragility into the whole system design.

> Not dream of clean slate designs that will never be deployed.

You're mixing the normative and the positive - "Unix should be improved" with "clean-slate designs won't be deployed". Are you making an argument about what should happen, or what will happen? (there's no guarantee that any improvements to Unix will be deployed, either)

> Plus this post doesn't actually propose anything. If you actually start trying to build your Lisp machine, I believe you will run into dozens of reasons why it's not a good idea.

Oh, yes, and I can start with a few: first, there's good reason to believe that high-level CPUs are a bad idea (as suggested by the fact that nobody has been able to make an effective one[2]); second, the security-less design of older Lisp machines is a terrible idea (in a vacuum); third, that a machine that only runs Lisp is going to be unpopular and there's no single Lisp and Lisps are not the final stage of PL evolution.

...but the arguments made by the article as to why Unix is inadequate are still solid, even if the author is suggesting a suboptimal solution due to nostalgia.

[1] https://www.lambdassociates.org/blog/bipolar.htm [2] http://yosefk.com/blog/the-high-level-cpu-challenge.html

[chubot]

This feels like a whole bunch of misunderstandings about what I'm saying ... Almost everything here was directly addressed in the article.

For that matter, why do you exclude the idea that JSON or XML aren't "common denominators" between Lisp programs, or even between Lisps, Python, and C++?

JSON is mentioned in the post as A narrow waist, not THE narrow waist (of an operating system or of the Internet). I also mention CSV and HTML as "on the same level".

Likewise, bytes and text have a similar hierarchical relationship. I mention that in the post and also show some diagrams in the previous post.

If anything, Lisps are more interoperable than Unix, because the "lowest common denominator" of Unix utilities is "plain text" (which by definition cannot encode structure), while the lowest common denominator of Lisps is some common subset of their s-expression formats, which can encode structure.

JSON, CSV, and HTML are all built on top of plain text. You can store them in source control and you can use grep on them, and you can build more specialized tools for them (which has been done multile times.)

What I'm contrasting this with is people who say that we should build s-expressions into the kernel -- i.e. passing linked data structures directly, rather than bytes/text.

See the threads linked in the posts -- a variant of this same argument came up. This is very related to the idea of building Lisp into hardware, which I view as a bad idea.

You say this, but you haven't given a reason for why that's the answer.

The article is analyzing why empirically Unix, the web, and the Internet have worked in multiple respects -- interoperability, generality, scalability in multiple dimensions and multiple orders of magnitude, extensibility over decades, the polyglot nature, etc.

These are obviously successful systems, and I think it is pretty crazy to have the opinion that because it makes you parse things that a design that tries to eliminate parsing much be better along many/all dimensions!

These kinds of (IMO naive) arguments are exactly why I wrote the last 2 posts. They were popular and highly commented upon, and most people got it, or at least appreciated the tradeoffs I highlighted.

So I don't need to convince everybody -- as I said in the intro to the post, the more important task is to build something that embodies these ideas. I look forward to your code and blog posts; I do think there is something to the top criticism in the thread: https://news.ycombinator.com/item?id=30812626

[throw10920]

> This feels like a whole bunch of misunderstandings about what I'm saying ... Almost everything here was directly addressed in the article.

I would definitely appreciate you pointing out exactly where the misunderstandings are! (also, ironically, most of the points that you make here were already addressed in my comment, including refutations of some of the ideas in your articles)

> JSON is mentioned in the post as A narrow waist, not THE narrow waist (of an operating system or of the Internet). I also mention CSV and HTML as "on the same level".

I saw that JSON was described as a narrow waist; however, the specific context of my question was about the use of those technologies applied to Lisps - if you believed that those were narrow waists, then you wouldn't have written both of the following:

> The lowest common denominator between a Common Lisp, Clojure, and Racket program is a Bourne shell script (and eventually an Oil script).

> What Is a Narrow Waist? [...] Small, simple mechanisms like the Internet Protocol, UTF-8, and JSON.

That is, there's an inconsistency between two statements both made by you, not a misunderstanding on my part.

> JSON, CSV, and HTML are all built on top of plain text. You can store them in source control and you can use grep on them, and you can build more specialized tools for them (which has been done multile times.)

Yes, and as I said in my initial post, that's both incidental and irrelevant. You missed or ignored several paragraphs of my comment you're responding to (starting with "Your article says"). I'll re-state for the sake of concision: the fact that those technologies are built on top of plain text does not mean that plain text is a narrow waist! All of those formats require the same level of parsing effort as they would as a byte-oriented protocol that didn't adhere to the subset of "plain text". This was, again, already addressed in my comment.

> See the threads linked in the posts -- a variant of this same argument came up.

You say "the posts" but there are at least four different "zones" that you could be referring to (the other posts on the Lisp machine article, links in your blog posts, or the Reddit or HN posts on your blog posts), and dozens-to-hundreds of individual links/comments in each of those "zones". "the posts" means nothing - please either link to a specific web page (or part of it, if it's long), or re-state the argument in your own words, so I can see what your argument is.

> This is very related to the idea of building Lisp into hardware, which I view as a bad idea.

I'm not defending that idea - I also thing it's a bad idea! I'm arguing against the specific idea of "text" being a "narrow waist".

> The article is analyzing why empirically Unix, the web, and the Internet have worked in multiple respects

You say this, but I can find zero evidence in either of your posts to support the assertion that the success of those things is a direct result of their use of plain text as a narrow waist. Thing like HTML don't count, because again, the fact that HTML builds on top of plain text is incidental and irrelevant.

> These are obviously successful systems

...and there are dozens of reason why a system can be successful, many of them social/cultural and not technical. Why is Windows so popular? Gmail? C++? Perl? SQL? Java? x86? DVD/Bluray? JavaScript? Are you going to tell me that the popularity of all of these systems are mainly due to technical reasons, and not business, social, cultural, and incidental ones?

In the specific case of Java, I can tell you that the main reason for its success was because Sun bribed universities to teach their CS courses in Java in exchange for free Sun workstations, and then the students went out and spread Java around. This is a perfect counterexample to the extremely naive idea that "technical superiority leads to market dominance", and there are dozens more out there.

Actually, given that we're on HN, any successful startup founder will tell you that technical superiority will not make your company successful (and of course we can then translate that to success in the open-source world).

> I think it is pretty crazy to have the opinion that because it makes you parse things that a design that tries to eliminate parsing much be better along many/all dimensions!

Then I'm going to say that I think that it is pretty crazy to have the opinion that a system that requires you to completely unnecessarily add serialization and deserialization for virtually no benefit whatsoever (and many significant drawbacks) is somehow better overall than a system that...doesn't.

> These kinds of (IMO naive) arguments are exactly why I wrote the last 2 posts.

I'm coming up with concrete counterarguments to every single point that you concretely state. Calling them "naive" instead of addressing their content is...not an indicator of sound reasoning, to say the least.

> They were popular and highly commented upon, and most people got it, or at least appreciated the tradeoffs I highlighted.

And, of course, none of those things are indicators of sound reasoning, either - appeal to authority and popularity fallacies, and all that.

> So I don't need to convince everybody

Well, unless you make concrete and valid arguments, I won't be convinced - although you can always find people who are convinced without those...

> as I said in the intro to the post, the more important task is to build something that embodies these ideas.

I am building things that embody these ideas. They're likely never going to be open-source, though, so I'm not surprised if you don't believe me.

> I look forward to your code and blog posts

The idea that blog posts are somehow more authoritative or substantial than any kind of writing is...something, for sure.

Unless you're saying that a design paradigm isn't good unless it's implemented in software, in which case that's clearly false, given the number of completely terrible design paradigm that have been used to design "working" software.