Could a language with S-expressions, macros, first-class functions, and a REPL cater to average programmers and be despised by hackers? How could you cripple these things?
There are languages that have useful features that lisp lacks. For example, Haskell has lazy evaluation and automatic compiler generated program correctness proofs (a decent type system).
Then there are languages that try to be even more powerful than Haskell, for example Epigram, which has dependent types.
Lazy evaluation is probably the language feature that is most mainstream that lisp lacks. For those not familiar with the abstraction benefits of lazy evaluation, consider a library for dealing with prime numbers. In a language without lazy evaluation, you are going to need an API with a function for getting the n_th prime number, a function for testing if a number is a prime number, a function for getting the smallest prime number larger than the number x, and a bunch of other functions.
In Haskell, the API only needs a single variable, called "primes" that is simply the infinite list of all the prime numbers:
primes :: [Integer] -- primes is of type list of Integer
Thanks to lazy evaluation, this single variable is all that is needed to support all of the above operations in an efficient way. For example, to get the n_th prime number you simply write (primes !! n)
Now, this may look cool, but what does it have to do with abstraction? Since "primes" is a regular list, I can use it with any list function and build more complex things out of it. I can also have a list of all the catalan numbers in the same way, and all my functions that I've written that do cool things with the prime numbers can now be instantly used against catalan numbers.
You can manually do lazy evaluation in languages that don't natively have it, but unless you do it everywhere you won't get the abstraction benefits, and if you do do it everywhere then the compiler almost certainly will not be able to optimize it as well as a native implementation.
Lazy evaluation is neat, but I can see it going into a lisp without really "breaking" the lisp.
What I see breaking in a lisp are the language features that come from limitations in the language. There's two ways to look at a language's power: What it enables you to do directly, and what it enables you to do by taking things away (in the form of constraints that are enforced on your code) and what it builds on top of that.
Looked at the second way, there are powerful things that can be built in some languages that Lisp is actually less-well suited for, because it is practically and philosophically opposed to constraints. Haskell's type system is actually the thing I see that would be the hardest to implement in Lisp. Haskell's type system allows the type system to say not just what a function returns, but in many ways, exactly what the function does and how it does it. If you have a function of type Int -> Int, then you know, with only a shadow of a doubt (unsafePerformIO), that the function does no IO, or, in fact, anything else except somehow manipulate an Int. (And as the doubt's name implies, you are taking your fate into your hands if you use that.)
Upon this foundation of constraints, Haskell can build in some features that Lisps can not, such as its safe implementation of STM. It's not that Lisp couldn't have something like a STM system, it's that it lacks the ability to make and use such strong guarantees about what is in an STM transaction that Haskell could, and I think you end up seeing a "reversion to the mean" effect where if the constraints are not enforced, they end up violated both accidentally and deliberately.
Similarly, almost every cute trick the Haskell type system allows, while certainly abstractly doable in Lisp, is not enforcable and therefore abstractions further layered on top of that are correspondingly less safe.
It is a viable opinion to say that you trust your programmer and that you feel that you should not care about such things; I'm not actually advocating these features in this post. (I'm still ambivalent myself, still gathering the data to have an opinion.) It's just that unless I'm very much mistaken, it's effectively impossible to get guarantees about the properties of closures passed in to your code in Lisp in the way that Haskell does. (You can examine the code before executing it, but along with the general tediousness of trying to prove these properties, I wouldn't be surprised you run afoul of Rice's theorem. It certainly seems unlikely it could be practical.)
When features are built on doing powerful things in arbitrary combinations, Lisp is one of the go-to languages. When you have features that have to be built on restrictions in the language (and not trivial ones, but tricky ones), Lisp has a problem. (Again, unless I'm very much mistaken.)
I don't think Haskell "blubs" Lisp, but I think it makes a very good case that the ordering of language power doesn't have a single apex in Lisp. Lisp may dominate in the "more power" arena, but that's not the only arena around.
There are (at least) two good points here: language power is a partial ordering not a total one, and Hindley-Milner type systems are probably the most prominent candidate for a purely language-level construct that Lisp wouldn't naturally extend to.
I occasionally think of adding lazy evaluation to Arc. I could make news.arc about 5 lines shorter if I had it. It could be that there are other types of code where it would make a more significant difference, though.
The biggest benefit of having it would probably turn out to be being able to say Arc has it in conversations like this. That might be a bigger deal than it seems, although I feel dirty suggesting the "nitpick preemption" philosophy of language design.
This is a significant psychological point that comes up in a lot of different contexts (sales contexts, mostly). But in the programming language context, I think it's a symptom of the dysfunction that the vast majority of discussions are at a superficial level (code snippets and theoretical features) and have little to do with building real systems over time.
That's true, although before we get too smug about those "out of touch with the realities of building software systems" academics, we should remember that often those seemingly absurd and fanciful little corners turn out to be serious game changers. Lisp itself, for instance.
Clojure has lazy versions of most of the collections API (or possibly it defaults to laziness now, it has been a little while since I last played with it). They're implemented using macros of course, but they get you surprisingly far. I have a bit of a Haskell background, and was doing the Project Euler problems in Clojure frequently using the lazy stream processing idiom you just described.
I think you need laziness to be a little more prevalent than that before it makes a difference. Haskell's super-concise Fibonacci sequence illustrates that:
How could you cripple such things? Well, first, culture. The primary difference between Ruby and Python is culture; the languages enable virtually identical styles of programming in practice (spelled slightly differently, but basically the same), but the cultures encourage different practices, such as how they feel about monkeypatching. Neither of these cultures "cripples" the language, I just use this as an example of the power of culture. You could cripple a powerful language with some sort of powerful pedagogical culture that made the powerful idioms verboten (because they're "too complicated", "unmaintainable", etc.).
A good language should make the right thing easier than the wrong thing. You could cripple a language that has all those bullet-point features by making putting hoops to jump through in the way of using macros or first-class functions. A klunky syntax, some sort of extra typing information to be manually added at every macro invocation, extra-verbose S-expressions, etc.
A REPL could be crippled by making it less than a full REPL, such that there are things that you still have to build modules for. See Erlang's REPL, which is mostly nice, except you can't define new records or do a handful of other useful things.
The most likely way this could happen is a language that tries to be LISP while still looking as much like C(++/#) as possible, and bringing over impedance-mismatched concepts better left in C(++/#). The second-most likely would be in some way constraining the power so as not to scare programmers or so as to avoid some "trap"; for instance, see Java's dropping of multiple inheritance. Thus, even though Java has "OO", it is less powerful than a Java that had MI too. You might have a "first class function" that is somehow limited to be less useful. (Perhaps you get first-class functions, but themselves are not allowed to return functions, only "values".)
Do not underestimate the power of language implementors to cripple a language, both intentionally and otherwise.
Could a language ... cater to average programmers and be despised by hackers?
Who a language "caters to", who an "average" programmer is, and who a "hacker" is are all social constructs which have no relationship to the objective reality of what features are in a language. People who wish to assert their superiority are quite willing to do so regardless of the technical merits of the matter.
Thus, rather than wasting one's time with meaningless geek-on-geek pissing matches, you should probably just get back to writing software which solves problems for people. You can do that in most languages -- even in Lisp.
"Blub" is just a derogatory word for "you only know one programming language". Become fluent in a bunch of them, and then you have the information you need to make your own conclusion.
As you write more and more of your own software, you'll see what features make that easy for you. You'll also see what features you can live without. You might even come up with ideas for new language features, and you'll see the value of being able to implement new language features without writing a new language. Very meta, but everything is related...
This question would be the length of a book if I had to define those words. Any hostility you detected was imagined. I was only pondering if a language in a certain style could be overly restrictive.
I've been thinking about this again because of the recent re-post of one of pg's Lisp essays (one of the two that originally provoked me into learning Lisp). The essay makes an argument that the answer to your question is roughly "No". The argument goes like this: no language that lacks Lisp-style macros can be as powerful as Lisp. Macros aren't possible without code=data, and code=data comes from sexps. But any language that represents programs as sexps is a variant of Lisp. So the only language that can be as powerful as Lisp is another Lisp.
Now obviously we can argue about how to define "powerful", and the whole discussion can easily become another pointless language flamewar. But let's not do that. Let's provisionally grant the essay its definition of "powerful" and the corollary that Lisp macros are currently the apex of that power. The question is, is a more powerful (in this sense) language possible? (You don't have to agree with that definition of "powerful" to find the question interesting, by the way. Just rephrase it as, could another language beat Lisp at its own game?)
In my mind I've always referred to the above argument rather pompously (i.e. half-jokingly) as "Graham's Thesis" (because it reminds me of what used to be called Church's Thesis when I studied logic - it's a non-provable-because-non-formal assertion that expresses an intuition about something - in that case computability, in this case programming languages). We can state Graham's Thesis as: Any programming language that's as powerful as Lisp is isomorphic to Lisp.
So, is this true? The critical thing is code=data. Is there a fundamentally non-Lispy way to represent code as data, that's as good or better for programming than sexps? (That qualifier is important, because there are definitely ways to represent code as data that make a lousy notation for programming, e.g. machine language.) If there is such a representation, then a language based on exposing it as notation could be as powerful as Lisp without being Lisp. But if there isn't, then variations of sexps are the only game in town, and those don't count as new languages. Replacing parentheses with different brackets, as was suggested here either trollingly or stupidly the other day, doesn't cut it.
As anyone who's seen a Lisp program and knows the first thing about a compiler knows, sexps are just the simplest notation for a syntax tree. That is, the thing that parsers turn programs in other languages into, Lisp programs just are. That makes sense, because parsers turn source code into data, and Lisp programs just are data. And any language in which you write programs as parse trees is Lisp (or will soon become Lisp as people add the obvious things you'd want in such a language).
So Graham's Thesis (man I feel silly writing that) reduces to the following: the only representation of programs suitable as both a data structure and a notation for human programmers is the syntax tree.
I'd really like to know if this is true. (Apologies to anyone who read my comment on this here the other day, as I'm repeating myself.) What languages are there whose source code gets turned into something that is fundamentally not a syntax tree? And what would the simplest explicit notation for that structure look like? I think this would be the area in which to look for an answer to the question.
There's one thing that makes me think such a representation might not exist: sexps are really just function composition, and function composition is how humans have done math for a long time. But if one does exist, I'd like to see it. There are a lot of people here with much wider backgrounds in programming languages than mine, so perhaps someone can just answer this.
Does'nt smalltalk have code=data behavior without s-expressions. I'm not an expert, but I thought that was the reason why Smalltalk has the IDE that everyone raves about along with image persistence
Or can Smalltalk be modelled as an s-exp based language?
I don't think Smalltalk has code=data. It has object-based metaprogramming, i.e. the code that you write turns into objects that you can write other code to manipulate at runtime. But perhaps that's a distinction without a difference? I don't have enough experience with Smalltalk to say.
I was expecting someone to suggest that stack-based languages are fundamentally different from tree-based ones.
Any language could be "blub". You're talking about a power continuum. If something more flexible than lisp gets discovered, then lisp could ostensibly be less powerful than "New Language X" for the developer.
Clojure risks heading that way; for all its beauty, clojure is losing its culture fast! You can already see Design Patterns being shoehorned on top of it, Java programmers will embrace it and extend it in earnest.
The sort of applications being written with the language are a huge factor in making it attractive to other users. All the truly beautiful languages had operating systems or huge desktop applications written in them; you used the language to extend something already powerful. It rewards your programming. Clojure will most likely become a server-side programming language, with little user interaction.
for all its beauty, clojure is losing its culture fast! You can already see Design Patterns being shoehorned on top of it, Java programmers will embrace it and extend it in earnest.
A neat hack to get something working quickly, but very unlispy. I can't even make sense of the AUTO-PROXY macro. A syntax barf mixed with a gratuitous breaking of LET. Also note that Rich Hickey decided to use the proxy design pattern to interop with Java, instead of doing all business with Java through FFIs; in a sense, Clojure's type system is embedded in Java.
This is a minor aesthetic nitpick from a concerned Lisper, mostly for selfish reasons. My thinking goes: "Today I have Common Lisp for my projects and I am happy with it. But tomorrow if I need Clojure, I hope to find it in a sane, Lispy world. I don't want to learn Java so please don't make me" ;-)
For a far more alarmist polemic, albeit a satirical one, see this:
I don't think you understand Clojure as well as you think you do. Java libraries often expect interfaces to be passed in as a parameter, and Clojure proxies are just a way of generating anonymous interfaces. This is not the de facto method for Java interop, it's just a simple means of creating the IFactoryFactory behemoths that Java tends to expect. It's practical, clean, and useful. I honestly don't understand what the nature of your objection is.
The blub argument is about not understanding that there are tools out there that do things that you can't currently imagine due to your immersion in your current toolset or a very narrow world view.
Then there are languages that try to be even more powerful than Haskell, for example Epigram, which has dependent types.
Lazy evaluation is probably the language feature that is most mainstream that lisp lacks. For those not familiar with the abstraction benefits of lazy evaluation, consider a library for dealing with prime numbers. In a language without lazy evaluation, you are going to need an API with a function for getting the n_th prime number, a function for testing if a number is a prime number, a function for getting the smallest prime number larger than the number x, and a bunch of other functions.
In Haskell, the API only needs a single variable, called "primes" that is simply the infinite list of all the prime numbers:
primes :: [Integer] -- primes is of type list of Integer
Thanks to lazy evaluation, this single variable is all that is needed to support all of the above operations in an efficient way. For example, to get the n_th prime number you simply write (primes !! n)
Now, this may look cool, but what does it have to do with abstraction? Since "primes" is a regular list, I can use it with any list function and build more complex things out of it. I can also have a list of all the catalan numbers in the same way, and all my functions that I've written that do cool things with the prime numbers can now be instantly used against catalan numbers.
You can manually do lazy evaluation in languages that don't natively have it, but unless you do it everywhere you won't get the abstraction benefits, and if you do do it everywhere then the compiler almost certainly will not be able to optimize it as well as a native implementation.