As a fallback. I like LISPs as much as the next guy/gal, but editing the AST by default is too low-level. The importance of paredit is a language smell in my opinion, not something to aspire to.
Can you explain in which direction something better should go? The AST is very much at a higher level than how the vast majority of programming is done, which is by manipulating the text that later gets parsed into an AST. I'm having trouble envisioning what editing a higher level than the AST would look like, without jumping straight to visual programming, which I find very inefficient in every incarnation I've seen. (And we could debate how far above an AST such systems actually are.)
It depends on the AST, but most languages' ASTs are quite unwieldy to work with directly. They get marked up with all kinds of semantic information useful for a compiler or interpreter quite early in the parsing process, and are often more explicit and regularized than the syntax is. For example, you can dump a Python AST with the 'ast' module, but it's not pretty. The statement "x=5" balloons into:
I think a comment up this thread (https://news.ycombinator.com/item?id=16386702) is right that Lisp manages to make this work because the s-expression structure of Lisp source code isn't really an AST, though it has some relationship to one.
That Python doesn't look like abstract syntax; it's an intermediate form full of additional semantic objects geared toward further translation or interpretation.
A Lisp compiler might build up something like that, from a fully macro-expanded body of code. That's not usually accessible to programs.
Like the fact that n is converted to a Name() node, where it has a ctx property indicating Store() has nothing to do with syntax; it is the result of an analysis arising from how x is being used semantically. The user didn't specify any such attributes on the x just that x is assigned and that is already recorded by something like Assign(x, 5).
That seems backwards to me - AST-aware editing is basically embedding more of the compiler into the editor. It seems more valuable to me to be writing in a language that can express higher level concepts with lesser boilerplate, which would eliminate the need (and make far more difficult) for AST aware editing.
You might be surprised how little a Lisp dialect brings to the table specifically for developing the text editor features geared toward editing specifically Lisp.
The data structure representing Lisp data (i.e. code) is stripped of relevant text editing attributes, like how it is divided into lines and using what indentation, and, oh those semicolon-delimited comments.
Text editors also have to let the user deal with bad syntax.
So it's not just a matter of, "oh, we're written in Lisp, so just hand the buffer to the Lisp reader, do something and spit it out again".
I'm not sure what that is even supposed to mean. Having high quality tooling is completely independent from having a programming language with less boilerplate code. You can have both at the same time. So why are you advocating against it?
Take Java for instance.
In Eclipse the a lot of AST-aware editing features are hidden in the "quickfixes" which you can trigger with CTRL+1.
For example you want to use an ArrayList but you haven't imported it yet. Use quickfixes: Bang the import was created with just two keypresses.
In a raw text editor you have to remember the full package string or google it but when you want to look up what a "XYZ" it's rather easy to find it.
The java import system is verbose but it makes it obvious what you're importing.
one class = one file
Want to know what X is? Just open X.java.
Compare it to something like C/C++ where you include a header that contains multiple definitions. I honestly cannot use C++ without an IDE because of this. At the bare minimum my editor needs to be able to jump from a function call to the function definition or variable declaration to the struct definition.
In this case a feature that reduces boilerplate actually made the language harder to use without AST aware navigation.
Then there are quality of life things like extracting parts of a big function into seperate smaller functions. Just select the code you want and the IDE is going to create a function definition based on it and replace your selection with a call to that function.
I don't see how you could avoid this by making a more expressive language.
Does your fancy programming language automatically organise your code into functions or data types and all you need to do is crank out code without a care? No, you still to need think about that yourself, but the IDE can automatically move the code around for you.
Also let's talk about my favourite feature that I would never use without the assistance of an IDE: typeinference.
value blah = foo.bar([1,2,3])
What the hell is a blah? With an IDE I could just hover over blah and instantly know what the function returns. Without that feature I have to go to the definition of bar which is still easier with an IDE. After all if foo is also defined via type inference I have to lookup several function definitions and that's going to take some time. On top of that a tool like grep is imprecise and will give me hundreds of call sites but only one definition.
You can't fix everything by making a programming language more expressive and in my experience the dumber a programming language is, the easier it is to use with a dumb text editor.