[graphene]

Honest question, how is this different from what you can do in e.g. python? The python interpreter supports reloading of modules and evaluation of expressions, is there functionality that CL has above and beyond this that makes it more powerful?

[TeMPOraL]

Can you recompile a function on a running program and have it pick up the new definition? Can you change a class at runtime, and then adjust already existing objects to match the new definition in a controlled way, without data loss? Can you forget to define a function, or initialize a variable, only to be allowed by the debugger to provide a new definition / value on the fly and continue execution?

Those are some examples off the top of my head, but the general point is that unlike mainstream languages, Lisp languages evolved into environment for interactive construction and modification on a continuously running program, which is a bit different way of working than the regular write-compile-run cycle.

[aerique]

Some, most?, of the things you mention are available in other languages but they generally do not come together so well and they feel clunky.

I think it's not so much whether the features exist in other languages since we all know they can be added but the whole being more than the sum of its parts.

[didibus]

I find Clojure to be practical today, and it supports most of those. I believe it lags a little behind CLs development features, but since it tends to force immutable and referentially transparent code, I find that helps too. Since swapping things out is trivial in those cases.

[Arelius]

With Javascript as an example, it can indeed be used in this fashion, as I expect also Python. But the key difference is the community of CL is this is a standard way of working, and the tools are built around this.

[SomeHacker44]

I'm not much of a Pythonist, but I have been doing CL since before ANSI CL.

To me the thing that CL has that has yet to be matched by any other languages, REPL-based or not, static or not, is the amazing condition system and dynamic restarts for handling problems. For building and debugging systems this is an utterly amazing thing. It also allows composing exceptional conditions in a reasonable way.

Every other language I use (e.g., Clojure these days a lot) I constantly miss this amazing stack-walking ability and conditional restarts.

Every time I boot my old Symbolics computers and something happens and I get a GUI restart, I'm just amazed at what we have lost through two decades of non-CL living.

[elcritch]

The sound of it makes me think or Erlang's OTP system. I've just been really digging into it lately. Though the stack walking seems different, the entire OTP system is wonderful in letting you deal with errors by composing restartable processes and supervisors.

Heck, it's easier to handle recovering from an error by just restarting a process and letting it follow the normal unit logic.

Do have any knowledge of how CL and Erlang/Elixir/OTP are similar or vary on error handling?

[mikelevins]

Common Lisp's restarts are not restartable processes in the Erlang sense. They are part of a facility for suspending execution of a thread of control when it encounters an error, modifying the dynamic state of the process during the suspension, and resuming execution afterward.

When an error occurs in a Common Lisp program the runtime signals a __condition__. A condition is an instance of a class that represents unexpected or otherwise exceptional situations that can arise during execution. When a condition is signaled, control searches for a registered handler that matches the type of the signaled condition. If one is found then control passes to the matching handler. If none is found then control passes to the default handler, which by default starts an interactive repl called a breakloop__.

The breakloop has access to the full dynamic state of the suspended computation, including all local and dynamic variables and all pending functions on the stack. From the breakloop a programmer can alter variable values and redefine functions and classes. For example, if you conclude that the error happened because of an incorrect function definition, you can redefine the function and tell the breakloop to resume execution as if the new definition had been called instead of the original one.

A __restart__ in this context is an option for continuing execution. Common Lisp's condition system offers the programmer the ability to choose from among available restarts in a breakloop, or to write a handler that will make the choice automatically when a condition is signaled, and it also offers the ability to define custom restarts.

[didibus]

I've been wanting to experiment with this. There are some libraries in Clojure that try, but it seems that the real value is the interactivity of it while developing? I haven't found the exception handlers to really add much value over normal exception handling at least in Clojure.

[guicho271828]

As an interesting example, conditions and restarts can be used to implement an interactive system that is composable with other programs. For instance, you can implement a simple y/n question handler as follows:

    (defun fn1 ()
      (if (yes-or-no-p) (print :yes) (print :no)))
    
    (defun fn2 ()
      (fn1))

[yes-or-no-p](http://clhs.lisp.se/Body/f_y_or_n.htm) is an interactive function that reads from the stdin. However, you cannot programatically answer "yes" to fn1, as other functions in the call stack (fn2) has no way to know that fn1 halts because it waits for the input from the stdin. Instead, a condition system allows this:

    (defun fn1 ()
      (restart-case (error "yes or no?")
        (yes () (print :yes))
        (no  () (print :no))))
    
    (defun fn2-interactive ()
      (fn1))

    (defun fn2-automated ()
      (hander-bind ((error (lambda (c) (invoke-restart 'yes)))) ;; handler
         (fn1)))

When you call fn2-automated, an error is signaled, handled by the handler, which invokes a restart 'yes, then :yes is printed. Interactivity is still maintained by fn2-interactive.

[didibus]

I'm not sure I get it. Where is the user io in the second example? I'm not sure I understand how fn2-interactive still works.

[brudgers]

After fooling around with porting some Common Lisp code to Python as an exercise in reinforcing my understanding of the underlying algorithms and learning Python, I would describe the difference as the Python interpreter was mostly designed with the assumption that the typical use case would be executing the interpreter with a source file as an argument [e.g. python spam.py] and not primarily as an interactive development tool that is left as a running process for days or weeks. Instead Python ships with IDLE which more reflects the idea of an interpreter running source files serially.

Or to put it another way, trace is an option that is passed to the Python interpreter. In Common Lisp (trace ...) is a function that can be called in the REPL.

[BeetleB]

The standard Python interpreter is crap. One should use IPython. I'd like to see a comparison with that one.

[dangerbird2]

The standard repl for Steel Bank Common Lisp is worse than Cpython's with no basic readline support. The Common Lisp environment really shows its strengths when using a fully interactive development environment like a Lisp machine environment or Emacs with SLIME.

[omaranto]

I'd call it the "built-in repl" rather than the standard one, since I don't think it's actually standard for people to use it directly.

[lispm]

CLISP has readline support.

[patrickg_zill]

That is exactly correct. Guido van Rossum's original paper on a scripting language called ABC which strongly influenced his later design of Python, was basically a scripting language for system administration tasks on Unix and Unix-like operating systems.

[reikonomusha]

It's extremely different. You have to resort to hacks in Python to reload modules, which has unclear semantics. Python doesn't let you update function definitions while running, or let you add new polymorphic methods to dispatch on while developing. Not to mention interactive error handling isn't as powerful as Lisp's.

Python really has a glorified edit-compile-run cycle. The interpreter loop is nice for quick tests or calculations, but isn't too helpful for the incremental development of your program.

[sedachv]

> You have to resort to hacks in Python to reload modules, which has unclear semantics.

This is the crux of the issue. Even in Python 3 the semantics for reloading modules is wrong:

Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.

If a module imports objects from another module using from ... import ..., calling reload() for the other module does not redefine the objects imported from it — one way around this is to re-execute the from statement, another is to use import and qualified names (module.name) instead.

If a module instantiates instances of a class, reloading the module that defines the class does not affect the method definitions of the instances — they continue to use the old class definition. The same is true for derived classes.

https://docs.python.org/dev/library/importlib.html#importlib...

In principle Python has a runtime with type-tagged objects that you can inspect and code you can load, so it should be dynamic. Because of the way module and class loading behavior is defined you cannot actually take advantage of this dynamic runtime. Python does not really have a REPL, it has script loading akin to most Unix shells, and a second-class REPL that is useful for some limited debugging. This unnecessary dichotomy imposed on code loading makes it very different from Lisp and Smalltalk.

[mikelevins]

Common Lisp supports stopping execution of a function in the middle, redefining the function, changing the values of all accessible dynamic and lexical variables, adding new variable definitions, redefining the types of the values, and then resuming execution from the start of the redefined function, or from any other accessible control point.

It also supports writing condition handlers that can do all of the above under program control, rather than under human control. In other languages you would say "exception handlers", but Common Lisp conditions include representations of situations and responses that exceptions typically don't.

These facilities, along with things like a debugger, a tracer, support for built-in documentation, a compiler, and a disassembler, are part of the language standard, not the development environment.

Common Lisp is a language designed from the ground up for writing programs by interacting with them as they run. There aren't very many other languages like that. Smalltalk is one of them.

[guicho271828]

I've been writing in Common Lisp for 7 years and am working on a python project for ~8 month (for a machine learning task w/ Tensorflow). I am insanely irritated by countless points, to name a few:

* Python REPL does not check and warn the trivial type/arguments errors when a function is defined. It is only after the program reaches that erroneous line that an error is signaled. SBCL always warns about the function signature. Perhaps this may be customized by some global variable or a command line option, but not being warned by default is a huge waste of time, as it takes non-negligible amount of time to reach there (specifically because it is a ML task). (SBCL also warns me about type errors in most of the standard functions as well as ftype'd functions.)

* Python does not have restarts. I mean the condition system in Common Lisp. This again imposes a significant amount of slowdown on the debugging cycle as an error may be signaled only after a multi-hundreds-MB data file is loaded in the beginning of a function and then the erroneous line of code is run. In CL, when an error is signaled, the program enters the debugger and halts there, during which you can correct the program, redefine the function, then resume the code from one or two stacks above the current. In python an error always rewind the stack to the toplevel, so you have to rerun the whole function, loading that huge data file again in the middle of the stack. Don't say you are afraid of some side effects in the code, I follow "Mostly Functional Style" in CL.

* Python debugger shows the stack frame, but does not by default show the values of the arguments in the function call stack when an error is signaled. It forces you to use an ugly printf debugging style. I would like to know if it is possible to alter the way the stack is printed. In CL, a debugger is just a function stored in debugger-hook. I hope you can do something similar.

* Somehow Emacs C-M-f selects the whole line, not an expression.

[weavie]

With CL if you have a bug on your web server you can just connect a repl into the running server, inspect the state of the system, modify that state and update code on the fly if necessary.

[DigitalJack]

I guess the key here for me would be inspecting the state of the system. I tried learning common lisp a few times and it never clicked.

When clojure came around, I tried lisping again, and for whatever reason i was ready for it, and clojure is my goto language.

That said, I'd never connect to a repl on a server and do any kind of bug fixing unless it was really dire. It's too easy to get the run state out of sync with the text source files, and I don't see why this would be different with common lisp.

I think the repl is great for development and figuring things out, but the whole idea of "program as state" seems antithetical to the ideas of functional programming. With the main issue for me being that I have to somehow keep track of this program state in my head once I start tinkering with it.

For example, say I find a function has a bug and I go into the repl on the server and fix it, in situ. Say I spot some other issues, or had them on my list of things to fix when I had time... I start modifying functions here or there. Now my "program state" no longer matches my source code, and I have to remember to fix it so that it matches. That's error prone. I could enforce my own methodology of fixing the source and copy-paste to the repl or send it in whatever way, still error prone.

I have to remember what changes I made, to what functions, and if I iterated, now I have to remember which iteration was final. Multiply that by any and all functions that are touched. It's a lot to keep track of.

So when you say "inspect the state," I have to think common lisp must have some feature for state inspection that is missing in clojure. I know there are a lot of features of common lisp, especially in stack-tracing, that clojure has no parity to. But I'm curious if common lisp has some way of completing a "round trip edit" such that an edit/correction made on a server process, in situ, are able to be easily propagated back to your master source?

If you read all that, thanks :)

[sedachv]

> But I'm curious if common lisp has some way of completing a "round trip edit" such that an edit/correction made on a server process, in situ, are able to be easily propagated back to your master source?

In SLIME you just connect to the Lisp process, open up your versioned-controlled file, edit it, hit C-c C-c for slime-compile-defun or C-c C-k for slime-compile-and-load-file. Isn't that how CIDER works also?

[weavie]

True. If you are going purely functional I guess you wouldn't have much state to inspect. Plus in a stateless web server you generally want to avoid keeping state around on the server.

I remember one time, it was actually with Clojure, I had to deal with Paypal. Since paypal only sent results back to the server and not to localhost I couldn't debug the problem locally. So I connected to the server, bunged all the debug information into an atom and made a few requests. Then I could inspect the atom, make some changes and find out what the problem was and fix it right away. It took just a few minutes.

At about the same time I was also working on a C# server. There was a bug there that could only reproduce itself on the server. To fix that bug I had to put a load of logging into the source. Check the file into source control. Wait for the CI system to build and deploy. Then kick off some requests. Inspect the log files. Make some changes. Check in.. wait.. etc.. It took days to fix that bug.

Admittedly the Clojure project was mine on my own server and the C# project was on some corporate enterprise behemoth so there were other factors that hindered that project, but it did highlight to me how much quicker you can get stuff done when you have that instant feedback.

When I say inspect the state, generally I just mean inspecting the values of global variables. In CL it is possible to put breakpoints in the source and inspect all the variables up the stack. Mind you, this can be a bit hairy on the server if there are multiple people connected as there is no guarantee as to whose Slime connection will pick up the breakpoint.

The round trip edit is pretty similar to Clojure. You would generally have your source file open in Emacs, change the source code and press Ctrl+CC with the cursor in your function definition to apply your edits to the server. Then you save the file and check it in to source control. You don't have to type directly into the repl.

You don't have to work like this if you don't want to. Without mentioning any names, it has been known for the server image to be completely out of sync with the state of the source files to the point where there weren't really any source files. The work around was to just treat the server image as the master source of truth. Luckily in CL you can just dump the image to create a new executable with its current state. So for several years this one server existed only as a constantly updated binary image. I wouldn't recommend working like that, but it has been done.

Depending on your implementation - I know Clozure CL at least allows this - you can extract the source code out of the running image. So it is certainly possible to propogate back to master source. It was eventually how this one server was restored to a more orthodox source code tree..

[DigitalJack]

Thanks for the reply. It seems a bit fragile and convoluted to me, but I suppose if you are used to it and are comfortable with the pitfalls it would be fine.

[weavie]

Yes, working directly on the server is definately fragile and fraught with all kinds of dangers. I try to avoid it as much as possible, but it is very handy having the option there as a last resort.

[j_s]

C# edit-and-continue sounds like a pale shadow of what's possible, and it's usually only useful locally!

[zeveb]

If you wanted, you could have any bug in a running web server pop up a window in your editor, fix the bug within the call that's buggy, then resume execution from that point at the call stack, successfully returning results to the client. Doing so doesn't necessarily make a lot of sense, but it's a very powerful technique if you choose to use it.

[SomeHacker44]

This is true and awesome.

To be fair, though, I run an nREPL TCP listener on my production servers too, so I can do the same thing with Clojure. :)

[kbp]

Not quite. In Lisp, when you get an error, the system pauses and lets you resolve the issue before resuming the program. Sometimes this even means typing in a function really quickly before a request times out (which isn't such an insane suggestion in the context of development). In Clojure, errors like that unwind the stack and tear everything down.

[vram22]

Reminds me of Paul Graham's essay about how at Viaweb, they sometimes used to fix bugs (in the Lisp code) while the customers who reported them were still on the phone, and then the support staff would tell them to check whether they were sure it was a bug, by trying again - and since the bug was now fixed in the server, when they did that operation again, it would work.

I think it was in his Beating the Averages essay - the one in which he talks a lot about the advantages Lisp gave them at Viaweb, over the competition.

[krapp]

You could do that with a PHP site if you're working directly in production. That's not necessarily an advantage of Lisp.

[vram22]

I think I had not described it fully (since had read it long ago). Searched for the article just now. The Beating the Averages article [1] does not have the point I mentioned. But a more detailed article [2] linked from it, does:

[1] http://paulgraham.com/avg.html

[2] More Technical Details: http://paulgraham.com/lwba.html (note: redirects twice)

Here's the relevant section:

[ Interactive Toplevel

Lisp's interactive toplevel is a great help in developing software rapidly. But the biggest advantage for us was probably in finding bugs. As I mentioned before, with Web-based applications you have the users' data on your servers and can usually reproduce bugs.

When one of the customer support people came to me with a report of a bug in the editor, I would load the code into the Lisp interpreter and log into the user's account. If I was able to reproduce the bug I'd get an actual break loop, telling me exactly what was going wrong. Often I could fix the code and release a fix right away. And when I say right away, I mean while the user was still on the phone.

Such fast turnaround on bug fixes put us into an impossibly tempting position. If we could catch and fix a bug while the user was still on the phone, it was very tempting for us to give the user the impression that they were imagining it. And so we sometimes (to their delight) had the customer support people tell the user to just try logging in again and see if they still had the problem. And of course when the user logged back in they'd get the newly released version of the software with the bug fixed, and everything would work fine. I realize this was a bit sneaky of us, but it was also a lot of fun. ]

[vram22]

Good point, see my other comment though.