Yes, that restart is painful exactly because it interrupts the "built your program in memory while you’re writing the source code file" OP talks about.
my inkling is that the more special forms the more difficult it is to repl a language. if everything is an expression not only can you eval parts of it, but that is enough to support quite a bit of extensibility sans macro.