| Thanks for sharing your valuable experience! Yes I have the same feeling (for now), that at many times I have to implement things from scratch using Common Lisp, and that pushes me to understand both the essence of the algorithm and how computation works. As I am still in the early stage of learning Lisp and just switched to CS, I am learning many mind-blowing things at the same time (SBCL+Emacs+Slime), this combination sum up to create a quite steep learning curve, and yes they are just time black holes, I have invested more time than expected every day in them and sometimes have to use Python to get the job done first to make my advisor happy, and then try rewrite in Lisp in leisure time. Maybe it can be good for me to learn a bit of Lisp every day, because I don't expect to understand all the ideas in the SICP book and remember all rules in the Practical Common Lisp book in 21 days. As I am doing some research, Python is indeed effective in exploring data/testing preliminary models. But most recently as I just experienced, it gets unbearably slow sometimes. (By unbearably I mean >15 minutes on PC.) Performance is also important for me, not only for the sake of "big data", but also for self-fulfillment: feels much better if the code gives result in 10 secs rather than several days. (By the way, does CL have mature support in parallelization?) My lab mates also use Mathematica a lot and it is even more convenient in testing models for data mining tasks, but I have been warned that some packages are buggy at this moment, and most modules are just black boxes that don't allow fine-grained tuning. I also tried R and at most times don't quite understand the logic of the language. Maybe as you pointed out, it is the gene of the language that determines its shape. Thanks again for your response, it is good to see someone doing real stuff with Lisp for data mining tasks nowadays. |
Be aware that SICP is written for Scheme. They're both lisp dialects, but they are different. I've done bits of it here and there in Common Lisp, but the two variable namespaces, no first class continuations, different function names, less emphasis on recursion (due to specific iteration forms/macros in Common lisp) and native efficient data structures offered by common lisp might make some things non-applicable or difficult even if you're already familiar with the language. Tail call optimization is, i believe, not required in the Common Lisp standard, but I believe SBCL performs it, at least above certain low optimization declarations.
I've only just started looking at multi-threading in Common Lisp myself. I imagine the proper answer about "mature" parellelization is no, relative to some other languages (i have no real basis for this, just side-knowledge that other languages focus on such issues more. Hell, its probably better than R.). Multi-threading is not established in the language specification, but in practice, implementations have their own version of it, including SBCL, which is the version I'm sticking to for numerical work. If you're looking into it, I believe you might also get acquainted with Bordeaux Threads. Once you've got quicklisp installed, you can install that easily from there :P
That being said, when you're performing something 10-500 times faster than the interpreted memory-hungry mess that is R for example, the complication of multi-threading can fall down the priority list...
Totally independent of my decision to try to the language, i later found this paper,which suggests that one of the authors of R is well aware of and in agreement with some of my own observations, plus there's a benchmark in there of common lisp, R, and python, though take all such microbenchmarks with a grain of salt, because you will be spending more programmer time in Common Lisp:
https://www.stat.auckland.ac.nz/~ihaka/downloads/Compstat-20...
Unlike some language zealots, i feel its my responsibility to remind you this is the road less traveled. I don't think its the way industry/academia is moving. I think there is, perhaps arrogantly, a selection bias in who becomes a Common Lisp programmer, such that inquisitive/smart/alternatively-thinking people tend to be attracted to the language, but if you think it will magically makes you 100 times better than you already are because some other smart guy uses it, put that idea right out of your head.
But also honestly, I've found the experience extremely rewarding, and I code my own projects in Common lisp now where I can.
I came in the top 20 of http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extr... by literally coding something up from scratch in common lisp, and my model had easy room for improvement and was among the fastest, so you can most certainly compete writing in lisp.
Now, aside from my eternal side-project, I'm trying to put something together in the field of data linking in Common Lisp. I think its one of the only options that will let me combine the flexibility of dynamic languages with the performance of C...(i'm actually hoping to perform better than the US Census Bureau C program)...I already know it blows the current python/R options out of the water, but time will tell whether I can match it with C performance-wise...