[ICWiener]

> but a quick search didn't reveal a syntax for hash table literals, which I would consider to be a prerequisite for calling them first class citizens

http://en.wikipedia.org/wiki/First-class_data_type

Hash-tables are part of the standard, and available in conforming implementations out-of-the-box. (make-hash-table) produces an instance of a table which can be passed as argument to function, be assigned to variables, ... I don't think syntax defines what feature is or isn't first-class, even though it has an impact on it's usage.

Adding support for litteral hash-tables is possible, as it was already said: define a reader macro; that macro could even simply use the utility function "plist-hash-table", from Alexandria, which is used like this:

     (alexandria:plist-hash-table '(:a 1 :b 2))

So that you could write #H{:a 1 :b 2} and get what you want.

On the first hand, people say that CL is bloated, but on the other hand, they want to have everything available by default. Do you expect Python or C++ implementations to come with regular expression built-in, or is it okay if you need to add an 'import' statement or link with Boost?

CL allows you to define libraries that can change even the basic surface syntax, for your convenience. Considering that there is more than enough defined in the specification, what benefits would be in having an "official" hash litteral syntax? consistency accross different projects? many people use the "cl-ppcre" regex library without problem.

Yes, regular expressions come in a library even though they could easily be labelled as a fundamental feature of a language, and hence be expected to be "first-class", or "built-in"; but again, that is not the case in CL, which is neither Perl nor Lua.

And about keyword parameters, this is just the surface syntax, which is on purpose based on a simple, basic representation of the syntax tree as lists of expressions.

Then, the compiler will work with the argument list in your function definition and take one or another approach to represent them in the object code. And I doubt that they are stored in a hash-table, which is unlikely an appropriate way to handle a couple of keyword arguments.

Similarly, when you define a class, everything is declared by a simple list of slots, where slots are lists of parameters: but in fact, the class might store the actual instance slots in a hash-table, an array, a list, a database or a memory mapped file.

The fact that you are mainly manipulating lists does not mean that everything is eventually represented as supposedly inefficient linked-list at runtime.

[ANTSANTS]

> http://en.wikipedia.org/wiki/First-class_data_type

Fair enough, pretend I said "first class syntactic object."

> CL allows you to define libraries that can change even the basic surface syntax, for your convenience. Considering that there is more than enough defined in the specification, what benefits would be in having an "official" hash litteral syntax?

Reader macros are powerful, no doubt. The benefit of building this into a language would mainly be syntactic regularity. CL as it stands has keyword args, alists, property lists, and real hash tables (probably amongst other things I don't know about), that all fill basically the same purpose of mapping names to values. If the creators of Common Lisp and its parent lisps had started with hash tables with standard reader syntax, they probably wouldn't have felt the need to introduce so many subtly different but conceptually identical concepts. Lua, for instance, uses its single table data structure both as syntax and as an internal representation for pretty much any case where it needs to map some names to some values; from keyword arguments to environments to "metatables" that provide fairly powerful metaprogramming support, they're all just tables that you can write literally and use the same functions to manipulate. You could say that alists fill this purpose and are even more syntactically regular since they're still sexps, but then why doesn't CL use alists for keyword args, and why don't Scheme's keyword arg extensions use alist syntax? Historical accident, or are alists just ugly? If you can admit that they're a little too ugly and verbose to include in function calls, then maybe you can see why I don't like writing alist literals.

> The fact that you are mainly manipulating lists does not mean that everything is eventually represented as supposedly inefficient linked-list at runtime.

I'm well aware that a compiler can easily expand keyword arguments into regular ones. I strongly doubt your compiler will transform all the literal alists in your program into hash tables, and replace all the alist-ref functions and so on operating on them with the equivalent hash table functions. As long as alists are more syntactically blessed than hash tables, people will use them where another data structure would be more appropriate.

[lispm]

Assoc lists make little sense as argument lists. Named arguments are basically like property lists. The point of argument lists for functions in Common Lisp is that they enable a contract. This requires special built-in support in the language. I want to see during compile time if an arg is missing or additional.

Example: the function WRITE takes several keyword arguments, but not :BAR. The compiler complains about wrong use.

    * (defun test () (write "foo" :bar 10))
    ; in: DEFUN TEST
    ;     (WRITE "foo" :BAR 10)
    ; 
    ; caught WARNING:
    ;   :BAR is not a known argument keyword.

Now if you allow a hashtable or some other data structure to be passed, then it would also be great, if one could specify at definition time which keys it takes, their default values, etc. We also may want to find out which values were default values, and which were actually passed.

The keyword argument facility for functions in Common Lisp provides all of that.

This allows you compile-time checks for code and makes interfaces easier to use.

Common Lisp also allows access to arguments as lists. This is simpler and more Lispy than using hash tables. For most purposes lists are useful enough and hash-tables would just add overhead.

[ICWiener]

> I strongly doubt your compiler will transform all the literal alists in your program into hash tables,

Time for experiment. SBCL doesn't, as far as I know. But you seem to imply that it is always better to use hash-tables, and I don't think so (even though hash-table can be implemented in a smart way, like in Lua).

Under roughly 10 elements, alist result in shorter code.

     (defun my-fun (x)
       (declare (optimize (speed 3) (debug 3) (safety 0))
                (type (member a b c) x))
       (let ((a '((a . 10) (b . 21) (c . 31))))
         (cdr (assoc  x a :test #'eq))))

     (disassemble #'my-fun)
     ; disassembly for MY-FUN
     ; Size: 53 bytes
     ; 04A7FF2F:       483B1592FFFFFF   CMP RDX, [RIP-110]         ; 'A
                                                                   ; no-arg-parsing entry point
     ;       36:       7511             JNE L1
     ;       38:       488B0D91FFFFFF   MOV RCX, [RIP-111]         ; '(A . 10)
     ;       3F: L0:   488B5101         MOV RDX, [RCX+1]
     ;       43:       488BE5           MOV RSP, RBP
     ;       46:       F8               CLC
     ;       47:       5D               POP RBP
     ;       48:       C3               RET
     ;       49: L1:   488B1D98FFFFFF   MOV RBX, [RIP-104]         ; '(B . 21)
     ;       50:       488B0D89FFFFFF   MOV RCX, [RIP-119]         ; '(C . 31)
     ;       57:       483B157AFFFFFF   CMP RDX, [RIP-134]         ; 'B
     ;       5E:       480F44CB         CMOVEQ RCX, RBX
     ;       62:       EBDB             JMP L0

This is roughly equivalent to a "case" construct (not shown here). Then, this is what I have with a hash-table:

     (let ((table (make-hash-table :test #'eq)))
       (declare (optimize (speed 3) (debug 3) (safety 0)))
       (setf (gethash 'a table) 10)
       (setf (gethash 'b table) 21)
       (setf (gethash 'c table) 31)
       
       (defun my-fun-hash (x)
         (declare (optimize (speed 3) (debug 3) (safety 0))
                  (type (member a b c) x))
         (gethash x table)))

     (disassemble #'my-fun-hash)
     ; disassembly for MY-FUN-HASH
     ; Size: 115 bytes
     ; 05467EB0:       .ENTRY MY-FUN-HASH(X)                       ; (FUNCTION (#)
                                                                   ;  (VALUES T # ..))
     ;      EE8:       8F4508           POP QWORD PTR [RBP+8]
     ;      EEB:       488D65E8         LEA RSP, [RBP-24]
     ;      EEF:       488B7805         MOV RDI, [RAX+5]
     ;      EF3:       4C8BC7           MOV R8, RDI
     ;      EF6:       498BF8           MOV RDI, R8
     ;      EF9:       BE17001020       MOV ESI, 537919511
     ;      EFE:       488B05FBFDFFFF   MOV RAX, [RIP-517]         ; #<FDEFINITION object for SB-IMPL::GETHASH3>
     ;      F05:       B906000000       MOV ECX, 6
     ;      F0A:       FF7508           PUSH QWORD PTR [RBP+8]
     ;      F0D:       FF6009           JMP QWORD PTR [RAX+9]
     ;      F10:       6A20             PUSH 32
     ;      F12:       B9604F4200       MOV ECX, 4345696           ; alloc_tramp
     ;      F17:       FFD1             CALL RCX
     ;      F19:       59               POP RCX
     ;      F1A:       488D490B         LEA RCX, [RCX+11]
     ;      F1E:       E917FFFFFF       JMP #x1005467E3A

The function with an hash-table has a constant size relatively to the number of elements in the table, which is fully known at compile-time. The size of the code with a case/alist grows linearly with the number of elements.

Now, let's compare speed.

In the following versions, I have added more elements in the alist, just to be sure the code with an hash-table is the shortest (from 'a to 'l).

     (time
      (dotimes (i 10000000)
        (dolist (x '(a b c d e f g h i j k l))
          (my-fun-hash x))))

     (time
      (dotimes (i 10000000)
        (dolist (x '(a b c d e f g h i j k l))
          (my-fun-case x))))

     (time
      (dotimes (i 10000000)
        (dolist (x '(a b c d e f g h i j k l))
          (my-fun x))))

Results:

   HASH:
    2.491 seconds of real time
    33,072 bytes consed
  
   ALIST:
    0.852 seconds of real time
    0 bytes consed
  
   CASE:
    0.778 seconds of real time
    0 bytes consed  
 

Even though in terms of footprint, the code tend to be larger with alist quite rapidly (10 elements), the resulting code is faster and does not allocate memory.

Also, just to clarify, alist and property lists have a different behavior than hash-tables, namely that the sequential access allows you to shadow values from another list: if you write (cons (cons 'a b) older-alist), you have a new list where the value for key 'a is b, and where values for other keys are those found in older-list (even though older-list also contains key 'a).

I don't quite remember what is my point anyway ;-) but it was fun to test the different behaviors.

[ANTSANTS]

Microbenchmarks are fun but silly. I tried to write a similar one for Lua, but LuaJIT ultimately recognized the program was useless and boiled it down to a 3 instruction loop:

  loop:
    add ebp, 1
    cmp ebp, 10000000
    jle loop

Anyway, in regards to your results, I could be wrong, but I think I read somewhere that LuaJIT uses linear search for tiny tables for this reason. Dunno what the threshold is, if there is one. The performance then would be similar to an alist, but saving a few bytes and cycles by not needing to deal with the extra list pointers and indirection.

> Also, just to clarify, alist and property lists have a different behavior than hash-tables, namely that the sequential access allows you to shadow values from another list: if you write (cons (cons 'a b) older-alist), you have a new list where the value for key 'a is b, and where values for other keys are those found in older-list (even though older-list also contains key 'a).

Oh yes, doesn't emacs make good use of this trick for its configuration variables? You can get the same effect with Lua's metatables, however, with a little elbow grease:

  parent = { a = "beep", b = "boop" }
  print(parent.a) --> beep
  print(parent.b) --> boop
  child = { a = "poing" }
  print(child.a) --> poing
  print(child.b) --> nil
  mt = { __index = parent }
  setmetatable(child, mt)
  print(child.a) --> poing
  print(child.b) --> boop

Not quite as easy as "cons", but very flexible; __index can be another table to searched if the lookup on the child fails (which in turn can have its own parent and so on), but it can also be a "metamethod" that is called whenever a lookup fails and can then do arbitrary things. A neat example off the top of my head: OpenGL has the peculiarity that you do not know the address of any of its functions until runtime, requiring a program to call a lookup function for each function to get a usable pointer. Declaring FFI function prototypes for many OpenGL functions is not such a big deal, but looking up hundreds of functions that you will never use can add significantly to startup time. So someone (possibly an HN user?) wrote an OpenGL FFI library that uses the __index metamethod in a clever way; the first time a function like "GL.CreateShader" is called, the lookup fails, and the __index metamethod mangles the index name a bit and in turn calls (on Windows) the C function wglGetProcAddress to look up its address, which it then stores in the original table. Using this library, you can write code that uses GL functions willy-nilly, and their addresses will automatically be looked up at runtime the first time they are used.

Sure, you can do the same thing in any language with macros or a preprocessor, but is that cool or what?

[ICWiener]

> ... recognized the program was useless and boiled it down to a 3 instruction loop

The loop itself is quite useless, why wasn't it also removed ? (Just kidding)

I don't have anything agains Lua or hash tables in principle. And of course tables are used in practice in CL code, but they aren't the primary data-structure.

> __index can be another table to searched if the lookup on the child fails (which in turn can have its own parent and so on)

> the first time a function like "GL.CreateShader" is called, the lookup fails, and the __index metamethod mangles the index name a bit and in turn calls (on Windows) the C function wglGetProcAddress to look up its address, which it then stores in the original table. Using this library, you can write code that uses GL functions willy-nilly, and their addresses will automatically be looked up at runtime the first time they are used.

So, it is an association list implemented using tables, where links are given by the __index property, using a metatable.

So maybe it is convenient after all to have a very simple data-structure like cons in a language and let more complex data be implemented with it, instead of the opposite.

[ANTSANTS]

> The loop itself is quite useless, why wasn't it also removed ? (Just kidding)

Good question! I guess LuaJIT isn't optimized for programs that don't do anything.

> So, it is an association list implemented using tables, where links are given by the __index property, using a metatable.

I think that's a matter of opinion. It's interesting that being able to form these simple hierarchies is emergent property of alists, but just because Lua provides another mechanism to implement hierarchical lookups, doesn't mean that the language designers were trying to ape alists. If anything, I'd assume that Roberto and company were inspired by Smalltalk's doesNotUnderstand message when they implemented __index metamethods.