[chaoky]

A higher order function doesn't serve the same purpose as a macro. A higher order function is meant to be applied, called, composed etc. A lisp macro is a different type of abstraction. For example, many people think that macros are just hiding lambda's of higher order functions. This is wrong. A macro abstracts over implementation details of a construct to make it read naturally. For example, you can write a function that opens and then closes a file like so...

  with-open-file(filename, lambda file: do stuff with file)

but with a lisp macro, you only have to write

  with-open-file (filename):
     do stuff with file

The point of an abstraction is so you don't have to think about the implementation. Written like the latter, with-open-file is simply more natural to write this way. You only have to think, "oh, its a construct that opens a file then closes it after the body is done", rather than "oh, its a higher order function that i have to pass another function into that takes the file as an argument..." etc.

When you write with-open-file as a macro, it could be implemented as a higher order function, or it could be implemented as a low level set of GOTO statements. It doesn't matter. The macro abstracts away the low level detail, just providing the most natural way for you to use the construct. It might not seem like the macro is doing much in that particular example, but a construct that defines a class (like defclass) is something you can write as a macro, which can expand into functions that do the actual defining. You could write a class defining construct as a higher order function, but then you'd have to constantly worry about how your construct was implemented as a function. instead of just writing something natural like

  defclass tiger (animal):
    age init-value: 0
    name type: string

which you could write if you implemented defclass as a macro. otherwise, you'd have to do something crazy stupid like

  defclass(name='tiger', inherits-from = find-class('animal')
           slots=['age, name'] ...)

how could a higher order function possibly implement that without making people using it tear their hair out? They have to bend to the implementation, not make the abstraction bend to what's natural.

[pugio]

Your point seems to be that macros allow for a slightly more natural syntax for certain things, but I can do pretty much the same thing in a language with a natural HOF syntax (Ruby):

  with_open_file filename do |f|
    do stuff with file
  end

And for your second example:

  # our 'macro' function
  def defclass(name, parent, &blk)
    k = Class.new(parent)
    k.instance_eval(&blk)
    Kernel.const_set(name, k)
  end

  defclass :Tiger, Animal do
    attr_accessor :age, :name
  end

Yes, macros allow for a superset of what you can reasonably accomplish with higher order functions, but I haven't yet seen a simple practical example of where the added power is useful. I'm sure it's nice to have, but there's something to be said about a language which generally gets you 95% of the way there using a simple set of built-in operations.

[chaoky]

Sure, ruby has nice syntax for lambda, but the entire point is not even having to worry about what goes into blocks and what not. Syntax is a bad excuse for abstraction. Lisp has higher order functions too, but a deceptively short (or "natural") syntax can easily sweep ugly semantics under the rug. I know no one would ever write your defclass hof anyways because of the extreme runtime overhead that causes. If you wanted to reimplement your defclass macro more efficiently, I don't see how you would be able to do so while not breaking all of a users code relying on that function. The point of a macro is so the implementation detail is hidden, and the interface doesn't have to change when you change the implementation. Lisp has higher order functions too. Hell, that's where ruby got it from. Higher functions have their uses, but they are not for abstracting patterns in code like macros are. They are for abstracting procedures in code on arguments received at runtime. The difference is macro expansion happens lexically, at compile time, while functions are part of the program at runtime.

If you haven't seen a practical example of what it's useful for, that's akin to the attitude of a C programmer not understanding the usefulness of higher order functions. They say, I get 95 percent of the way their with good old functions and function pointers. We would find that absurd, just as how I find the claim that a 'simple practical example of where added power of a macro is useful does not exist' is absurd

For example, see 'A unit testing framework' in 'practical common lisp' available online by Peter seibel. I can't possibly imagine how you'd be able to create a unit testing framework abstraction in Ruby as nice or efficient as the one presented with higher order functions in 26 lines of code. But it'd be cool if anyone could prove otherwise.

[davexunit]

Notice that in your Ruby code you have to use quoted symbols like :Tiger and :age and :name, because you cannot extend Ruby's syntax with your own. Ruby has good metaprogramming facilities, but it's no substitute for a real macro system.

[eatonphil]

In what language are you writing?

[chaoky]

Semantically, I'm writing in common lisp. If you mean the syntax, that's an ad hoc thing that comes up when writing in an HTML text box and emacs isnt here. But the syntax is shallow and unimportant compared to the abstraction presented. (it's just normal lisp syntax with implied parentheses anyways, so it's a bit ambiguous)

[mbrock]

That might be pseudo-code, but Dylan has a similar syntax, like Lisp without the explicit S-expressions.

[eatonphil]

It seemed a little unfair to be talking about such beautiful and extensible syntax without including the parentheses. But I am unfamiliar with Dylan.

[mbrock]

Well, typing parens in a HTML textbox can be pretty tedious, so I understand the desire for pseudocode.

Dylan has an interesting hygienic macro system that's similar to Scheme's. I don't think Dylan allows for arbitrary code-generating macro procedures, but it's possible in principle; see [1].

[1]: https://people.csail.mit.edu/jrb/Projects/dexprs.pdf