You are committing the same folly you accuse him of. It would be fine to simply point out his use of it, but following that up with: "I'm surprised you got as many credulous responses as you did." You are basically doing the same thing. Prime that he is not an expert, then state that he should not be getting credulous responses. The implication of course being that his not being an expert means that his argument does not deserve credible thought and response.
Also, there's always a fine line with appeal to authority with regards to experts, which you would hope a judge would be in the field of law. For instance, is it folly to point to consensus in science? That is, after all, nothing but an authority.