[venuzr]

Basic question(s) as I am not a data scientist but have just taken a machine learning course (https://www.coursera.org/learn/machine-learning/ )

Won't trying different combinations of hyper parameters/lambda (over a small range) help us arrive better instead of manually tuning it? Or is that what the author meant by manual tuning?

[rjbwork]

I'm not a data scientist per se, but I've been working with some (boss and co-worker) to get some stuff operationalized and into production, so I've been responsible for generating inputs, helping analyze/visualize outputs, and building linear optimization models, so I've got some very basic experience.

As I understand it, one of the pitfalls of automatic tuning is that it becomes hard to account for seasonality and you will likely end up with useless parameters - for instance a customer ID is rarely a good parameter to optimize on, even as a categorical variable, except in very specific cases. It is probably a proxy variable for one or more other ones that you need to tease out of the rest of the data.

(warning, potentially me talking nonsense coming up) Automatic tuning is no substitute for a talented analyst who knows the data well and understands the goal. But if you've got hundreds to millions of parameters, you may not have another choice really.

[dasboth]

Depends what you're tuning. If it's something like the number of trees in a random forest, definitely do that automatically. If it's the number of clusters in a clustering problem, that's where you'd be asking an expert something like "how many distinct groups of customers do you think we should try to split them into?" and go from there. But even in that scenario, the expert's opinion might just be your starting point for automatic tuning.

[dswalter]

Author here: whether the hyperparameter tuning is done automatically or manually is not as important for what I was trying to say here. But yes, any of {grid-search, random-search, bayesian-optimization, etc} is likely to be more effective than manually tuning to squeeze out those last ounces of performance.