[protomikron]

Congratulations to the release.

I am a little bit skeptic about your running example, the "ML Editor". A model that helps you asking "good" questions, e.g. on StackOverflow.

Isn't that like an extremely complicated problem, I would even say AI hard? How do you want to evaluate if a question is "good" (and sure thing, it's not the number of upvotes it gets)? Is there a working example of such an editor in action, because I highly doubt that this is currently possible.

[e_ameisen]

I answered the next comment down the chain, but I agree with you. As framed, there is no satisfactory solution to making a question "good".

Improving question quality could however be a product goal for StackOverflow, or for a company focusing on writing tools (Grammarly, Textio,...). The book describes a process for turning that vague product goal into a more tangible set of metrics, which lead to choosing an ML approach, and iterating on it.

Eventually there is a finished prototype, for which you can find a GitHub link in the PDF preview. It has definitely not solved how to evaluate if a question is "good", but aims to provide a narrower set of recommendations (the first chapter actually dives into an approach for this).

[IanSanders]

Didn't read the book yet, but from your description absolutely agree on the trickiness of the problem - it almost requires solving the "proper artificial intelligence" problem first, like you say.

However, I imagine it could be applied to refine an existing question? There certainly exist "obviously poor" questions on SO, and it's a good first step to make otherwise poor question "look" like a good question - trivial things like formatting and misuse of the language. It won't get other, high-level attributes of a genuinely good question however, but some poor questions are poor in just that - formatting and language, the "requires editing" queue.

Regarding "intrinsically poor" questions, on the other hand, if everyone used the described model, readers would now have an increased cognitive load to distinguish between good and poor questions. Over time, the described model would drop in performance, as the "typical good question attributes" are used in poor questions which wouldn't have those otherwise.

(Forgive me for trivialising the concept of the quality of a question)

It's still a very interesting problem for a book. It's just as suitable for demonstrating the model development process, and it's likely very relevant to the vast majority of the readers (I imagine).

[e_ameisen]

Yes, this is the approach in the book. The concept of question quality is nuanced, and does not have a clear definition. It can be easy to feel like you've solved the problem by just throwing in ML and calling it a day, but producing something useful is a real challenge.

The book covers multiple aspect of that process, from choosing an ML approach that isn't too simple or ambitious, to iterating on a model within the context of its final use case (i.e rather than only optimizing for a metric, testing how the model helps with its end goal).

In my experience, I've found that it is often those challenges that make or break the quality of an ML product, so the book focuses on tools to make complex problems more tractable, and less risky.

[bordercases]

I think it's better to believe that the problem isn't hard and see how far you get, than assuming it's AGI complete without being precise about why. And we only really get a sense of why we ought to strongly believe it's AGI complete if we try to make progress on the problem.