| Great question. We built a dataset of past success and fails with training, validation and test sets. Afterwards, the model got trained with some context from human experts at the beginning such as "Being a repeat entrepreneur is positive". Then, the model built out a explainable decision tree without human intervention. Lastly, a human looked into the models that machine came up with, and improve its logic further. For instance, the model might be asking a vague question such as "Is the entrepreneur based in an innovation hub?". The expert may prompt it to be "Hey, be specific and put specific cities or regions to improve this question". Then the model would re-run and try to improve that question with a goal to increase the precision. So this goes on and on with expert-in-the-loop. Sometimes, expert may give wrong advice! :) And, in that case, the performance would decrease... Lastly, your example is a great one. Human may introduce some bias into the process. For that reason, we also built a model with no human intervention with 10x cross validation. And, the model still outperforms all humans... Given that expert-in-the-loop is a time intensive and expensive process, we did not do 10x fold validation for that. However, our initial observations indicate that the magic happens when humans and experts work together. |