|
|
|
|
|
|
by xivzgrev
587 days ago
|
|
|
curious how the human intervention worked? is there a risk of bias? when VCs make decisions, there is no hindsight. but when you ask humans to evaluate predictions, you necessarily are using past (training) data and the humans may recognize general successful patterns or even the examples themselves. For example: the model outputs that it's considering an investment in a company that lets drivers pick up passengers on the way to their destination and earn some money as well. a person may think, "Duh, this is uber! invest!" Thus inflating "success" rate. |
|
|
We built a dataset of past success and fails with training, validation and test sets. Afterwards, the model got trained with some context from human experts at the beginning such as "Being a repeat entrepreneur is positive". Then, the model built out a explainable decision tree without human intervention. Lastly, a human looked into the models that machine came up with, and improve its logic further. For instance, the model might be asking a vague question such as "Is the entrepreneur based in an innovation hub?". The expert may prompt it to be "Hey, be specific and put specific cities or regions to improve this question".
Then the model would re-run and try to improve that question with a goal to increase the precision.
So this goes on and on with expert-in-the-loop.
Sometimes, expert may give wrong advice! :) And, in that case, the performance would decrease...
Lastly, your example is a great one. Human may introduce some bias into the process. For that reason, we also built a model with no human intervention with 10x cross validation. And, the model still outperforms all humans...
Given that expert-in-the-loop is a time intensive and expensive process, we did not do 10x fold validation for that. However, our initial observations indicate that the magic happens when humans and experts work together.