[gamerDude]

This is a very interesting product, and since I have some data that could really benefitted from this, I tried it out.

I went through the upload process. But then I don't really know what to do from there. I tried some filters. I went to the invoke page, but I had no idea what invoke does or what the example output is. (Eventually figured out that I can just put text in the invoke and run it). All in all, there are a bunch of things that I don't really know what they are. I was a statistician before ml became popular, so I understand the underlying premises, but none of the modern language.

I would also really have liked to been able to filter by say, if the confidence level is over 80%, how accurate is the model. Because then I can say, well, if we use this, I can knock out tons of work at the 80% confidence rate and then just manually work with the rest.

I'm also not sure if you are seperating training/test data. All in all, looks nice, it was very easy to get started, but I'm a bit lost on what to do next and I'm having trouble judging how useful this will be to me and if I should invest more time.

[saintarian]

Thanks for trying us out and for the feedback! I agree that our filters are a little confusing right now and we're working on fixing it. In the meantime, here are a couple of filters you could try:

- To see all cases where the model disagrees with your annotation: Function Output = Disagrees, Desired Output = Any.

- To see the least confident predictions from the model: Function Output = Any, Desired Output = Any, Sort By = Least Confident Prediction.

Your idea us helping you pick a confidence threshold is a good one. We'll get that into our near-term roadmap.

We use a technique called cross-validation to seperate training and test data. We have that documented here: https://www.nyckel.com/docs#cross-validation

[gamerDude]

So, yeah. I could actually use some help on language here. Is Desired Output what I tagged it as?

I think output is confusing me a bit. Output being predicted value? And then desired output is user tagged value?

[saintarian]

Desired Output is what you tagged it as. Function Output is what the model predicted.

We tried to make the lingo developer-friendly. We think of models as functions that transform inputs to outputs. Instead of writing code to do so, as developers usually do, you train the function by providing desired outputs to sample inputs.

[danott]

Thanks for the feedback! We actually had a feature for “what is the accuracy if I only consider >80% confident samples” but we iterated away from it because people found it complicated. We’ll definitely bring it back when we can make it simple enough.

We’ve also found that people can get lost in the filters; in particular the “Not assigned” annotation filter we probably need to remove for people who have annotated all of their data.

In terms of separating training / test data: we use cross-validation so that we can abstract away the concept of train vs. test vs. validate sets.