[doctorpangloss]

Hmm, but that will filter out more than half the players on the Match Wins and Trophies based leaderboards, many of them Diamond and Mythic. So I think your choice of 62% match win rate is almost certainly disproportionately selecting for people who received very good draft choices, even if it includes some actually very good players in the data set.

I mean 62% might feel like a good number, but it's arbitrary, you'd have to justify how you chose it, and just eyeballing it, it is filtering out a lot of very good players with many, many more match wins.

Perhaps you can sort by Latest Rank, and filter out people with 2 or fewer trophies. Or you will have to validate with known bad draft choices in the prompt, to see what it does. Suffice it to say, I still don't think the 17Lands data represents what you think it does.

Like without a direct discussion about measuring and accounting for luck in the draft... for all I know the data is seriously flawed. It probably isn't, but it's maybe one of many, many issues to address when dealing with strategy card game AI problems.

[dmakian]

Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts!

Definitely not perfect data though, and agree that defining good in this context is hard -- a lot of the variance of "good" depends on how you play the cards either way. All good points!

[doctorpangloss]

> I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts!

Hmm, but there are a lot of players with greater than a 62% lifetime win rate with very few drafts, but there may be many of those players... do you see? The win rate isn't a good filter. You chose it, you are trying to justify it, and I'm not convinced, not without the hard numbers.

I'm not confused about what filter you chose. I just think it's a bad filter, and you haven't thought very deeply about how it affects the data, which includes presumably your test and validation data - however you're choosing to test and validate, apparently by hand, by some eyeballed examples.

Anyway I think you have to compare with a non-LLM, non-random baseline to have any sense if this stuff is working at all. I could be dead wrong. I would maybe compare with a community draft picker.

[sdwr]

A lot of words to say "good players can't actually be good"

[donpark]

Data selection depends the use-case. Two contrasting use-cases I see are:

- Emulation

- Advisor

In case of MTG player emulation for example, I think it makes sense to group data by some rankable criteria like winrate to train rank-specific models that can mimic players of each rank.