|
|
|
|
|
by aprentic
71 days ago
|
|
There's certainly an interesting question here, even if Tokenstree doesn't provide a solution or even define the problem well. The broader questions are still interesting. If an AI is trained more on language A than language B but has some training in translating B to A, what is the overhead of that translation? If the abilities are combined in the same model, how much lower is the overhead than doing it as separate operations? ie is f(a) < f(b) < f(t(B,A) ? where a and b are in A and B and f() and t() are the costs of processing a prompt and the cost of translating a prompt. Then there's the additional question of what happens with character based languages. It's not obvious how it would make sense to assign multiple tokens to a single character but there's the question of how much information in character based vs phonic based words and what the information content of sentences with either one is. |
|