|
|
|
|
|
|
by rhdunn
1165 days ago
|
|
|
I've noticed that it correctly splits warm|est, cold|est, bleak|est, but darkest is a single token. I've also seen it group `?"`, `."`, `!"`, and `.--` into single tokens. It also splits some words like "Elton" as El|ton. Presumably in that case it has mis-idetified a -ton prefix. |
|
|