|
|
|
|
|
|
by vanderZwan
27 days ago
|
|
|
So I want to preface my next point by stating actually really like how syntax reads, hitting a sweet spot of leaning just enough into APL's symbol-focused terseness without losing the familiar structure of literally every other popular programming language ever that makes it easy to follow the code. I imagine it could be a great language to pick up for working out an algorithm with pen & paper (the best limus test for how well a language functions as pseudocode imo). Having said that, I have my doubts about the reduced token claims because of a lack of "begin" and "end" or "{" and "}" delimiters. Surely, if your language is identation based, LLMs will treat significant whitespace as a token too, since they need to keep track of it? Which would result in an extra token per newline compared to using explicit begin/end token? I suspect that to really optimize for tokens one would have to go the concatenative route, which significant whitespace very much isn't. |
|
|
I appreciate you sharing your thoughts, the whole point of a public preview is to find out how things are received, what’s working, what isn’t.
We more or less agree on the vibe-coded website (especially the copywriting). It’ll improve over time, but this is a side-project, time is limited, and I wanted to release something on 7/4.
Tungsten’s grammar and design are mine.
Tungsten’s implementation, is almost entirely LLM (lowering, escape analysis, SSA), with a good bit of prodding, benchmarking, troubleshooting; in order to have the performance land remotely close to where I knew it could.
While my primary motivation was to build the language that I wish existed, if I can make something that others enjoy or find useful, even better.
To address your other point:
Whitespace is almost certainly getting tokenized whether it’s significant to the language or not. And I would expect most folks are indenting their blocks anyway, even when they are delimited.
I think you’ll find the token reduction comes from more than just dropping delimiters.
I won’t try to litigate the point here, I’ll save that for a longer-form write up that is lacking on the website.
Thanks again for the feedback.
Related (in the repo):
WValue layout https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...
Ruby Lexer (regex-based) https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...
Ruby Lexer (spec) https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...