[WalterBright]

Um, it is necessary to compile a program before being able to interpret it. I don't know how early BASICs were implemented, but the usual method is to compile it to some sort of intermediate representation, and then interpret that representation.

D's compile time function execution engine works that way. So does the Javascript compiler/interpreter engine I wrote years ago, and the Java compiler I wrote eons ago.

The purpose to going all the way to generating machine code is the result often runs 10x faster.

[wvenable]

Early BASICs didn't compile a program before interpreting it. The interpreter read the code as written and executed it step-by-step. There was some tokenization; keywords were turned into single or double bytes and that was literally done when you pressed enter on the keyboard. Your source code was these actual tokenized bytes. On the Commodore 64, you could type the tokenized versions of keywords instead of the full keyword as a shortcut. Even numbers were not transformed into bytes ahead of time.

This was used to save memory -- there wasn't much room to hold both the source code and an intermediate form. But also it wasn't that necessary, with the keywords tokenized and the syntax so simple that there wouldn't have been much savings in space or performance.

[tasty_freeze]

You have an idiosyncratic definition of "compiler" then. Many BASICs, including the MS family of BASICs, did tokenize keywords to save on memory storage.

But 99.9% of people take "compiler" to mean translating source code to either a native CPU instruction set or a VM instruction set. In any tutorial on compilers, tokenization is only one aspect of compilation, as you know very well. And unlike some of the tricky tokenization aspects that crop up in languages like C++, BASIC interpreters simply had a table of keywords with the MSB set to indicate boundaries between keywords. The tokenizer simply did greedy "first token which matches the next few characters" is the winner, and encoded the Nth entry from that table as token (0x80 + N).

When LIST'ing a program, the same table was used: if the byte was >= 0x80, then the first N-1 keywords in the table were skipped over and the next one was printed out.

There were also BASIC implementations that did not tokenize anything; every byte was simply interpreted on every execution of the line. There were tiny BASICs where instead of using the full keyword "PR" meant "PRINT", and "GO" meant "GOTO" etc.

[stevekemp]

It is not necessary to compile a program, in the general case, before executing it.

Many programming languages parse their program to an AST then walk that AST interpretting as they go. But for BASIC you can parse/execute statement by statement - no need to parse the whole program ahead of time, and certainly zero need to compile to either machine code or any internal representation.

Remember at the time we're talking about 64k was a lot of RAM. Some machines had less.

[WalterBright]

The parsing, even if line by line as necessary, is still compiling.

[vidarh]

In 45 years of writing software, I've never before seen anyone call tokenizing a BASIC program compilation. It's decidedly not common usage.

[WalterBright]

I've been writing compilers for 45 years now. Tokenizing is a big part of every textbook on compilers. To resolve expressions (which are recursive in nature) it would have had to do more than just tokenizing. While this isn't hard at all, it's "parsing" which is also qualifying it as a compiler.

I.e. the basic program was lexing and parsing. It's a compiler. A very simple one, sure, but a compiler.

[vidarh]

Yes, but tokenization on its own is not compilation any more than whiskers are a cat just because a cat has them.

"Nobody" uses it that way, and language is defined by use.

[jacquesm]

Compilers generate code in another, usually lower level language that is executed by reading all of the code that could be executed first. Interpreters (such as the BASIC interpreter we are discussing here) read only that part of the code that gets executed and typically call functions rather than that they generate code (never mind JIT). Tokenization prior to interpretation is technically an optional step (it's just an efficiency boost) and is not normally confused with compilation even if there are some superficial similarities.

You of all people should know this, come on.

[WalterBright]

You and I have a different point of view.

[eichin]

> necessary to compile

Um, no? your experience is probably at least two decades after the time period in question.. The more advanced versions of, for example, the TRS-80 BASIC (part of this "microcomputer BASICs that all share a common set of bugs") did no more than tokenize - so, `10 PRINT "Hello"` would have a binary representation for the line number, a single byte token for PRINT, then " H E L L O " and an end-of-line marker. Actually interpreting the code involved just reading it linearly; GOTO linenumber involved scanning the entire code in memory for that line number (and yes, people really did optimize things by putting GOTO and GOSUB targets earlier in the program so the interpreter would find them faster :-)

[EvanAnderson]

I was going to post this, but you beat me to it.

It's a VM of a sort, and the p-code the VM executes is tokenized input.

[WalterBright]

Tokenizing it and interpreting the token stream is still a compilation process. Even if it re-tokenized it each time it executed a line.

[wvenable]

Tokenizing is a necessary but not a sufficient task for compilation. I could tokenize this comment to efficiently store it in a database but that would have nothing to do with compilation.

[WalterBright]

Recognizing `3+x*(2+y)` is compilation - even if the program is being executed while compiling it.

[wvenable]

You can continue to argue the point but that goes against every single definition of compilation that exists. Compilation is a transformation of programming language into another form. For example, taking `3+x*(2+y)` and transforming it into a series of byte codes, machine language instructions, ASTs, or even C code would be compilation.

The BASIC interpreter doesn't recognize `3+x*(2+y)` nor does it compile it instead it evaluates that expression using a pair of stacks. You've expanded the definition of compilation to cover almost all computation. It's compilers all the way down to the electrons.

[eichin]

No problem calling it parsing, but yeah, "compilation" feels like a huge stretch. And they didn't do recursive descent - just tokenizing for compactness (when you only have 4k or 16k of RAM you do things like that) - you could still get syntax errors at runtime. In some interpreters it also served to normalize abbreviations to save typing.

[WalterBright]

Well, all my compilers use recursive descent for expressions, meaning the stack is used to maintain the current state. Whether you evaluate it while doing this or produce an IR is a trivial difference.