[kbirkeland]

The algorithm for generating those tokens is explained in the Lexical Analysis reference[0], and a quick look at the cpython source shows the logic is implemented in Parser/tokenizer.c[1].

[0]: https://docs.python.org/3/reference/lexical_analysis.html#in...

[1]: https://github.com/python/cpython/blob/master/Parser/tokeniz...

[xg15]

Very cool! Thanks much for the links.