Hacker News new | ask | show | jobs
by 0x1ceb00da 23 days ago
Token list, not token trees. There are official libraries for parsing token stream as rust code but you can parse it as anything (eg json, html) if you want to.
1 comments

I think you meant TokenStream. They are trees, behind the scenes, because matching delimiters happens early on between lexing and parsing. By the time the rustc_proc_macro::TokenStream is exposed, the rustc_ast::tokenstream::TokenTree is hidden to the proc macro API.

https://doc.rust-lang.org/stable/nightly-rustc/rustc_ast/tok...

https://doc.rust-lang.org/stable/nightly-rustc/rustc_proc_ma...