[catskindleyou]

The spider-rs GitHub project has concurrent chunking of AI while trimming the context of unwanted things html elements, css, and etc. You should check out the project. https://github.com/spider-rs