[dscpls]

Why a service and not a library?

It looks like a great way for you to discover URLs but like a terribly slow way for people to avoid implementing robots.txt rules.

[fooock]

The project have multiple subprojects, and one of these is the parser. Any developer can compile or extend it without more effort and create a library. Just know to code in Kotlin / Java.

The aim of this project is only check if a given web resource can be crawled by a user-agent, but using a API