Probably okay to have the source code to the engine.
However, if you have used the the system which creates a database of questionable dark web links on your machine, that could be tricky to explain... and easy to implicate
Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example.
I am guessing you have some personal instance you use at least for testing/"education", right?
> On June 20th, board members of the „Zwiebelfreunde“ association in multiple German cities had their homes searched under the dubious pretence that they were „witnesses“ while their computers and storage media were confiscated.
That's something very different though. Exit nodes are providing a service and are, for all intents and purposes, the only visible client on the clearnet (and might not even be involved: there's nothing stopping you from running a private proxy on the same machine you run your exit node on). TOR-developers that do not run exit nodes but contribute to TOR typically don't get searched, at least to my knowledge.
Content that's illegal to possess is a different issue, though I'm sure they'd make for an interesting case because a crawler downloading, saving and parsing an HTML page isn't as clear cut as a human evaluating and deciding what to download and store. "The suspect has the hard- and software necessary to download this content" shouldn't be enough to convince a judge to issue a search warrant, but then again, judges probably have very little technical knowledge.
The Zwiebelfreunde raids were not because of their TOR (hi dewey) activities, but rather they collected donations for the riseup email service.
If the police can convince a judge to raid the board members and their families of registered club just because they, among many other things, collected some donations for an US org, then some overzealous police detective or DA going after some dev who made a webcrawler for the "dark web" and is probably in possession (knowingly or not) or illegal content isn't much of a stretch either.
The OP wrote a crawler and used it to crawl Tor. Depending on where they live, accessing the content might be illegal, and storing some of the content in your computer might be illegal as well.
Law enforcement might be monitoring some domains, or have set up some honeypots that the OP might crawl automatically.
You don't want to end up in court having to argue about why your computed accessed some child pornography and downloaded it, and trying to explain to a jury that you did not did those things, but that the crawler that you programmed to do those things did.
Sure, nobody might end up raiding the OPs home, and even if they do, the OP might be able to successfully survive a jury. But just having to go through that might suck.
If the OP only wrote the software and never used it, then they are fine. But from the article, they did use it, so who knows where the crawler landed. Chances are nowhere good.
I didn't talk about the dev getting contacted about third parties abusing the software, but about the dev keeping a DB of indexed content for development/testing/"education" that would most likely include illegal-to-possess content.
And that some eager police people like to "inconvenience" people connected to TOR somehow isn't exactly new, either.
E.g. there have been multiple raids against TOR exit node operators in different countries around the world in the past, even when the police was fully aware it was a TOR exit node that did not store information.
Maybe I'm just too paranoid - then again I used to run a TOR exit node myself and had a bunch of less than pleasant run-ins with the police, tho thankfully no raids.
However, if you have used the the system which creates a database of questionable dark web links on your machine, that could be tricky to explain... and easy to implicate