[searchcord]

Hey,

Thanks for your feedback.

For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon.

Admins of these servers weren't contacted, as the content indexed is already publicly accessible, comparable to a forum like this or public subreddit. That said, I understand the sensitivity around data visibility, and I've made it very simple for any user to opt out of indexing at any time. Private or invite-only servers are, of course, completely excluded.

[klntsky]

I suggest you to remove the opt-out functionality and let it scrape private servers that it discovers via publicly posted invite links. You don't owe anyone posting on a public forum any privacy. Moreover, the most valuable data to search for is probably somewhat obscured.

[searchcord]

Hey,

Thanks for your suggestions. However, this does not work for a few reasons:

1. Joining servers is protected by increasingly difficult to solve captchas that have no commercially available solver. This is not a battle I want to fight.

2. There are a LOT of CSAM rings that spam invite links in public servers. This is also not something I want to go anywhere near.

Moreover, after the fallout of spy.pet, I think it is very important that users are able to opt out.

[squigz]

It is arguably more important that the world doesn't lose as much information as Discord now contains, or leave it gated behind logins and captchas and whatever other nonsense each server implements.

[hofrogs]

That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own

[searchcord]

I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.

[uniqueuid]

Wow, that must be quite expensive! You said the files alone are a few PB. So at least 2PB / 8 servers ~= 250TB per server, which would probably put each server at > 20k $ (unless you’re putting it together with duct tape and scraps, but even then the disks will cost a ton).

[searchcord]

Hey,

Not exactly. Attachments are only fetched from Discord as the user requests them. This means that the vast majority of attachments are never stored on my server. Right now, I only have about 280TB of attachments locally on my own infrastructure. You can see more stats here: https://searchcord.io/about

Thanks for your question!

[hofrogs]

Thanks for this. Well good luck with keeping it up, it's a really useful service.