[Taek]

Is the two billion page index open source?

I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be able to compete. Tesla, Uber, Waze also have a huge amount of data on your life.

Everyone is pushing the envelope on how much data they are collecting, and the companies which collect more data will compete better. As tech gets better we will increasingly be unable to resist sharing our whole lives with the companies who are powering modern living.

Even worse, there's a huge monopolization effect to having data. Nobody else has anywhere near as much data as Google. That means nobody else can compete. Nevermind the engineering, your algorithms can be 2x as good but you won't have 0.1% the data as a company with billions of daily users.

So Google and Facebook are left untouchable. Microsoft, Apple, and maybe Amazon can get in range. Is there anyone else?

We can fight back by giving up the privacy war and blowing the doors open instead. Take your data (as much as you dare) and make it public. Let every startup have access to it. Let every doctor have access to it. Give the small players a fighting chance.

That does mean a massive cultural shift. It means your neighbors will be able to look up your salary, your fetishes, your personal affairs. It's a big deal.

I don't see any other way out of this though. Surveillance technology is getting better faster than privacy technology, because surveillance tech has the entire tech industry behind it. Smarter phones, smarter TVs, smarter grocery stores, smarter credit cards, smarter shoes... smarter everything. Privacy is melting away and we aren't getting it back.

[deusu]

The software is already open-source.

A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation.

And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format.

BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

[siegecraft]

How does the search / indexing compare to sphinx or lucene?

[deusu]

I don't know sphinx at all, and my knowledge of lucene is very limited. Which means I don't know how they would compare to DeuSu.

[karussell]

You should really have a look into both. I'm curious: Why did you create a new software from scratch if you do not know existing?

If you'd base a project on lucene/solr/elasticsearch or sphinx you'd also increase the chance of contributing to it.

[jahewson]

If you're looking for an open source web crawl, commoncrawl.org has billions of pages.

[greglindahl]

... and Common Search made an index of the homepages, too.

[mi100hael]

Interesting perspective. What about the other extreme, though? This circumstance has only really existed in the past 20 years or so, maybe less. Why not just revert some of your behaviors?

- Delete your Facebook account. If you really need it to keep in touch with people across the country, at least delete the app from your phone and don't leave it open in a browser tab.

- Don't place asinine Amazon orders just because they ship free. Stop by a drug store or hardware store on your way home from work for odds & ends. You can even pay cash and kill the CC data bird with the same stone. Bonus points for instant gratification.

- Don't use GMail. Use an email provider like Protonmail or Tutanota that doesn't index all of your emails for advertising or other purposes.

- Don't sign in to Google (or Waze or whoever else's site) when navigating or searching so all of your actions online aren't automatically tied to a single account.

- Don't buy some internet-of-shit appliance that inevitably phones home with a bunch of telemetry data just so you can get a push notification when your laundry is dry.

Want to go really extreme? Now that you've uninstalled Facebook and aren't getting push notifications from your toaster, ditch your smart phone. Pay $15/mo for calls/texts on a Nokia 3310 and save some cash while you're at it. Want to listen to music on the go? Buy an MP3 player like everyone did 10 years ago. Want to play games on the go? Buy a GameBoy and experience the wonder of physical buttons while gaming. Really need directions on the go? You can probably get by with a Garmin or even gasp a paper map. Want to read HN or Reddit on the toilet? Try a book. Yes, I've written mobile apps and my iPhone is sitting on the desk right in front of me, but I hate the fucking thing and its always-connected mobile data. I've done the rest of the above bullet points and don't plan on buying another smart phone when this one hits planned-obsolescence in another year or two. Aggressive data collection depends on user engagement. The easiest way to fight it is to just quit engaging. Usually it'll save you money, too, which is nice.

[allendoerfer]

I am on your side. Except I need to contact some friends and - more importantly - customers, so I cannot entirely ditch everything. I cannot get rid of Facebook (Pages, API, relatives, even some customers), Skype (customers) and Whatsapp (friends). I am waiting for Whatsapp to become available on either Ubuntu Touch or Firefox OS. I would use these before buying a dumbphone and an MP3 player.