[saurik]

Would it really be that hard to reimplement this yourself if you needed to? The idea seems to be that this is a service: it isn't doing anything that difficult to accomplish, it is just doing something that happens to be resource intensive.

[nwh]

CPU resources or human ones? This package by Bitpay solves the latter.

https://github.com/bitpay/insight

[saurik]

That's the wrong repository: that's a front-end (a UI written using angular); the backend repository (which could be said to be an open-source implementation of chain.com) is the one I link below. All this package does is provide a node.js wrapper for the bitcoind API. Almost all of the code (in both repositories) is project boilerplate or data marshaling: everything even remotely difficult is just "run bitcoind". I maintain it would be difficult to get locked in to a service like this just because it isn't open source: it isn't that difficult to rewrite.

https://github.com/bitpay/insight-api

As for "resource intensive", I was thinking memory, disk space, and network connectivity: the reason these services are valuable is because the blockchain is 18GB at this point, so downloading it to clients or even storing it at all on some clients, is at minimum "egregiously painful" and sometimes "pretty much impossible". If you want to be able to answer tons of different kinds of questions about transactions with low latency you are likely going to be storing this information with a number of specialized indexes, further increasing the database size requirement.

However, "people" is also a valuable point, but no: having an open source project doesn't solve that; if you make a program that uses bitcoin, you either have to download and manipulate the blockchain locally (which sounds awesome, and is arguably half the point of bitcoin to many reasonable people, but in practice is not realistic or even possible in many if not most interesting situations) or use a service. You can either run your own service or use someone else's. I enjoy running services: I accept that most people do not ;P. Instead, they would rather use a hosted API such as this one (or any of the numerous existing alternatives to this one) so they don't have to have people maintaining servers.

[nwh]

The reality though is that most people will never, ever need this sort of information. Unless you have a particular desire to run a block explorer or other lookup service, you are only going to be interested in addresses that you own and not those of the rest of the network. You don't need to store extended lookup tables and indexes for an ever increasing amount of information, doing so is foolish as it doesn't scale at all with the size of the network.

If a service needs to have a wallet connected to the network without the resources for storing the entire blockchain, they can just use a SPV [0] client like BitcoinJ to set bloom filters on their peers. They then get a stream of trustless information that's only relevant to the wallet at hand. You end up with a local service that's faster, completely under your control, and has a tiny resource footprint. You could happily run that a device like an iPhone, no problem.

[0]: https://en.bitcoin.it/wiki/Scalability#Simplified_payment_ve...

[saurik]

So, this is the first I'd heard of the bloom filter feature: the descriptions I've seen of SPV allowed the client to only store a subset of the data, but still needed to download everything at least once in order to have any semblance of trust in the data it was seeing. The web page you link you also has a "See Also" to a page that makes bitcoinj sound horribly insecure, but maybe that's no longer the case with the bloom filter extension? This was all written before this bloom filter feature existed (which was late October of 2012); despite that being a little over a year and a half ago, there is almost no mention anywhere of it... I will have to spend more time learning about this feature and see how it changes things. (Maybe you are then correct: having a service like this is totally useless. However, I would then argue that it should be a really big priority to make people understand that this feature exists, is deployed, is practical, and is secure, as otherwise people are going to keep going about their lives as I have assuming this is all impossible with the current implementation of Bitcoin.)

[nwh]

Ugh, yeah the documentation for anything related to Bitcoin is terrible. Originally it didn't use bloom filters, they're a new thing as of Bitcoin client version 0.8 and above.

> the descriptions I've seen of SPV allowed the client to only store a subset of the data, but still needed to download everything at least once in order to have any semblance of trust in the data it was seeing.

With bloom filters you don't need to bother with all of that. The headers of the blocks are separate from the transaction data itself (that's in the merkle tree), so the SPV client can download all of the headers up to the current date and verify their difficulty in a few seconds. As the wallet gains addresses it sets bloom filters with it's peers, when they announce a new block on the network they send the SPV client the block header and also a trimmed merkle tree with only the transactions relevant to the client left in it. The SPV client can verify that the transaction is in the block by following up the tree to the block header, without ever needing to download the entire thing.

Privacy is gained by setting false positives in the filter, the more junk results the less likely it in the peer can discover what the SPV client is really interested in and what is being discarded. The SPV client ultimately doesn't need to store anything but the outputs that are unspent, and transaction history if this is required. There's no trust in the peers as at worst they can withhold blocks from us that contain transactional data, to counter this we can connect to multiple peers with the same filters set and compare the difference.

> However, I would then argue that it should be a really big priority to make people understand that this feature exists, is deployed, is practical, and is secure, as otherwise people are going to keep going about their lives as I have assuming this is all impossible with the current implementation of Bitcoin

This is a problem really, we have lots of awesome crypto features but everybody just insists on using solutions that don't scale like blockchain.info. The original whitepaper describes some of the scaling features but doesn't mention bloom filters specifically I don't think.