[kelvinn]

I have been pondering the op's question for over two years, and this is the route I will probably take. Some websites like TicTrac do a reasonably good job at aggregating your stats from the web, but, like the op's example of electricity, will obviously never be able to query every service.

Much of my quantifiable data is sent to Google's Fusion Table, but I do not feel this is a good long term solution.

My intention is to define a base set of criteria for how a certain database needs to be formatted, similar to another comment here, and then let anybody make their tools available as either for "collection" of "visulisation/analysis". As long as some core fields are standardised, e.g. "quantity" and "date", then the data can be easily analysed. Each individual would control their own database, either on their own host or as a DBaaS, but tools to collect and visualize data could be shared.

I am leaning towards a document store (e.g. CouchDB and Cloudant), as that would allow any tool to push data in without knowledge of the schema. One of the nice things about some of the DBaaS is you can easily create individual username/password or API keys with specific permissions, so a third-party tool could write records, but not necessarily read any of your data. A standardised database would also benefit by having other tools able to utilised it, unlike something like Google's Cloud Datastore (which I do like!) In particular, I am thinking about the CouchDB and ElasticSearch integration.

So, why not just wait for apps like TicTrac or Saga to support every service? I have two reasons. Firstly, many of the other tools to aggregate data seem to have gone out of business. Secondly, there are some services I do not like to give third parties access to; email is one example, as is the ability to log keystrokes on my computer. However, I would like to see the summarised data from these services recorded with the rest of my Little Data.

Another option would be to dump everything to text files and upload them to Google's BigQuery, but I am leaning towards a shared tools / individual database model, as it would probably encourage better collaboration with other people.

[rpedela]

Very interesting. I am building a service called Datalanche (~2 weeks till release) which sounds a lot like what you describe. I would love to further discuss this with you offline if you have time.

If so, please contact me: rpedela [at] datalanche [dot] com