I just started using hledger (http://hledger.org/) for personal double-entry bookkeeping. So far it is pretty awesome. Some of the things I like over, say, Mint:
* I can keep arbitrary asset, liability, and expense categories in a tree-like structure.
* I can split up transactions between different accounts. This is useful because I can split a dinner bill between my credit card and cash. It is also useful because I can split items from a large Amazon order into their respective categories.
* The ledger file is a very simple text file, so I can keep it in version control.
* I haven't taken advantage of the report features yet, but they look very flexible. You can add up any accounts you want based on regular expressions.
* You can have "virtual" accounts that can be used for budgets. For example, you can set aside arbitrary chunks of your savings account for different purposes and keep track of them separately, even though they all actually reside in your savings account at your bank.
I will take a look a hinance to see how it could be useful for me!
I'm going to plug a personal finance application; I have no connection to it other than using it and wanting it to survive. I hope that's not inappropriately hijacking this thread, but downvote me if so.
It's a mature application with a good GUI, with double-entry and sophisticated functionality, cross-platform (Linux too), has an open API and is scriptable in Python, [1] and IIRC data is in XML. Last I needed them (years ago), support was responsive and forums were technical and helpful.
From one of the reviews of the mobile app: "now, wifi sync is no longer available. The developer has seen fit to force you to sync your personal, financial information over the Internet via Dropbox."
There's a problem with having to use and trust Dropbox, however. It's a company that's consistently proven that they can't be trusted, both in terms of truth and competence.
Thankfully, with sane encryption you don't need to trust them. (Or any other cloud outfit).
They're more than welcome to distribute and analyze as widely as they see fit, the few hundred megabytes of random-looking data sitting in my account :)
This is an awesome attempt at a big problem. My financial data seems one of the most monetizable aspects of my life and I'm particularly reticent to give it to Intuit et al., but never seem to have much of a choice.
I'm actually curious why broader community effort hasn't sprouted up around web scraping for banks, given how horrible API support has traditionally been. Plaid (plaid.com) purports to make this easy, but it's not very mature yet and will be a paid service.
Weboob (http://weboob.org) has moderately active community. There's even a couple of finance tools built around it. Though web services it scrapes are mostly European.
Yes, scraping is legal in most cases, and if you do it yourself you don't give your login credentials to a third party either, which might be a liability issue.
From personal experience, I've tried scraping my own TD Ameritrade data and quickly got flagged by their system as a virus because my request pattern likely failed some heuristic check. They disabled my account "until I get that cleaned up" and I had to plead with multiple people over the phone just to get it re-enabled.
The overall experience was quite terrible, but I wouldn't be surprised if it's the norm.
Certainly can imagine this happens. Currently I'm scraping 4 banks and 4 shops on a daily basis, and the only issue was when scraping for the first time from new IP address. They asked a security question or displayed a captcha. I resolved this by logging in first time from the browser through SSH tunnel over that IP. All subsequent scraping went well since then.
Opposing personal experience - I've been scraping my bank data (scotia) for about a year now, haven't had any serious problems (there seems to be some kind of signin counter that thresholds and then forces secret questions to be reset, but that's it).
I want to make a similar tool, I have accounts over two banks and a shared account with my girlfriend. I really want consolidated reporting. This sort of thing would be much easier if more banks provided a json api to customers.
My business has bank accounts in four countries, and none of these banks provides a scriptable way to access transaction data.
My current solution is to manually download data from each acount for processing, but when I get some play time I want to see how the bank apps communicate, and attempt to turn this into an API. Anyone had any success with this approach?
Take a look at Weboob (http://weboob.org). It's an attempt to create a well-defined uniform API for websites lacking one. I wrote a few modules for it, and they're scraping 4 banks and 4 shops on a daily basis for last few months.
As I see it, upside of this approach is that you can scrape pretty much everything you can open in your browser.
Downside is that you have to update your scraper every time website changes. From my personal experience this happens at least once a month.
Thanks for this. I've been looking for a solution for this problem for a while and this looks like a good attempt.
While bank statements scraping or API is good, it may be useful for somebody to note that that it's often possible to import your historic bank statements using scanned images or PDFs. Excel and many accounting programs have this feature. (I haven't personally done this yet though)
Excellent point about PDFs. Some Weboob modules scrape PDF statements as well: at least Amazon Store Card, Wells Fargo and Citibank (disclaimer - written by yours truly). But even with these statements sometimes whole history cannot be restored. For example, Citibank stores statements for only so many years.
I've been using iBank [1] for a couple of years now. It's able to pull data from different banks, import CSV, and other format. It has many other features that I'm not using. It's not a hosted solution so everything is on your machine.
For me the only missing feature of iBank is the limited support for charts and reports (it has quite a few predefined that you can customize, but creating new ones is not supported).
I think "writing my own fits-like-a-glove tool" pretty much covers it. I wanted a very minimalistic tool which just scrapes the data and transforms it into a report. Probably I could have achieved the same result by writing a few plugins for HLedger. Though I doubt the overall code size would've been smaller (currently it's <1000 SLOC and I'm not planning to add any more features). I've been using it for last few months without adding more features. The only thing that changes/expands is scraping, which is a part of a separate project - Weboob.
Those interested might also check out beancount. It's double entry accounting in text files similar to ledger. Martin, the author, has written some pretty fantastic documentation around it.
* I can keep arbitrary asset, liability, and expense categories in a tree-like structure.
* I can split up transactions between different accounts. This is useful because I can split a dinner bill between my credit card and cash. It is also useful because I can split items from a large Amazon order into their respective categories.
* The ledger file is a very simple text file, so I can keep it in version control.
* I haven't taken advantage of the report features yet, but they look very flexible. You can add up any accounts you want based on regular expressions.
* You can have "virtual" accounts that can be used for budgets. For example, you can set aside arbitrary chunks of your savings account for different purposes and keep track of them separately, even though they all actually reside in your savings account at your bank.
I will take a look a hinance to see how it could be useful for me!