[viraptor]

> It is also worth mentioning there has never been a single case of fraud or loss attributed to "screen-scraping"

This response makes me angry. Every service worth attacking will have security problems at some point. You're running a store of bank credentials, which you have to have access to (as opposed to password managers for example which can store user encrypted data). Given enough time, one of these services will get hacked and "this has never happened before" is not going to be a good answer. Someone will be the first one.

I'm happy your service will push more banks to provide APIs. But I'm already doing bank screen scraping for myself because I don't trust services which require my credentials. I hope people consider that risk seriously.

[BrandiATMuhkuh]

I like how Jude solved the problem of the password. They do all the scraping on the phone. So no password is stored on their servers. https://blog.jude.io

[djsumdog]

This is what always worried me about services like Mint.

[curun1r]

Mint is probably the one you don't have to worry about. The service that stores credentials and scrapes bank sites is the same one that powers TurboTax and QuickBooks. Intuit spends a lot of money keeping that service secure and they're well aware of how catastrophic a breech would be.

In short, that service is run out of private datacenters, not publicly available. You'd have to compromise not just Mint/TurboTax/QuickBooks but then the FICDS service, which only has methods that take a token representing the credentials and passes back transactions, not the actual credentials.

It's the competitors to Mint who can't afford their own datacenters and don't have the many millions of dollars to put into securing their service the way Intuit does that should be much more worrisome.

Disclaimer: I used to work at Intuit, though not on any of the products I mentioned. But I have had many conversations with the people that run Mint and FICDS and I know the overall architecture used. I give my bank credentials to Mint without any worry. It would probably require a state-level entity to get in, and they'd likely need to get someone hired into that group to facilitate it. The service is very secure.

[tempestn]

I don't know, if the general level of bug-ridden-ness of their products is any indication, I wouldn't put much stock in their security. There are glaring bugs in Quickbooks for example that persist year after year, having been reported repeatedly, even as they continue to release a new version every single year with various visual tweaks and seemingly not much else.

[curun1r]

QuickBooks, even the online version, are decades old and have major issues that prevent the teams from getting much done. They took a year to do clean-up and basically didn't introduce any new features. Not much got done just due to the complexity of changing anything without causing problems.

FICDS, on the other hand, was spun up more recently and is in much better shape. I wouldn't look at an application like QBO and see much that would inform a behind the scenes service like FICDS.

The more interesting product, to me, is TurboTax. The seasonality of it allows Intuit to basically rebuild it from the ground up every year. It's really a different mindset on the San Diego campus (where TT is developed) and in Mountain View (where QBO is developed). San Diego is much more willing to take (non-security) risks with the product because small changes can add up to big wins. I remember a talk by one of the guys in charge of A/B testing on TT who said that a tenth of a percentage point increase in conversion rate was good for tens of millions in added profit.

Contrast that with QuickBooks, where the difficulty to use is actually a feature. Accountants spend years learning the app and learning the work arounds for those bugs you mentioned. That knowledge and experience becomes a barrier to entry into the field and a job skill that they're compensated for. The result is that QB/QBO is aimed at accountants with years of experience in the product. Their livelihood depends on it and they don't like change. So the teams there do as little as possible that can cause problems and know that they'll still get good reviews if the get almost nothing done so long as they don't cause problems.

It's deeply disfunctional and yet explains why you can put a lot of trust in FICDS. They too have had bugs for years. For example, they don't save cookies between scraping sessions and so are always an untrusted browser to the banks. I have at least two accounts that constantly send me 2FA tokens whenever Mint tries (and fails) to sync. But no one gets fired for not fixing bugs. You get fired for compromising the security or stability of the service, and the easiest way to not do that is to push as little possibly vulnerable code to prod. It's the anti-Zuckerberg environment...move slow and don't break things.

[tempestn]

I suppose you could have a point about the fact that (good) accountants know the work-arounds. And many of the bugs are annoyances and UI things rather than critical stuff. There are some that will really bite you (meaning, result in incorrect accounting) if you're not familiar with them. For example, the home currency adjustment in the multi-currency mode skips accounts with a current balance of 0 in foreign currency, even if the balance in home currency is non-zero, and so an adjustment is needed. If you know that, you can go in and do those ones manually (although it's a PITA), but if you don't, your taxes will be off. That seems like something that should not be allowed to persist in accounting software. It's the most glaring example I can think of, but there are others.

Ha, actually, one more - the only way to change the date format in invoices (from mm/dd/yy to YYYY-mm-dd for instance) is to change the overall Windows OS setting. That's crazy enough, but what's worse is, when you do that, it screws up the dates of many pre-existing, closed transactions in your company files. Fortunately in my case (iirc) it reset those dates way in the future, so I was able to find and fix them manually. But I mean, seriously...

[curun1r]

Heh...you're talking QBD. That's the true dysfunction of Intuit. It's the product Intuit doesn't want you to use. They want customers to transition to QBO and only "maintain" Desktop because many customers refuse to "upgrade" to Online. I never met a single employee who was currently working on QBD. A product that accounts for more that 1/4 of their revenue and they had literally no engineers participating in company events, or at least disclosing what product they worked on. Hell, I met the entire Quicken for Mac team at one point.

My favorite WTF for QBD: The data upload was literally just using the replication feature of the embedded Sybase database. One of the Distinguished Engineers that I worked with, "I worked on that feature and I probably know more about how it works than anyone and I can't get it to work reliably.

[pbreit]

what is FICDS? What does it mean "quickbooks online is decades old"?

[curun1r]

> what is FICDS?

Financial Institution something something Service. It's the internal initialism name for the service that communicates with financial institutions on behalf of the applications that need to pull transactions and other information.

> What does it mean "quickbooks online is decades old"?

The project was started in the late 90s. It's as messy a codebase as you'd expect to find when you've got close to 20 years of commits.

[dugmartin]

The level of bugs and slowness in Quicken is mind bending. I've been using Quicken since it ran in MS-DOS and I've watched it get slower and buggier every year.

My latest version (Quicken 2016) literally takes seconds to tab between fields and often does not keep a correct tally balance until I close the check register view. I only have a year of data in it (I thought the slowness might have been due to importing my old data so I restarted fresh).

Since I've been getting bombarded with upgrade ads that I can't seem to disable each time I start Quicken I'm thinking of moving to another app like Moneydance. It feel like an end of an era.

[curun1r]

First, Quicken was sold to a private equity firm at the beginning of last year, so you can't really blame Intuit for it anymore.

Second, even before it was sold, it was like Intuit's vestigial tail. It was an important part of the evolution of the company, but it didn't fit into what Intuit was trying to do and was largely ignored inside the company. Even before they announced plans to sell it, it felt like the kitchen table they keep at headquarters...something that's there to remind people of Intuit's past but feels really out of place with its present.

[mfrye0]

Agreed. I used to work for a division of Intuit (not financial stuff) and a lot of the dev work I saw was pretty shoddy. Lots of weird bugs constantly.

With that said, security is taken very seriously and most of the stuff I experienced was UI related. So hard to say...

[fweespeech]

Tbh, its only worth considering for pure "credit card" services where its essentially read-only already and any damage can be attributed to fraud you can perform a chargeback against.

That being said, if you run all your spending through your credit cards except obvious notables you update manually you are golden. (i.e. Rent/Mortgage)

[d0while]

Me too.

[gregable]

It would be interesting if there were a way to run a local client that holds your credentials and does the screen scraping to send back to this API (or others).

[thwarted]

I believe Wesabe had an implementation that did this.

From http://blog.precipice.org/why-wesabe-lost-to-mint/

   Wesabe built our own data acquisition system, first
   using downloadable client programs (partially because
   that was easier and partially to preserve users’
   privacy)

[viraptor]

Technically, they could provide a self-hosted version. But that may not be a very profitable business. Also, screen scraping needs continuous updating as the bank's interface changes.

[echion]

> I'm already doing bank screen scraping for myself

Any tools you recommend, or is it all hand-rolled?

[viraptor]

The best option I found (for read-only access) was chromedriver > download CSV statement > process that into your own database.

Using minimal standard screen scraping will be hard because most bank apps are a complicated mess of client and server side processing using ancient tech. After many tries, I just used python+chromedriver and that's doable in ~100 loc per bank.

CSV statements because OFX and others are a complete mess in most cases. I haven't seen a single compliant and correct file so far. (This may be AU specific)

Own database / spreadsheet so you have a local copy you can easily filter on. Obtaining specific date range from a bank may be an interesting challenge. I default to importing last 30 days every day with duplicates detected using a hash(date-description-amount-idx) where idx increases for every date-desc-amount duplicate.

So far I'm happy. Actually I'm in the process of writing a post about it - will submit sometime next week most likely.

[mkurz]

Looking forward to that blog post since we need to solve something similar.

[deejbee]

Coming from a slightly different angle but it may be of use, I wrote a chrome browser extension that automates logging into banks and other financial sites. It also scrapes the balances once logged in to provide a net worth total. It currently works on around 25 banks (mainly UK). You can also build your own logins and all passwords/private data are stored encrypted in the browser.

I wrote it as I just spend too much time digging out passwords all the time and the usual crowd like LastPass are a) stored online and b) fail to login automatically to complex input sequences like banks.

[smichael]

I hand-rolled a wells fargo scraper recently using the haskell webdriver package and Selenium. It was working fairly well until they started requiring a captcha - something I've never seen as a human user. I'm not quite sure what triggered that or how to get around it.