Hacker News new | ask | show | jobs
by hacker_newz 887 days ago
> used two levels of caching (Redis and a typical relational DB) to subvert rate limiting/IP blacklisting

How do those subvert rate limiting / blacklisting?

1 comments

The Robinhood API is of course the API for stock broker Robinhood. Their API, last I used it, probably could have been optimized to provide data in fewer API requests, but aside from that, there are many pieces of data that change constantly, like the current price of a stock. There are pieces of data that change occasionally, like the next earnings report date for a company. Then there are pieces of data that should never change, like a company's CUSIP (essentially the primary key in a database of stocks). When you create an app that relies on data from another API, there is usually a "setup" where all the data your app requires is brought in from the API. I was getting rate-limited during this process (downloading all my watchlists and current holdings, as well as my orders/transactions, metadata about the companies such as the next earnings date, and so forth).

By using Redis and MySQL, I minimized the number of calls the app has to make directly to the Robinhood API. For things like the next earnings date, which changes typically once per 90 days or so, I have Redis store the result of that endpoint from Robinhood for some number of days, probably not 90, but not 1. For things like the current price of a stock, Redis might store that for a minute or two, so that my app can "spam" for the latest prices (maybe I have a refresh button in the app) without triggering a 401 or 403 from the Robinhood API.

TLDR: Since rate limiting prevents my app from pulling all the data it needs at one time - it has to throttle requests - I don't want to have to wait a couple of minutes each time the app is started because it needs to download data from Robinhood, much of which probably hasn't changed since the last time I started the app.

> Then there are pieces of data that should never change, like a company's CUSIP (essentially the primary key in a database of stocks).

I hate to break it to you, but tickers and CUSIPs do change.

The world of 'corporate actions' turns out to be a catchbasin requiring human intervention.

Yeah fair enough. I see a ticker change somewhat often (eg ETH to ETD, FB to META), CUSIPs change much less often, but unlike tickers CUSIPs aren’t reused. A company’s CUSIP will always be their CUSIP, but it might not be their current CUSIP.