Hacker News new | ask | show | jobs
by willarson 6897 days ago
Epi0Bauqu is definitely correct that you do not want to access google 10-50 times with each page refresh, that would be very slow.

However, with the number of pieces of data you are requesting you'll have to be intelligent with your storage and querying mechanism. Example: you will very likely want to fork your queries to google to make the entire batch of requests simultaneously instead of sequentially. When you start implementing a caching/storage mechanism you'll want to keep this in mind too: accessing memcached 50 times sequentially will be slower than querying PostgreSQL once (in general, although if the database is very large and has to be paged in and out of memory, etc, this may not hold). So you'll probably want to figure out a way to store these requests in your database, and have a single query that can recall all the relevant geocodes (I don't know the details of how the geocodes will relate to each other, so I can't really be specific), and then you will want to cache that entire request. This will mean that you will (when dealing with data you have already retrieved from google... and prefetching data that you expect to see wouldn't be a bad idea) have at worst one SQL query, and at best a quick access in memcached.

If you can't think of a way to group requests like this, I strongly suspect you will need to rethink your application. Performing 10-50 external requests per page refresh will put a serious damper on your site's performance, and making 50 database queries per page refresh or even 50 memcached queries per page refresh is probably untenable (although you could start caching the entire created pages, depending on what exactly your application does). To get sufficiently quick speed to do 50 queries, you'd probably have to use local memory (which is significantly quicker than memcached), but using local memory will open the door into a thread safety hell.