[dazamarquez]

Honestly I asked my question without any expectation. I thought most people scoff at it because it make me seem as a newbie/stupid person. Instead you gave me a interesting and useful reply. I now have much more leads to dig deeper in that topic. Thank you!!

[Nextgrid]

I originally upvoted your question without replying because I expected there to be a right solution for this and I was hoping some experts will chime in.

Looking back at it I'm not sure if there is a "right" solution for this (or maybe there is and the experts are still laughing) but at least here's my take on it and how I would approach the problem.

Whether it's the "right" solution or not is up for debate, but at least it will get you started. When attempting to solve a problem I recommend writing the worst, most hacky solution you can as long as it gets the job done, and then see if you can optimize it incrementally (by fetching data in bulk, etc). Throwing hardware at the problem is also a valid solution to at least delay it (sometimes the "delay" can be measured in years in which case the hacky and terrible solution would've made you tons of money in the meantime so you still end up winning).