| Stock analysis program. I scrape a stock site using Java and load the data to mySQL database in my computer. I have cloud functions to scape any new announcements and send mail to me. The site I had built is pretty much scalable meaning announcements are stored in Firebase and bootstrap 4 website logins are handled by Firebase as well and each user can track the stocks, get announcements related to their stocks and see all the financials and see the reports as well. The roadblocks been, the source site from I scrape they change their site design often due to which I loose data or had to rework which some times takes couple of months. Data is sometimes disconnected like if the stock is part of a index the main page is updated with main items like revenue and net profit but the details pages are not updated and they might be updated within next 5days. So scraping takes lots of time in my home computer amd insertion takes time as well. Initally I was generating around 50 reports, after site upgrades due to change in data presented data, currently have around 35 reports. Since I am not seeing that many reports currently I am going to drop more reports. Scraping part is the part I would like to move to cloud but currently I don't have enough time but 3yrs back I was working like a machine 9hrs office. 5-7hrs each day on this. Tech stack all of it I had to learn
Scraping : Java, Python (introduced a year back);
Analysis : Java & SQL queries
. Currently trying to add pandas;
Realtime scraping : NodeJs, Firebase in GCP;
UI : initially generated reports using Java velocity, Js datagrids in bootstrap. Later developed a full fledged bootstrap 4 web application;. Spent RIDICULOUS amount of time on this. Worse thing is when I started there wasnt much of apps did what I tried to do in report perspective. Now there are apps which does almost 50% of analysis what I do. Now the thought process is do I want to proceed ? No. I am trying to continue the scraping part but sort of minimize all the UI stuff and when some analysis is required write a query to get the data and subscribe & pay the app which solves 50% of my work. Due to this I didn't focus on any other areas in life which right now is lot frustrating. Sacrifices I used to be a marathon runner, not anymore. Used to workout regularly 5days a week in gym, not anymore. Used to have social life, not anymore. Another interesting thing, it's a stock webapp which I was building, I was soo into the programming in bringing this up actually I didn't actually get time to review the reports. I did miss a lots of Market Ups.... |