Hacker News new | ask | show | jobs
by koliber 699 days ago
Reduce how often something runs and the amount of work it does when it does run. Sometimes, things are done too often, or process unnecessary data.

I once had a project where a SAP system was crawling and literally causing company-wide stoppages. We found a job that literally ran every minute of every day and it processed a table that contained a few thousand tasks. This was something that could be done once per hour, and only during business hours. Furthermore, it was re-processing thousands of records each time it ran. In reality, after a record was processed once, it could be deleted from the table.

We emptied out the table, and scheduled the job to run hourly. The whole company noticed an immediate improvement.

This pattern happens a lot. Someone builds a polling system that hits the server once a second in order to see if a task finished. A cron job runs every 5 minutes. All data is processed in a daily job, instead of doing a 24 hour cutoff.

The world is filled with computers doing useless work. Mostly, no one notices.
2 comments
ozim 699 days ago
How it usually goes for me.

I ask business how often should something be updated - they say “real time” (not going into details what real time means really) - it is hard to explain processing all data all the time so everything is fresh takes forever….

After couple of months it turns out they never ever open their “super important dashboard” or do it once in 6 months.

Great after a year of bogging down everything I can clean up and make jobs run once a day instead of once a minute because now it is acceptable.

link
jononor 699 days ago
Why take the unspecific "real time" answers as gospel? Just propose daily initially (or weekly or monthly), listen to their protests, and see if there are good reasons why it needs to be more often. Then pick a suitable interval that fulfills their needs and that you can guarantee. If you offer pink fluffy unicorns for free, people will always pick them, without thinking.
link
ozim 699 days ago
Well it is not clear from my initial post but I do propose daily or whatever else than just "every SeCoND BecAuSE we NeeD iT NOW!!!" but people that are clicking the system right away expect stuff to happen "right away" - and any amount of explanation does not work.

Bonus points for trying to explain you want to implement "eventual consistency" - keep in mind I have added {“real time” (not going into details what real time means really)} in text to indicate that I am not some junior whining around but someone with deep understanding of computing...

link
NicoJuicy 699 days ago
Just an alternative related to pulling.

We had a task that checked every minute what could be executed.

Instead of doing that, we scheduled the executing code to execute on that time ( if it didn't exist yet) through service bus. Since events can be planned...

Easy peasy and avoided a micro service named "scheduler", a db and an Serverless function... ( Hoboy)

link