Hacker News new | ask | show | jobs
by rs23296008n1 2356 days ago
That is the choice of software tool. You are literally grinding through gigabytes of data. If your tool didn't require so much data processing it would be faster. This may or not be possible to improve by you. Often there are workarounds.

Case in point: a matrix library I used to use needed to a full row/column pass each time. We put a layer in between it and our code. Reduced lookups required by 30%. We were processing the same amount of data and getting the same results but requiring far less time. That layer also reduced memory requirements. Now we could process larger datasets faster with the same hardware. Thats just one example.

Your choice of CPU and other hardware isn't always the limiting factor. Even the language choice has an impact. Some languages/solutions require more data processing overhead than others to get the same final result.

Even the way your program's Makefile or module composition can have an effect on compiling performance. I remember the use of a code generator we included that meant it had to regenerate a massive amount of code each run due to its input files being changed. We improved it by a ridiculous amount simply by hashing its inputs and comparing the hashes prior to running the code generator. Simply not running that code generator each time meant we sped up the build significantly. 30 minute build times reduced by 5-10 minutes. Same hardware. And that was easily triggered by a trivial file change.

2 comments

I understand your point, I think. c++ has an inefficient build system, and over time projects can end up with very suboptimal build systems. it's definitely worth spending time to pick the low-hanging fruit like in your example, or if possible, to choose a language that builds faster.

still, even twenty minutes is a long time to wait and see if your latest change actually works. in the foreseeable future, there will be complex projects that take a long time to build. you will eventually have to touch things that everything else depends on and recompile most of the code. people that work on these projects can always benefit from faster desktop-class hardware.

That's just picking the lowest hanging fruit which is quite common sense. Not to diminish your achievement but if tomorrow, say, I figure Rust's compiler is slow, there's nothing I can do [in the foreseeable future].

Having much better hardware in these cases helps you be actually productive and not twiddle your thumbs waiting for the compiler.