Hacker News new | ask | show | jobs
by oarabbus_ 2424 days ago
Would love to hear some examples of experts decreasing complexity and advanced engineers increasing complexity.
3 comments

A hypothetical example would be one person spending a week or two setting up an extensive in-house Spark cluster to solve a problem that a second person in one day realizes can be solved on one machine using some clever shell scripting tricks. The former person knows a lot and may have been fully capable of the solution the second came up with, but they followed the wrong guiding principles in analyzing the problem and arrived at a solution of considerable complexity.

An alternate formulation of this would involve the second person above arriving at a new job, observing that they're using Spark on an expensive cluster to regularly perform a computation, and noticing that actually they could do the whole calculation on a single node using simpler tools.

Thanks for the example. Once I did some analytics consulting with a stealth startup with some very talented engineers. One of them was usingusing rsync to replicate their data from their eng server to the analytics server, and I (not a software engineer, but familiar with shell) was initially like "that's it? You're just using rsync instead of <complex, paid replication tool>?" But it was the simplest and most straightforward tool for the job.
You could answer that example problem just by being someone who reads Hackernews a bunch, since the "replace spark cluster with random unix tools" article appears here regularly. IMO, that doesn't really define being a principal engineer, it's basic.
Principal engineer: arrives at company, notices expensive Spark cluster, replaces with one server running shell scripts

Double digit-strong data science team: speechless

I don't even agree that this is a defining quality of a principal engineer. This is part of our standards for a senior developer, and it's something we coach people on from the get-go.
Designing a delta-sigma modulator.

Team engineers: we need a high-speed design with advanced adders and someone who can do the difficult static timing analysis.

Expert: No, you need someone who actually understands VLSI design who can interleave time and space in an algorithm. At that point, you can basically use a single step up from stupid-simple ripple carry adders. And if you use non-overlapped clock generators you don't even have to do static timing analysis. The biggest wins are in architecture, not implementation.

Yeah, 33MHz design in 125nm means VLSI like it's 1999.