[nilkn]

A hypothetical example would be one person spending a week or two setting up an extensive in-house Spark cluster to solve a problem that a second person in one day realizes can be solved on one machine using some clever shell scripting tricks. The former person knows a lot and may have been fully capable of the solution the second came up with, but they followed the wrong guiding principles in analyzing the problem and arrived at a solution of considerable complexity.

An alternate formulation of this would involve the second person above arriving at a new job, observing that they're using Spark on an expensive cluster to regularly perform a computation, and noticing that actually they could do the whole calculation on a single node using simpler tools.

[oarabbus_]

Thanks for the example. Once I did some analytics consulting with a stealth startup with some very talented engineers. One of them was usingusing rsync to replicate their data from their eng server to the analytics server, and I (not a software engineer, but familiar with shell) was initially like "that's it? You're just using rsync instead of <complex, paid replication tool>?" But it was the simplest and most straightforward tool for the job.

[saberience]

You could answer that example problem just by being someone who reads Hackernews a bunch, since the "replace spark cluster with random unix tools" article appears here regularly. IMO, that doesn't really define being a principal engineer, it's basic.

[foobarian]

Principal engineer: arrives at company, notices expensive Spark cluster, replaces with one server running shell scripts

Double digit-strong data science team: speechless

[nilkn]

I don't even agree that this is a defining quality of a principal engineer. This is part of our standards for a senior developer, and it's something we coach people on from the get-go.