[cxr]

NB: Jason is referring to "S2" aka Samsung Austin Semiconductor. I'm familiar with Jason's previous comments in threads about Samsung, but outside of that I don't know him (even, say, by name). This makes sense, given the separation between the work of an SE manager and the actual day-to-day operations that are involved with getting wafers fabbed and shipped.

I worked at SAS in lots of different areas—Metro, Photo, and PIE (process integration).

The emotional defense above notwithstanding (which I'd characterize as not actually being supported, either...), SAS indeed has very, very deep cultural problems that make for a shockingly bad example of how to do engineering in a large organization. There are several factors contributing to this.

SAS doesn't treat information systems, or things IT-related generally, as an engineering concern. If it doesn't look like a materials science problem, SAS doesn't have the capacity to critically evaluate its impact on rates of production and yield.

Problems, though, range from lots of things, like:

- far too many things in-fab being handled with pen and paper

- reports that have to be tediously assembled manually, so they get sent out once a day, while realistically they should be available to an engineer at any given time (behind a button that generates them on the fly instead of throwing $45k/year technicians at the problem like a 1960s-era secretary pool)

- workers committed to this state of affairs remaining so indefinitely

- Conway-style balkanization and groups who have no idea what their role is ("are they asking me or are they telling me?")

- a bunch of dumb macho shit that is used to prevent critical productivity problems from being addressed because being lazy and not addressing it is somehow virtuous and not lazy—it's the application of basic engineering principles to eliminate a problem that's lazy!

- an effect that I have no way to describe except as the proliferation of mid-40s to 50s men of mediocre talent (to a degree that's even worse than what you can expect to see elsewhere, generally)

- an overreliance on the labor pool of people with military experience / government work for no reason other than they've been conditioned to survive in the kind of bureaucratic senselessness that SAS exemplifies

- unabashed credentialism, which gets people hired who shouldn't be

- Microsoft Office products being abused for everything—including areas where a different Microsoft Office product would be the most appropriate tool for the job (reaching for PowerPoint when the problem calls for Word—or any word processor)

- work showstoppers caused by truly ridiculous problems, like Windows crashing a half a dozen times a day; Caps Lock on a fab tool keeping people from remoting in from within the office, so they have to suit up, go into the fab, press the Caps Lock key, and return to their desk and resume work; lots of monkeying around just to share a file with someone, because everything has to be ferried through a proprietary document sharing system that amounts to a crippled, quasi-web-based thing that wants to be Windows Explorer-cum-Google Drive

- unit parts (i.e. manufacturing divisions) basically re-discovering/re-inventing the value version of control and implementing it in an ad hoc way for the tools that they ineract with, poorly—not to mention all opportunities for human error (which does happen regularly but no one does anything about it because 40% of the job is CYA and the CYA approach tends to be "don't let anyone find out this happened")

- tons of janky-ass software from S1, like "Simax", which is written in a proprietary Lisp-ish(?) that uses ActiveX to run in an IE9-10-11 window and that re-invents its own window management and basic form controls, was never fully localized into English (so you're clicking buttons with terse messages in Hangul that still don't make sense even if understanding the language is no problem), and times out and ends your session if you look away from it for 10 minutes (or whatever)

- overall just really bad engineering discipline; people putting wafers on hold and not being clear about they're expecting to do with them next or what they need someone else to do; people inventing new acronyms all over the place (because job security and also "mediocre talents") and using them liberally in the notes they do leave for others; people not documenting why something happened, which also happens to help when covering something up and hoping that no one has the time budget to untangle things to the point where it's obvious/incontrovertible that something did go wrong; also lots of people not having a dedicated workstation of their own, so when shifts change they have to be out of their seat so it can be ready—no putting in a little extra time to make sure that the kinds of fires that were put out that day won't just happen again tomorrow and the day after, ad infinitum

[astrange]

> - tons of janky-ass software from S1, like "Simax", which is written in a proprietary Lisp-ish(?) that uses ActiveX to run in an IE9-10-11 window and that re-invents window management and basic form controls, was never fully localized into English (so you're clicking buttons with terse messages in Hangul that still don't make sense even if understanding the language is no problem), and times out and ends your session if you look away from it for 10 minutes

I've always been surprised when I see things like this, from websites with all the controls reinvented to MDI-based Windows apps. Where do they get the _time_ and designer resources to waste all their time doing this?

Maybe I just learned programming in a different way, since one of the first things I learned was "laziness is a virtue".

[ravel-bar-foo]

These browser apps are relatively common inside Korean companies. The IE base dates to the pre-crypto-export days, when the Korean government mandated crypto be provided as an ActiveX module. This requires specialized secure input forms within the browser and within-browser error popups, so why not build a within-browser windowing system...

As a user, interfacing with these systems was always hell. Setting up online banking on a new computer would take the better part of a day, and would fail if you didn't have IE security exceptions set, if you were missing Korean font packs, or if your name was too long. And different ActiveX controls installs were required by every webapp. Our company (healthcare) finally rolled out a native Windows app in 2019. It includes its own floating tile manager, and for tasks like viewing PDFs, browser frames are now embedded in a tile...

[asdfasgasdgasdg]

I don't know the lineage of this particular system. But I have worked on janky systems in the past. In one instance, an intern built a language during a summer project. It was an expedient solution to a problem that a team was having, so they began using it. Then they put work into making it faster and integrating it to other systems. Eventually they wrapped a service interface around it and reexported it to other teams. But there were serious semantic problems with the language as a concept and with the technical implementation of the language. Once it had users who came to depend on its bugs, it became very hard to fix, and the whole mess became more convoluted as people tried to build sensible extensions on the rotten core.

It's finally gotten to the point where there is the action potential to fix it in my case, but for a lot of systems with a complex and expedient lineage like this, it never gets there. So garden well, I guess, is the takeaway.

[ethbr0]

Almost every instance of this I've come across can be described by repeated localized, short-term optimization.

No one ever sat down and said "How can I design a terrible system that's a pain in the ass to work with and unable to be extended?"

Everyone sat down and said "How can I make that one thing that's actually my job faster?"

[seanp2k2]

sometimes people also sit down and ask "how can I build a platform to do this kind of thing so expanding and maintaining that platform can be my new job here?"

[ethbr0]

Sometimes that's the end result, but I'd question if that's frequently the actual goal.

Never ascribe to malice, that which can be described by laziness, etc.

It seems like it would take more work to specifically design a system to ensure job security than to just haphazardly design without coherent architecture, and consequently be the only one who understands it.

[patrickk]

Your experience closely matches what I’ve seen in another Korean company, in a different industry. Tedious manual reports (for cover your ass reasons), PowerPoint used for everything, crap internal tools because of a strong ‘not invented here’ culture, mid 40s to 50s guys of mediocre talent but often inflated egos, and so on. Thanks for typing it up.

[sidewndr46]

I've personally known a number of SAS employees and even more secondhand. My general impression was that it was dysfunctional.

One guy I know works 6 days a week, Monday-Saturday. He goes in on work around 2pm Sunday to get caught up. This isn't the 'crunch' schedule. It's just his regular work schedule.

[jason-phillips]

I hear what you're saying and I totally get your frustration. Many of my co-workers also shared similar frustrations.

In a massive operation running 24/7 that never stops to catch its breath, it's chaotic. There are certain techniques I adopted that helped me be successful there, but I definitely understand your frustrations.

[cxr]

> In a massive operation running 24/7 that never stops to catch its breath, it's going to be chaotic.

No, it isn't. Describing it like this is needless apologetics.

The "chaos" is not even necessarily the worst part of the problem. It's these kinds of just-so dismissals that have the effect of framing the chaos as somehow unavoidable—the response that goes, roughly, you'll learn; this a natural consequence of what we're doing; it's a big operation, and this is what it looks like when you play with the big boys. I can tell you: I am a big boy. I have too much experience to the contrary for these casting couch excuses to work. Much of the day-to-day toil at Samsung is inexcusable and entirely avoidable. There is no excuse, for example, for dealing with version control in a billion dollar manufacturing operation like we're living in 1995 or a neverending sophomore-level group project.

[ethbr0]

I liked your framing of version control requiring a culture of encouraging admission of failure.

I'd never looked at it that way, and it succinctly describes why the least devops-mature organizations I've worked with have been those that heavily penalize any admission of failure. And vice versus.

[pm90]

Cultures where failure is punished are just the worst. It’s not just the vain pursuit of something unachievable (systems will break) but all the effort involved in covering up rather than simply admitting the problem and then coming up with solutions to fix it/not repeat it. It’s a lost opportunity to improve the system.

Unfortunately, without active checks and balances, orgs will tend to the blame game. Avoiding it requires an org-wide commitment to openness around failures.

[dogecoinbase]

I genuinely hope you can appreciate how funny this exchange was, in which you asserted there were no problems, someone pointed out a bunch of specific problems, and then you said "sure, I knew about all those problems".