[texaslonghorn5]

This and part 1 are interesting reads.

> The reality is that building systems is really hard. It requires a lot of resources, and a lot of engineering. I think the incentive structures in academia aren't very suited for this kind of costly, risky systems-building research.

I think this is the key point. I've heard similar from friends in industry and academic positions.

[goethes_kind]

The hardware resources are there. I for example have access to clusters with hundreds of A100.

The engineering resources are not though. The work has to either be done by the Phds themselves or by student workers. Phds don't have enough time and student workers have very little time and no expertise.

There are some limited SWE positions in support roles, but as you can expect the pay is not competitive.

For some reason even in academic departments where we can build billion euro clusters the people are still paid utter crap.

[sseagull]

I am really feeling this now.

We are an academic group (sort of) that is looking to stand up a server or so for some public-facing apps and databases, but it's like no one has ever heard of such a thing. They keep giving me servers that are behind a firewall and only accessible on-campus.

We could maybe do AWS or Azure, but that is really expensive and not really covered by our (government) funding. Plus, any time you want to do something like that (particularly with a company the university doesn't have an existing contract with), you have to get several layers of bureaucracy (including lawyers sometimes) involved. We are a state school, so you can guess how fast that moves.

In general, it's easier to ask for a $500k instrument, or take $10k trips to conferences, than get $10/month subscription for some web hosting company. Academia can be really behind on this front.

(For me, academia is generally ok and fun in my current position, but this part is getting ridiculous)

[shapefrog]

> In general, it's easier to ask for a $500k instrument, or take $10k trips to conferences, than get $10/month subscription for some web hosting company. Academia can be really behind on this front.

Had to fight to get paid most months; but the $2mil grant for a computing cluster went through and the Supervisor never had a problem getting their monthly trip to a conference through.

[ftufek]

Have you looked into cloudflare tunnels/ngrok? You can expose any service to internet without dealing with ip addresses, firewalls or opening ports, etc.

And tailscale can also give you a private access to any resources even if it's behind firewalls and all that (say you want to access the gpu cluster from your laptop at home).

And they can all run in user space without elevated privileges.

[g42gregory]

Why not to buy $500k NVIDIA GPU server?

[throwaway2037]

This does not address GP's original concern: <<public-facing apps and databases>>

[xwdv]

Buy the $500k server and get a VPN.

[sseagull]

We’re trying that :) (minus the GPU. Think more like a database/web server).

Although to a lot of reviewers a server is a server. “You have a multi-million dollar compute cluster available, what do you need another server for?”

[secabeen]

We are starting to build up the field of research facilitation, but it's hard to bring together permanent funding for things like this; quality engineers don't want to be on funding that may vanish in 4-5 years, and most universities don't have the culture to put the necessary central funding forward at the scale required.

> For some reason even in academic departments where we can build billion euro clusters the people are still paid utter crap.

Billion euro clusters are one-time money; people expect benefits and durable employment.

[solarkraft]

> quality engineers don't want to be on funding that may vanish in 4-5 years

I may not be a quality engineer, but I'd be interested. That's a long enough time frame for me.

But by what I hear from German academia the pay is bad, the environment is "meh" and the contracts have to be renewed in relatively short periods (yearly?). That makes an otherwise interesting field a hell of a lot harder to choose.

[indigochill]

Haven't national research labs solved this problem? Nobody I talked to at APL seemed to have any concern about shortage of R&D opportunities and although it wasn't Silicon Valley money (which is not sustainable long-term anyway), I don't recall anyone complaining about the salary either.

[nl]

There was a thread on this on HN last month: "Ask HN: Has anyone worked at the US National Labs before?"[1]

Most comments note how the pay is less than FAANG; eg the top voted comments:

"Pay is pretty good by almost any standards except FAANG."

And elsewhere:

"The pay at a DOE lab is less than FAANG (PhD student interns might be around $80k/yr and starting staff scientists maybe $130k/yr), but the tradeoff for some people would be the research-flavor of the work, and the flexibility."

etc etc

[1] https://news.ycombinator.com/item?id=34414527

[fragmede]

If we consider that Silicon Valley started in the 1970s, I'm not sure how much more long term it needs to show to be considered sustainable.

[krageon]

I'm assuming they meant sustainable for a single person. The percentage of people holding such a salary long-term could be very low. I don't know whether or not it is, but that is how I read the comment.

[secabeen]

The national research labs are great, but there are only 17 of those, and they don't employ that many people when compared to the software industry. Good gigs if you can get them, certainly.

[Nimitz14]

99% of Unis do not have Hardware like that lol

[gumby]

The R1 tier do; that’s kind of the point. But most aren’t really schools in the normal sense. E.g. MIT is basically a huge research lab with a small undergraduate school bolted on

[slowmotiony]

A friend of mine is part of the jury that decides on how to distribute millions of euros in EU research grants. There are 6 professors in the jury and the their compensation is exactly 100 euro per decision. You could essentially just make up a bullshit "project", bribe 3 of them with a few thousand bucks and secure yourself a million euro research grant.

[ModernMech]

Why would professors at the top of their field they sell their integrity for a few thousand dollars?

[discardable_dan]

You have access to a hundred A100s, at a price point of ~8500 each, for $850k. If only you could have half, and spend the other half on labor.

[oikawa_tooru_]

Damn. What's the university you work at?

[jvvw]

I've worked as a developer/engineer both in private sector and in a research department at a university (but not AI/ML). There are a couple of problems with working in an academic context.

First, as you say the incentive structure is wrong: projects are led by researchers and the incentives for researchers are around publishing not around creating good software. There's virtually no credit for making software that is well-written, robust, has a great user interface etc.

Secondly, the funding structure for projects tends to be grant-based. It's extremely hard to get funding for general ongoing development. I saw lots of projects get initial funding for a couple of years, the funding stopped and then the project had no money even for the most basic support or server costs etc. Sometimes you could grant funding for new features and could funnel a bit of that into support but it did require a constant regular stream of grant applications, as often of course you don't get them.

My salary was certainly largely funded by grants - I had a permanent job and there was some allowance for gaps between funding to cover that and because of the type of work I was doing I could do some internal work too in those gaps. It's hard to manage funding that comes with short-term grants. You just can't recruit good people to short-term contracts that quickly before they are supposed to start, especially for what universities can pay (in the UK there are set pay scales, so if you don't manage anybody you basically can't be paid more than a certain amount, plus there are strict processes about recruiting, for good reasons, so it's hard to do quickly).