[gambiter]

I wouldn't call $84k/year "well below" market. Especially considering that the project isn't under heavy development. Also, I have to echo what other comments have said... he seems to be spending it oddly. He's choosing to live somewhere else, even though he seems to think monthly physical proximity to the servers is important. I currently live near work... it would be like me choosing to move 1000 miles away and then complaining that my salary won't cover my travel expenses.

There are at least 4 simple fixes I can think of:

1) Move him and his family closer to the data center (Perhaps difficult)

2) Move the servers to a data center that's closer to his home (Not as difficult)

3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)

4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)

[vinceguidry]

You can't always treat humans as resources. Some fields are so complicated that you need superheroes, or else you need to build an organization to do the work. NTP is that complicated. What it's doing is relatively simple, but the surrounding infrastructure you need is ridiculous, and the security concerns enormous. If there's a vulnerability it affects the entire world. So you can't just find someone to replace him.

If you go with a superhero, then you have to deal with him. You can't just say, "Sorry bud, we need to uproot you and move you in-town." Even if he doesn't say no, which he very well could, leaving you and your plans fucked and the world unprotected, driving all the enjoyment out of the job will make him eventually say "fuck it, I'm retiring."

Okay, let's go with an organization. Who is going to build it? You still need the superhero. The critical nature of the infrastructure demands you use the guy whose been covering your ass all this time, because most anybody else you're going to find is going to fuck it up, and there can be no possibility of a fuck-up.

And you can't buy a superhero for any amount of money. There just aren't that many of them.

[moe]

Some fields are so complicated that you need superheroes

If there's a vulnerability it affects the entire world

Who is going to build it? You still need the superhero.

Hyperbole much?

If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.

Gladly NTP is not the mythical voodoo rocket science that you make it out to be. Most large corps run their own NTP servers, some of them public, e.g.:

    time1.google.com
    time2.google.com
    time3.google.com
    time4.google.com

You can also ask your friendly government for the time: http://tf.nist.gov/tf-cgi/servers.cgi

The official NTPd impl doesn't even have a very good reputation (cf. the recent debate about that security vulnerability).

As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing. Throwing even more money at a single guy doesn't convince me as a good way to improve the situation here.

PS: And I don't mind him being paid well at all. I'm very much in favor of important OSS projects getting sponsored and rewarded. But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.

[vidarh]

> If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.

Which is why he for a couple of years have tried to get corporate sponsors for a foundation to hire more people to work on it, but even that has been met largely with apathy.

> But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.

If you think that's all he does, then you don't understand what he's doing.

You also seem to have missed that $7k also covers all his costs, including hardware replacements and hosting for parts of the servers.

[vinceguidry]

> Most large corps run their own NTP servers, some of them public, e.g.:

Doesn't fix the problem. They have to sync to something, someone's gotta maintain the connection to actual atomic clocks. Something has to secure that connection. Someone has to maintain that something. You're severely underestimating the scope of the problem.

> As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing.

They should, but they probably won't, though they might as a result of this article. That would make protecting the world Google's responsibility and it doesn't make good corporate sense to do that. Google would have to find a real reason to throw $X0 million a year at this. The estimate given by Father Time was $4 million a year, that will only make sense if he builds it himself. If Google does it it will cost a lot more.

This is a resource allocation problem, one traditionally solved by governments. Nobody wants the government to do this, so we have to find some innovative way to fund critical technology projects.

[moe]

They have to sync to something

As I understand it, they usually sync to GPS?

You can buy those boxes on Amazon[1], starting at around $299.

Google would have to find a real reason to throw $X0 million a year at this

$X0 million?!

To construct and run their own atomic clock, or what would they spend so much money on?

I may indeed be completely missing what this guy does. He runs an actual atomic clock in that rack, like the one from NIST[2]?

[1] http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B00...

[2] http://en.wikipedia.org/wiki/NIST-F1

[vidarh]

The server you are showing is a stratum 1 time source intended for LAN use. It's not at all a given (and arguably not likely) that it would be suitable for exposing directly on the public internet to handle high traffic volumes. If not, then that means putting an ntpd in front of it, talking to it over the LAN, in which case you no longer have a stratum 1 time source available publicly.

I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.

[moe]

I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.

Um. We have "reliable publicly accessible stratum 1 servers".

Regardless of what happens with the NTPd software, Google will probably continue to provide time1-4.google.com.

NIST will not shut down their timeservers.

And the 3654 servers in pool.ntp.org[1] (which seems to be maintained by a different guy) also won't just disappear overnight, though I'm not sure if these are Stratum 1 (probably not).

I still don't understand what exactly this guy is doing that should cost more than the $7k/mo that he's getting, much less the "$X0 million a year" that vinceguidry wants to allocate to the task.

[1] http://www.pool.ntp.org/zone

[vinceguidry]

Reading the NTP wiki page leads me to believe he's maintaining several reference clocks that can be used if you have no atomic clocks of your own. I would guess that he has to maintain servers with different implementations of the time algorithms so he can test new code against them. He needs to be able to simulate network configurations so he can reproduce issues.

[xenophonf]

> Gladly NTP is not the mythical voodoo rocket science that you make it out to be.

Neither is DNS, but running the root DNS infrastructure (which serve hundreds of millions of users) is a wee bit more complicated than hosting forward lookup zones or even a public caching resolver on a handful of corporate nameservers. So too with the stratum-1 and stratum-2 NTP servers that serve as national or regional standards. They are not simple bits of kit that one throws together and shoves onto the public Internet, especially when they are ultimately used by just about every modern Internet-connected computer out there. That this guy's doing not only all of the software/release engineering but also operating some critical time servers for USD 84000 per year is too good of a deal to be true---and that he's running through his personal savings to finance the NTP project is just shameful on the IT community's (our) part.

[gpvos]

OpenNTPD, chrony, etc., do not have all of the functionality of ntpd.

[brohee]

  3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)

You don't seem to get that stratum 1 servers require connecting to actual hardware as the time source (stratum 0, typically GPS). I don't know of any VM seller with provision to hook actual hardware... And even if they did, I'm not sure they could offer the low latency needed.

For some stuff, you just need your own rack.

[msandford]

Agreed!

This goes to show that the person making the comment is thoroughly unqualified to make any kind of substantive comment on anything that this guy does.

[moe]

For some stuff, you just need your own rack.

Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.

Honestly, I have a hard time believing there is new NTP hardware coming out every month that the maintainer then personally has to write drivers for.

And even if that was the case, what prevents that hardware from working when it's located, say, on a desk in the house of the maintainer (I think most timeservers are fanless and don't consume very much energy)?

Perhaps there's valid reasons and explanations for his monthly trips to that datacenter, but the article only left me with questions.

[vidarh]

It would be nice if people questioning what he's paid would ask questions first, before making claims about what he should be doing...

> Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.

For starters, the majority of the servers are currently hosted for free at isc.org. You'd save a tiny amount on his monthly travel (I don't know what his costs are, but I used to stay in Palo Alto for weeks at a time in my last job, and it should be doable for him to travel to, and stay in, San Jose for a week for <$1k/month), in return for a as much or more in hosting costs unless you can find someone willing to give him free space.

That's assuming you can find a nearby suitable data centre.

> I wonder what prevents that hardware from working from, say, a desk in the house of the maintainer

Nothing. If you're willing to fund sufficient low latency, high capacity, redundant internet connections to his house. The price would almost certainly be far higher than flying him to San Jose once a month to maintain the servers there.

[moe]

If you're willing to fund sufficient low latency, high capacity, redundant internet connections to his house

What for?

Does he run public timeservers himself?

[xenophonf]

> He could start using VMs like the rest of the world

High-accuracy/high-precision NTP servers are extraordinarily timing-sensitive, never mind the fact that they must be connected to some kind of external time standard. None of the currently available hypervisors support the kind of precision required by a stratum-1 NTP server. Even low-precision applications---such as Active Directory, which can handle clock skew of up to 5 minutes by default---recommend against running time servers in virtual machines.

To get an idea of the engineering tolerances involved, please refer to "Clock Quality" in the NTP FAQ:

http://www.ntp.org/ntpfaq/NTP-s-sw-clocks-quality.htm

[lukeschlather]

If he were working directly for Google or another bigco doing the same work, he'd be making more in the neighborhood of 200k/year, if not more. Someone skilled enough to do this work is worth considerably more than $7k/month.

[gpvos]

2) The servers are currently hosted and maintained for free. Hosting them nearer to him would add an additional $1400, according to the article.

3) We are talking about accurate time here. Using VMs is not so conducive to that.

4) You would need someone as knowledgeable as him, and preferably as passionate about accurate time. And he's also doing a bunch of other things.

[wglb]

A VM is the last place I would want to hold an NTPD of any reference standard.

[vidarh]

> I wouldn't call $84k/year "well below" market.

You might not call it that, but for someone in devops with that length of experience, it's <50% of market value.

> Especially considering that the project isn't under heavy development.

According to the article, he spends ca. 100 hours a week on the project. It may not be under heavy development, but there's a constant stream of both development work and ops work.

> Also, I have to echo what other comments have said... he seems to be spending it oddly.

That is only relevant if you want to account for part of the $7000 as something other than salary, in which case he's even more woefully underpaid. Which he is anyway, given that he's also spending on some of the hosting and replacement parts out of his own pocket. Even so, his arrangement is not at all unusual at his level - I used to work for someone whose previous job involved flying from California to D.C. once a week for a couple of days. The cost of a travel arrangement like that is peanuts compared to his value. Especially given there's no established office or other admin costs.

> 1) Move him and his family closer to the data center (Perhaps difficult)

Why do you think he'd want to move for a job that's so underpaid, to one of the highest living cost regions in the world? Chances are good this would increase his monthly outgoings by more than he'd save on travel and board.

> 2) Move the servers to a data center that's closer to his home (Not as difficult)

Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it? If you can find someone with sufficiently low latency connection to major interchange points near him that are willing to host the bulk of the servers for free, I'm sure he'd be ecstatic to hear about it. If not, it'd almost certainly cost more than it'd save. It'd also represent a major risk to key internet infrastructure.

> 3) He could start using VMs like the rest of the world...

For stratum 1 servers? You clearly don't understand what it is he's doing. Stratum 1 servers needs a reliable physical connection to a reliable time source (such as a GPS receiver). He needs dedicated hardware. Try to get data center employees to properly handle non-standard hardware like that. I dare you.

> 4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)

Good luck with that. Pretty much anyone prepared to take on that job for $7k/month including covering server costs and hardware repairs, is going to be woefully underqualified. In any case, if the situation doesn't improve, we'll find out soon given that he set an April deadline before he'll look for other things to do.

EDIT: Also note the part where it is unclear whether the $7k grant will even continue after April.

[gambiter]

> he spends ca. 100 hours a week on the project

So he is apparently a family man, but he also spends 14 hours a day, 7 days a week on a project that hasn't largely changed in ages? Something doesn't add up.

> Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it?

It happened once. It could happen again. I refuse to believe there is no other organization willing to offer the same deal.

> For stratum 1 servers? You clearly don't understand what it is he's doing.

Yep, I realized my mistake after writing that, and the replies have made it clear that I was wrong. I still think there are options (remote access cards, for one) but I agree that VMs aren't the way to do it.

> it is unclear whether the $7k grant will even continue after April.

If that's the case, we'll see what happens in a month. If everything comes crashing down (it won't) I'll change my mind, but for now I believe one person being touted as the ONLY man who can do a particular job is incredibly wrong. Even if he were literally the only person on earth with the knowledge, it wouldn't be acceptable for everyone to rely on him... instead we should produce something as a community to replace this system. It's just not an acceptable option to have something this important in the sole hands of someone who could be in a plane crash tomorrow.