It seems likely that they don't trust anyone else to have physical access to the machines for security reasons. Their threat model probably includes national governments.
My first "real" job was in the mid-90's; I was the first technical hire at a small Chicago ISP (EnterAct) that grew into a relatively large ISP (when I left, we were default-free peered to several tier-1 providers and had more POPs than I can name). It was great, and the team that started it --- two Big-5 accounting firm programmers --- was inspiring, particularly when it came to business strategy.
Anyways, very early on, EnterAct managed to maneuver into a reputation for premium customer support. We got that reputation by doing some concrete things differently than our competitors: we staffed an appropriate number of CSRs, trained them to be nice to customers, did a lot of gratuitous tech support for basic computer problems, and were flexible about resolving billing disputes. Sadly, a lot of those things were differentiators at the time. A couple years in and we were essentially able to hang "best customer support" on our list of features, and eventually we became the most popular ISP in Chicago largely based on that.
But something I came to notice pretty quickly: the things we were doing to earn that support reputation stopped being empirical differentiators pretty quickly. Our largest competitor, run by Karl Denninger, did us a continuing series of favors by pissing off their customers. But other large regional ISPs pretty quickly learned not to set fire to their customer base, and, by the end, I think our customer service was pretty much at par for the whole area; we were no longer truly different based on support. The reputation, however, never left.
That observation has stuck with me for my entire career. I think about it all the time. It's banal, I know: "early impressions count a lot", but there's a little more to it than that: you can weaponize an early impression by turning it into your market positioning and having some message discipline.
I left EnterAct for a job in Calgary with a company called Secure Networks (SNI), doing development and security research. For the year prior to leaving EnterAct, I had also been working with the OpenBSD project, mostly by writing all their security advisories, but also doing a bit of part-time security research. SNI operated the world's first commercial vulnerability research team, and had a very close relationship with Theo; we had a full time employee who had essentially led the first OpenBSD security audit. I went drinking with Theo many times, and vividly remember hanging out in his basement with Tim Newsham eating bad pizza and trying to find vulnerabilities in Daniel Bernstein's qmail (we found one that would work if integers were 128 bits, but ironically missed the LP64 bugs that Georgi Guninski found; it was 1997, though).
This is all a long prelude to a simple point, which is that I think OpenBSD's reputation for security works in a very similar way to how EnterAct's reputation worked. OpenBSD started doing something very different than FreeBSD, Linux, and (particularly) NetBSD: they did an OS-wide audit for vulnerabilities, and aggressively fixed apparent bugs whether or not we could demonstrate that they were exploitable. That was a great move. But it was so obviously great that pretty much everyone (with the possible exception of NetBSD) quickly adopted the practice.
Among security research insiders, OpenBSD's reputation became a little bit farcical. Not that OpenBSD was comically insecure --- it wasn't --- but that its reputation so far outstripped its actually differentiation. People found a bunch of vulnerabilities in OpenBSD and laughed as the claim at the top of the OpenBSD changed from "no vulnerabilities" to "no remotely exploitable vulnerabilities in the default install".
And at some point in the last 10 years, didn't OpenBSD's distro servers get owned up?
I'm sure the OpenBSD project would like its threat model to include NSA. But OpenBSD is not a meaningful ally in a contest between you and NSA. NSA wins that fight. OpenBSD's userland was much stronger than FreeBSD's in 1999, but I'm not sure I think their kernel is stronger in 2013, and that's probably what matters more.
Let me wind this bloviation up with a caveat: one thing a reputation for security gets you is a feed of talent that is interested in working on security problems. OpenBSD certainly got that. So for instance, OpenBSD's developers designed and built privilege-separated OpenSSH. There is a lot of good security work that has started inside the OpenBSD project, and I don't mean to talk any of that stuff down. I'd just be careful about taking the project's overall reputation to the bank, especially if you have serious adversaries.
Sorry for hanging this sprawling comment off your (simpler) point; I just don't want the root comment on the thread to be me talking down OpenBSD.
I know OpenBSD's reputation is primarily security, but I use it for a different reason. It's simple, stable, and doesn't break.
Back when I was in high school and I had a lot of free time and all that, the various incarnations of Linux were a delight. Even after that, I still went with it out of inertia and spent many evenings tweaking Gentoo.
I eventually just goddamn gave up. I got sick of every upgrade breaking something in my system and then especially got sick of deciding between figuring out how to use wpa_supplicant and installing NetworkManager which screws up my network settings as soon as I plug in the Ethernet cable while I'm still on my wireless. In a flight of rage I thought ok, I've had enough of this crap, and went the OpenBSD route.
Seriously, it has all the nice parts of Plan 9 while still actually being able to run all the tools I need. I still have Linux and Windows boxes for the odd tools that don't work on anything else (I do embedded systems for a living, and there's a lot of vendor lockdown there), but for my day-to-day workstation, I found nothing better.
In 2009, our development team lost a whole 10 hours to a degraded Linux mdadm RAID1 that wouldn't rebuild due to an obscure error after a digger severed our power and internet connection. No internet access as power came up first so no access to online help. mdadm is buggy. Documentation sucks. Error messages suck. Only recourse was a full restore from tape which took a long time. This was the last straw after over a decade of dealing with this crap from network dropouts, laziness, half-arsed features, distro wars, politics and churn.
Some previous Unix experience in the late 1990s with OpenBSD on an old SparcStation 5 (the only thing that would run on that machine nicely) jumped into my mind on the way home. It had that warm, fuzzy, well-engineered, well-documented feeling about it, like an old HP RPN calculator. Got home, downloaded it and installed it on my laptop, replacing Ubuntu.
4 years down the line: one happy person with the same laptop running 5.4 still with that warm, fuzzy, well-engineered, well-documented feeling.
Not once has it let me down. Not for a minute in the 4000+ hours I've been using it. It just works.
And OpenBSD has the best-written man pages in all of Unix.
When I got thrown in the deep end with Solaris, many years ago, I'd read the Solaris man page for the options, but first I'd read the OpenBSD man page to work out what the hell the command was for and why.
The most offensive man pages are GNU project pages that effectively say, "for real documentation read the info page". Which, as someone that can never remember how to use info, is frustrating and just serves to piss me off...my first thought is "and a big fuck you to you, too". And then I look it up online so I don't have to read how to use info before I can read how to use the command I was looking for docs on.
I don't know if this is common practice anymore...I don't remember the last time I saw a defective man page like this, but I still remember it with great anger. I love GNU, but I hate the kind of condescension it takes to try to force someone to use a different tool because you believe it to be superior to the standard tool (when it's really not; I find info pages to be obtuse to create, and difficult to read).
GNU's stance on man pages is entirely correct! For real documentation, read the info page, but you rarely want real documentation, you just want a quick example or the command-line invocation syntax, or what a particular argument does. And 99% of the time, that will be in a man page.
The problem lies when you want to find something 1% of the time, and it's here that man pages become sprawling unindexed messes. For example, take a look at the man pages for perl or zsh: you'll have no chance finding anything, as those programs are so large that they need a wealth of documentation to go into them. At the same time, the info page for ls contains the things you rarely need to see such as exactly how things are sorted or the minute details of timestamp formatting. If this were all in the man page, you'd complain that you couldn't find anything in it.
This was my impression as well after using OpenBSD, and when I pointed that out a while back on HN, it was pointed out that the core linux manpages have gotten much, much better in many cases[1]. In that respect, it may be another example of the GP comment.
1: My go-to example was always ifconfig, but linux's manpage for ip(8) really isn't that bad, as is actually the linux equivalent. Quality probably varies quite a bit based on the package that supplies the utility though, while OpenBSD's quality is fairly universal.
I wonder where the best place to report manpage bugs to is - for things like the builtin commands that may not have a single upstream. Does Ubuntu pull in a manpage update from Fedora? What about the other way around?
> And at some point in the last 10 years, didn't OpenBSD's distro servers get owned up?
Yes, a cvs bug I believe. No kernel will protect you from bad user-mode code that really wants to execute everybody's shell script.
> Among security research insiders, OpenBSD's reputation became a little bit farcical.
I spent lots of time looking through the OpenBSD Kernel, togheter with FreeBSD and Linux kernel. It was my job for years, looking for vulns and writing exploits for them.
I still admire the OpenBSD Kernel for their simplicity and tidiness.
No comparision to FreeBSD kernel-side. FreeBSD kernel often have commits of several hundreds of KBs of mostly unaudited code. They still don't enable stack-protection today in 2014. It's a joke. My windows phone had stack protection in 2003.
No comparision to Linux either, the Linux kernel is so huge, so full of code that even if it's way more audited than FreeBSD, there are still vulns lurking everywhere and exploits for linux kernel came out almost monthly. Probably it's the reason it have so many security features, more than OpenBSD nowadays.
Windows, their kernel is a work of art. Microsoft only have to fire the guy that says "hey I got a great idea lets parse some random protocol inside the kernel"
But I disgress. OpenBSD is still very good. Very safe in the default install. It will protect your firefox from being owned by a NSA-sized enemy that really want to hack you? no. But the problem is in the browser, not in the kernel. Don't use a big browser. It's not in the default install :)
Thomas, thanks for that comment. If there was a "best of HN", this comment should be a part of it. Good storytelling, a great business lesson tidbit for all of us, interesting technical discussion, and a good reality check.
I don't recall if I stayed to the bitter end, but I started making provisions for a move after I ended up arguing with Karl over whether inbound mail was being corrupted, maybe around the time of a conversion to (from?) maildir.
I think I started when it really was only Karl, was Dawn his first hire? Hmm, I probably still have the t-shirt as well.
Totally off-topic, but I remember those days. At some point, I got a copy of my customer record and saw "MCS bailers" in the referral field. Got a good chuckle over that. I don't even remember what KD did, but I remember choosing EnterAct because you were one of the last ISPs in the area that offered a dial-up shell. That was in the days when I had a Commodore 128 set up in my home office to mess around on.
Why not just do what Linus Torvalds does and simply trust his hash function? For anyone to tamper with the Linux kernel sources and have him not notice they'd have to generate a SHA-256 collision and somehow get this change past thousands of clones of the repository.
My first "real" job was in the mid-90's; I was the first technical hire at a small Chicago ISP (EnterAct) that grew into a relatively large ISP (when I left, we were default-free peered to several tier-1 providers and had more POPs than I can name). It was great, and the team that started it --- two Big-5 accounting firm programmers --- was inspiring, particularly when it came to business strategy.
Anyways, very early on, EnterAct managed to maneuver into a reputation for premium customer support. We got that reputation by doing some concrete things differently than our competitors: we staffed an appropriate number of CSRs, trained them to be nice to customers, did a lot of gratuitous tech support for basic computer problems, and were flexible about resolving billing disputes. Sadly, a lot of those things were differentiators at the time. A couple years in and we were essentially able to hang "best customer support" on our list of features, and eventually we became the most popular ISP in Chicago largely based on that.
But something I came to notice pretty quickly: the things we were doing to earn that support reputation stopped being empirical differentiators pretty quickly. Our largest competitor, run by Karl Denninger, did us a continuing series of favors by pissing off their customers. But other large regional ISPs pretty quickly learned not to set fire to their customer base, and, by the end, I think our customer service was pretty much at par for the whole area; we were no longer truly different based on support. The reputation, however, never left.
That observation has stuck with me for my entire career. I think about it all the time. It's banal, I know: "early impressions count a lot", but there's a little more to it than that: you can weaponize an early impression by turning it into your market positioning and having some message discipline.
I left EnterAct for a job in Calgary with a company called Secure Networks (SNI), doing development and security research. For the year prior to leaving EnterAct, I had also been working with the OpenBSD project, mostly by writing all their security advisories, but also doing a bit of part-time security research. SNI operated the world's first commercial vulnerability research team, and had a very close relationship with Theo; we had a full time employee who had essentially led the first OpenBSD security audit. I went drinking with Theo many times, and vividly remember hanging out in his basement with Tim Newsham eating bad pizza and trying to find vulnerabilities in Daniel Bernstein's qmail (we found one that would work if integers were 128 bits, but ironically missed the LP64 bugs that Georgi Guninski found; it was 1997, though).
This is all a long prelude to a simple point, which is that I think OpenBSD's reputation for security works in a very similar way to how EnterAct's reputation worked. OpenBSD started doing something very different than FreeBSD, Linux, and (particularly) NetBSD: they did an OS-wide audit for vulnerabilities, and aggressively fixed apparent bugs whether or not we could demonstrate that they were exploitable. That was a great move. But it was so obviously great that pretty much everyone (with the possible exception of NetBSD) quickly adopted the practice.
Among security research insiders, OpenBSD's reputation became a little bit farcical. Not that OpenBSD was comically insecure --- it wasn't --- but that its reputation so far outstripped its actually differentiation. People found a bunch of vulnerabilities in OpenBSD and laughed as the claim at the top of the OpenBSD changed from "no vulnerabilities" to "no remotely exploitable vulnerabilities in the default install".
And at some point in the last 10 years, didn't OpenBSD's distro servers get owned up?
I'm sure the OpenBSD project would like its threat model to include NSA. But OpenBSD is not a meaningful ally in a contest between you and NSA. NSA wins that fight. OpenBSD's userland was much stronger than FreeBSD's in 1999, but I'm not sure I think their kernel is stronger in 2013, and that's probably what matters more.
Let me wind this bloviation up with a caveat: one thing a reputation for security gets you is a feed of talent that is interested in working on security problems. OpenBSD certainly got that. So for instance, OpenBSD's developers designed and built privilege-separated OpenSSH. There is a lot of good security work that has started inside the OpenBSD project, and I don't mean to talk any of that stuff down. I'd just be careful about taking the project's overall reputation to the bank, especially if you have serious adversaries.
Sorry for hanging this sprawling comment off your (simpler) point; I just don't want the root comment on the thread to be me talking down OpenBSD.