Wow. Netgate come off as incredibly unprofessional.
According to the article linked and the info here in that email you linked this is my conclusion:
* Netgate tried to ship flawed code that has multiple security issues.
* Jason Donenfeld, one of the lead Wireguard developers, went out of his way to work on rewriting it to be better in time for the 13.0 release of FreeBSD
* This Netgate employee is angry that they weren't able to ship their bad code and starts throwing accusations of a smear campaign.
Am I understanding what happened correctly? Because it really makes this Firewall/Router look really bad.
That was my impression too, then I went back a couple prior messages, and looked at the earlier announcement. Wihle Netgate looks to have overreacted (at least from the info we have), I can understand why they would be upset. This was in the original announcement:
The first step was assessing the current state of the code the previous
developer had dumped into the tree. It was not pretty. I imagined
strange Internet voices jeering, “this is what gives C a bad name!”
There were random sleeps added to “fix” race conditions, validation
functions that just returned true, catastrophic cryptographic
vulnerabilities, whole parts of the protocol unimplemented, kernel
panics, security bypasses, overflows, random printf statements deep in
crypto code, the most spectacular buffer overflows, and the whole litany
of awful things that go wrong when people aren’t careful when they write
C. Or, more simply, it seems typical of what happens when code ships
that wasn’t meant to. It was essentially an incomplete half-baked
implementation – nothing close to something anybody would want on a
production machine. Matt had to talk me out of just insisting they pull
the code entirely, and rework it more slowly and carefully for the next
release cycle.
I can understand being upset if that's how you're portrayed publicly.
Keep in mind, back in February of 2020 when Kip Macy first announced that Netgate had hired him to port Wireguard, Jason offered to help. First Kip declines the offer, then seems to warm slightly to it, but ultimately appears to have not actually engaged Jason.
If I'm Jason and I offer my help (for free), they don't take me up on my offer, then try to release code that would make my baby look quite ugly, I would probably also have a pretty severe reaction.
Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him, I'm sure he was frustrated that he offered to help multiple times and they both didn't take him up on the offer, and tried to release a hatchet job with his name (indirectly) attached to it.
> Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him
Oh, I don't entirely blame him. I just partially blame him for not seeing the obvious way this could devolve into a problem, even if it would (justifiably) seem unlikely to go to this level so fast. That is, he shouldn't be surprised there was a problem with what he said, although the scope of the problem is a bit more than I think most would expect.
Professionalism isn't just about making others feel good, it's about optimizing for useful outcomes, which includes covering yourself. Not taking care with your words is just like not taking care with your code. Sometimes there's a weird interaction and things go boom.
There's not a good way for me to respond to that without going off-topic. The following is assuming that wasn't a rhetorical question, if it was rhetorical I guess we may just agree to disagree:
Until he issues a public apology for his actions, I'll refer to him as Kip. Changing your name to run from the google searches is completely understandable, and I support second chances, but you need to show a bit of remorse IMO.
Wow, I was thinking the headline was underselling the story before I got to the part about him fleeing the country after his parents bailed him out of jail to the tune of half a million.
I don't really think that the 'online mob' has the right to hold someone's past actions over their head, and expect some public appeasement before it relents.
Sounds like Jason should trademark Wireguard (the name). Or build an alternative brand. That way Netgate's actions, or the actions of other wireguard implementations, will not reflect on the reputation of his project/product/technology.
He did trademark the name. I don't think Jason is going to tell the FreeBSD project that they can't use the name "wireguard" for their implementation of "wireguard" just because Netgate put out shoddy code. It's not the FreeBSD project's fault.
I think he should exactly tell the FreeBSD project to not use the mark if they cannot meet the quality requirements (especially if the quality issues were known prior to shipping). That is assuming they actually shipped the known-problematic code, though. Which they did not do.
I don't think this is an accurate description of the responsibility hierarchy. If in fact the code by Netgate or associates/ contractors/ employees of Netgate is not of professional quality, it has no place in the FreeBSD codebase ready for a stable release. Ultimately, the FreeBSD core community/ developers are together responsible for the codebase even if you cannot hold them legally accountable because of the license. They together hand out and take back (deny) the commit rights (bit, whatever). If a highly sought after component _in the kernel_ is (at least to Jason's account) not even up to the lowest security and code quality standard it has no place in the code base in preparation for a stable or probably even a beta release. Other developers (and that's where Jason is completely in the right if his account is correct) should protest the inclusion of such possibly very bad code into the codebase more or less in late preparations for a release as I understand it.
So it is first and foremost on Netgate, if Jason is right but right after that it is on the other responsible FreeBSD co-developers. I mean having so obviously bad code in any kernel of a modern operating system release would be really, really bad. There are many people and companies dependent on it that cannot really influence anything but have to endure the consequences either way. I mean, if you buy a storage appliance, a router or a firewall you trust the quality of the product to a degree and cannot really audit much even if you had the skill. You have to take the word for it and make some reasonable accomodations. No insurance is going the replace the full damage due to lost data to an attacker or a bug. Peoples lives sometimes indirectly depend on the full chain of competence and no insurance can resurrect the dead or right the good name of anybody. Remember, most of the time when you have to update anything for security reasons, somebody didn't understand the system fully or just plain messed up. The only exception is when the problem or times / requirements have changed (e.g. the computers got so fast, we have to transition to longer keys/ passwords whatever).
So yeah, if Jason's account is accurate it is bad the code landed in the codebase at all and raises questions about the quality and security of FreeBSD. I mean, it is code directly meant for a secure-as-possible VPN and something that often directly interacts with the open internet. Surely such code should experience extra scrutiny.
From the short personal interaction with Jason he came across as quite thoughtful and knowledgeable. Over the years, he and his supporters were able to convince many not so easy to convince people about the quality of Wireguard and some of its implementations. He and the supporters have shown a long term commitment and I am for these reasons inclined to trust Jason's judgement as well.
I duno, if true about the code I find it very difficult to empathize with Netgate
From what has been said it's not like they found and fixed a subtle and cryptic vulnerability in an otherwise reasonable implementation and then failed to disclose it properly. It's more like they turned over a rock and found a murder victim. The guy from Netgate is also coming across as very inward looking and seems to assume everyone else's motivations are also purely selfish (referring to his comment implying a "shower of contracts" they might receive for the publicity). His focus should be on how to prevent this mistake from happening in future.
I mean, Wireguard in these cases is a piece of code _in the kernel_, is meant to be directly in the path of a packet coming from an untrusted network and potentially does cryptography/ security critical stuff. Besides maybe memory management and such things, with what other pieces of code do you care more about quality and security than basically a VPN? People and companies that don't even know what Wireguard or a VPN for that matter is directly or indirectly depend on this and similar code being nearly perfect sometimes with their reputations or even lives. I mean, it is not a game that is taking too long to load or something, which is quite upsetting but I cannot imagine how that would be critical. The security of a kernel or a VPN is a very different story and we should all stop pretending this isn't the case. I hope (and am quite convinced) Jason and others want only the best for the respective kernels and the Wireguard implementation from the standpoint of quality and security.
Even if all of the above is true, it reads like an elaborate insult. And that's fine if that what the author set out to do for some reason. Pretending it wasn't after the fact isn't being honest, in my opinion.
A more professional and neutral announcement could just talk about code that needs to be refactored due to some incompleteness and vulnerabilities.
To a much greater extent than in other security protocols, implementation security is a goal of WireGuard. The protocol itself was designed to support secure kernel implementations; for instance, it's designed in such a way as to not require on-demand dynamic memory allocation.
It's part of the premise of the security model of WireGuard that it has secure kernel implementations. If you're building a kernel WireGuard implementation for a major open source OS without taking advantage of the WireGuard implementation design concepts, you're not really building WireGuard; you're building a compatible fork and calling it "WireGuard".
The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.
Instead, the WireGuard project seems to have been put into a position where they had to scramble to fix up an implementation that was being pushed into FreeBSD, as WireGuard qua WireGuard. I can imagine that being a frustrating experience. It certainly didn't generate the most political response ever, but I think you'd be reaching to call it a deliberate insult.
My read on it wasn't that it was an elaborate insult, but more that it was far more denigrating than it needed to be, if he was trying to be professional. That doesn't mean it was purposeful, sometimes people just don't really associate the statements they make with how it may be perceived.
I think it could have been communicated clearly and succinctly with something along the lines of: "The first step was assessing the current state of the code the previous developer had dumped into the tree. We noticed some quality problems, some unimplemented protocol sections and more concerning, security issues with the code. Given these issues, we considered asking they remove the code, but instead Matt convinced me that we should rework it slowly and carefully for the next release cycle."
Notably, I think omission of the following inflammatory statements would have prevented a lot of problems:
- "It was not pretty."
- "I imagined strange Internet voices jeering, “this is what gives C a bad name!”"
- "the most spectacular buffer overflows"
- "the whole litany of awful things that go wrong when people aren’t careful when they write C."
Whether those entirely subjective statements are accurate, they are not the things you say about someone else's work output when you expect a useful dialogue with them, which is exactly why they are considered unprofessional.
I'm not defending Netgate's code here, or even the vehemence of their reaction and how they went about it, but merely noting that not only can I see how it devolved into this, I would go so far as to say it's obvious that this is why that type of language is avoided by most people trying to work professionally. Jason wrote some very unkind things, and Netgate blew up about it. There's enough blame here that they can both share some.
> The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.
Err, wasn't that actually not the ask, because he thought they wouldn't do so, so instead they worked it over in a short time-frame, only for it then to be removed when this argument broke out and it came to light?
I get your point about perceptions, but there's also another aspect of why I found it important and necessary to describe just how poor the code was:
When you're talking about replacing and rewriting the implementation on the eve of release, you better have a good reason for doing so. Stuffing a rewrite of security critical code into the kernel at the last minute is a big red flag. The main question that immediately comes up in that context is, "how is it possible that having a last minute rewrite would be better than the code that was there before? You've only looked at this for a week." And that's a really good and important question.
That much code churn is not something I wanted when I set out to get started with this, but it's ultimately where things wound up. Why? For exactly the reasons I described in my email. The idea wasn't to be _insulting_, but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite. I see how perceptions could view that instead as denigrating, but that wasn't really the motivation. And it's not as though anybody really is rushing to defend that code either; it doesn't take a lot to look at that and make up your mind that it was probably unfinished stuff, not coded with much love, that was committed prematurely.
It also had the, I think, positive effect of leading to more scrutiny of the review process. A few people have piped up and mentioned to me that their concerns during that review weren't addressed. And as a consequence of everything, all of the code, including the rewrite, is being removed from FreeBSD until it can be carefully examined and completed, which is really the best of conclusions.
Sure, but it's easy to clinically examine any communication and refine it with the benefit of both hindsight and low cortisol levels. My read of this situation is that everyone involved was stuck in a shitty situation; it got very briefly heated, and ended up where it should have: with another dev cycle to iterate on FreeBSD WireGuard.
If the criticism is about the code instead of the person, it should never be interpreted as an insult. Proper developers have learned to split their ideas from their ego, and as such are able to receive harsh yet justified criticism concerning their code without being offended.
Similar reaction here. My first impression was Netgate being an arse. But then when you read the announcement I kind of understand why Scott is angry. Because while the post may have been in "good faith" in an Open Development and Open Source world, it surely isn't in a professional and business world especially when the work is sponsored ( being paid ).
Jason should have informed Netgate the quality of the code is shit in private and FreeBSD dev should have told Netgate will not be shipping any of it in Rel 13.
It is then up to Netgate to decide What to do with their Rel 2.5
> it surely isn't in a professional and business world especially when the work is sponsored ( being paid ).
To play devil's advocate: Netgate isn't paying Jason, and they're taking his open source code to create a proprietary commercial project. I'd say Jason owes them exactly nothing in the way of courtesy or consideration. Could he have been more polite for the sake of being polite and community goodwill? Probably.
>and they're taking his open source code to create a proprietary commercial project.
I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.
That is judging from the two pieces of information here. Jason doesn't need to be of consideration for Netgate. There could be other communication we dont know about. I can certainly understand why Scott is frustrated.
>I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.
Their sponsored work was based off of the Linux and OpenBSD code that Jason and others wrote. And even if it didn't utilize that code, you literally can't write a wireguard client without building on Jason's work.
WireGuard is an open-source project, and an important one. It seems to me that if you want to push to create the authoritative WireGuard implementation for a major open source OS, the commercial norms need to take a back seat.
It is unprofessional and bad business to deliberately sell your customers insecure code, and you should not expect anyone to support you in doing so. Jason would have veen within his rights to warn Netgate's customers about security holes in pfSense - but he didn't even do that, he made a comment on his own project's mailing list, intentionally not naming the company, about what he was doing and why, which is entirely within the realm of "professional."
(And there is, of course, the question of whether Netgate is in any way "professional" by building on top of an open project and not following its norms.)
NetGate spends a lot on FreeBSD development, which is great, but they also spend a lot of time running smear campaigns against people who offend them, which is ridiculous. They even started /r/opnsense on Reddit just to post shit-talking memes, and camp on the namespace to this day.
Netgate is weirdly hostile to a lot of opensource stuff, which should be strange given what all their tech is built on top of. This has been going on for years. (see opnsense etc)
They've recently forked their open and closed source products, so a lot of people have been migrating to OPNSense. I've been using it for a couple months now and recommend it.
Same. I didn't really care for open-source-ness or for the anti-OPNsense smear campaigns as much, but when they announced they were EOLing the product I used I jumped ship to OPNsense too.
It does everything my pfSense install did and then some. Eg DNS blocking and IP blocking are built-in instead of needing a pfblockerng-style plugin.
The only thing I've found worth complaining about is the accordion sidebar UI thing makes it hard to middle-click-open new browser tabs, because it's not obvious which entries in the sidebar are actual pages and which are fake hyperlinks that just expand the accordion submenu.
It seems clear to me this is a case of passionate coders with different personalities struggling with the difficult work of human communication in a world with limited resources and time.
No one has to be the bad guy here or end up hostile to open source.
I mean, Linus has openly acknowledged that he has behaved unprofessionally in the emails he sends to people who are trying to contribute. There isn't anything secret here.
Maybe. It's not necessarily without reason -- if you make a lot of contributions and they are generally very well received, it's quite sensible to anticipate that further contributions will be equally well received and to be surprised if they're not.
This was made worse by the unfortunate timing -- the final release candidate is just 3 days away. Any other time, we would have gone slower, had more discussion, et cetera; unfortunately this turned into an emergency.
Jason's reply is an impressive display of de-escalation. The NetGate person's message has a lot of hostility and Jason really doesn't return any of it. Hope NetGate comes around to working with the WireGuard maintainers more in the future.
tl;dr: "The developer of the project whose protocol I'm reimplementing and whose trademark I'm using to sell things is an attacker with ulterior motives. Anyway, everyone should be respectful and leave their egos at their door. Everyone else, that is.'
This isn’t even the only NetGate debacle. There is also the mess that lead to OPNsense. All in all, I’ve sworn off buying NetGate hardware despite it looking great in theory.
> But perhaps this is a good moment to step back and ask how we got here,
and what WireGuard itself really is.
> Traditionally, network protocols are specified in a document of protocol
behaviors. Then different organizations implement that specification.
Then everybody interoperates and all goes well. In practice, it often
doesn’t go well (see IPsec woes), but this at least has been the
traditional way of doing this on the Internet, and in some ways it
works.
> But that is not the approach taken by the WireGuard project. In
contrast, WireGuard is both a protocol and a set of implementations,
implemented with a particular set of security and safety techniques.
That’s a radical departure from the traditional model, and one surely to
raise some grumbles amongst graybeards. But I believe this is a
necessary and beneficial quality for having the types of high assurance
software that is needed for core Internet security infrastructure. When
you use WireGuard, you’re not just using some protocol that is capable
of producing packets that are legible by others. You’re also using an
implementation that’s been designed to avoid security pitfalls, and that
provides interfaces for using it that mitigate footguns. In that way,
the WireGuard project is more expansive than a mere protocol project or
a mere software project or a mere cryptography project or a mere
specification project or a mere interface project. It combines all of
those things into a single unified approach. (For this same reason, the
original WireGuard paper [2] has been difficult for folks to categorize.
Is this a systems paper? A networking paper? A crypto paper?)
> Because of that, I think this was an understandable predicament. After
all, why shouldn’t a company be able to task a developer with writing
some ring-0 WireGuard code in C? And why does it matter to me whether
the code is garbage if it can at least produce protocol packets? The
reason is that the WireGuard project’s mission is wider than that. We
deeply care about code quality and implementation particulars.
> While we now have the FreeBSD code in a maintainable state, there are
other projects too that could use some attention from us.
I get that WireGuard is his baby, but sweet jesus...
According to the article linked and the info here in that email you linked this is my conclusion:
* Netgate tried to ship flawed code that has multiple security issues.
* Jason Donenfeld, one of the lead Wireguard developers, went out of his way to work on rewriting it to be better in time for the 13.0 release of FreeBSD
* This Netgate employee is angry that they weren't able to ship their bad code and starts throwing accusations of a smear campaign.
Am I understanding what happened correctly? Because it really makes this Firewall/Router look really bad.