[Jonnax]

Wow. Netgate come off as incredibly unprofessional.

According to the article linked and the info here in that email you linked this is my conclusion:

* Netgate tried to ship flawed code that has multiple security issues.

* Jason Donenfeld, one of the lead Wireguard developers, went out of his way to work on rewriting it to be better in time for the 13.0 release of FreeBSD

* This Netgate employee is angry that they weren't able to ship their bad code and starts throwing accusations of a smear campaign.

Am I understanding what happened correctly? Because it really makes this Firewall/Router look really bad.

[kbenson]

That was my impression too, then I went back a couple prior messages, and looked at the earlier announcement. Wihle Netgate looks to have overreacted (at least from the info we have), I can understand why they would be upset. This was in the original announcement:

The first step was assessing the current state of the code the previous developer had dumped into the tree. It was not pretty. I imagined strange Internet voices jeering, “this is what gives C a bad name!” There were random sleeps added to “fix” race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows, and the whole litany of awful things that go wrong when people aren’t careful when they write C. Or, more simply, it seems typical of what happens when code ships that wasn’t meant to. It was essentially an incomplete half-baked implementation – nothing close to something anybody would want on a production machine. Matt had to talk me out of just insisting they pull the code entirely, and rework it more slowly and carefully for the next release cycle.

I can understand being upset if that's how you're portrayed publicly.

[tw04]

Keep in mind, back in February of 2020 when Kip Macy first announced that Netgate had hired him to port Wireguard, Jason offered to help. First Kip declines the offer, then seems to warm slightly to it, but ultimately appears to have not actually engaged Jason.

If I'm Jason and I offer my help (for free), they don't take me up on my offer, then try to release code that would make my baby look quite ugly, I would probably also have a pretty severe reaction.

Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him, I'm sure he was frustrated that he offered to help multiple times and they both didn't take him up on the offer, and tried to release a hatchet job with his name (indirectly) attached to it.

[kbenson]

> Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him

Oh, I don't entirely blame him. I just partially blame him for not seeing the obvious way this could devolve into a problem, even if it would (justifiably) seem unlikely to go to this level so fast. That is, he shouldn't be surprised there was a problem with what he said, although the scope of the problem is a bit more than I think most would expect.

Professionalism isn't just about making others feel good, it's about optimizing for useful outcomes, which includes covering yourself. Not taking care with your words is just like not taking care with your code. Sometimes there's a weird interaction and things go boom.

[Reventlov]

"Kip Macy" don't you mean Matt Macy ?

[tw04]

There's not a good way for me to respond to that without going off-topic. The following is assuming that wasn't a rhetorical question, if it was rhetorical I guess we may just agree to disagree:

Until he issues a public apology for his actions, I'll refer to him as Kip. Changing your name to run from the google searches is completely understandable, and I support second chances, but you need to show a bit of remorse IMO.

https://abcnews.go.com/US/exclusive-landlord-hell-defends-te...

[bscphil]

Wow, I was thinking the headline was underselling the story before I got to the part about him fleeing the country after his parents bailed him out of jail to the tune of half a million.

[Cyph0n]

Hot damn, that’s one crazy story!

[jimbob45]

Damn that link was an adventure from start to end.

[Reventlov]

I wasn't aware this was the same person. Thanks.

[generalizations]

I don't really think that the 'online mob' has the right to hold someone's past actions over their head, and expect some public appeasement before it relents.

[nullnix]

The actions of... pouring ammonia in his tenants' beds, throwing their stuff onto the street in trash bags, cutting holes in their apartments' floors while they were inside, cutting through floor joists under their apartments and physically attacking the building supervisor when he complained, fleeing the country to avoid arrest and sticking his own mother with a half-million-dollar bail forfeit as a result...

... no, no, there's no reason to hold actions like that over someone's head. It's entirely praiseworthy and I'm sure it's really easy to cooperate with such an upstanding character.

[nindalf]

His interactions with James have done nothing to convince me that he’s become a decent human being.

[Firehawke]

What you think has no bearing on reality. They will because they can.

[CameronNemo]

Sounds like Jason should trademark Wireguard (the name). Or build an alternative brand. That way Netgate's actions, or the actions of other wireguard implementations, will not reflect on the reputation of his project/product/technology.

[tw04]

He did trademark the name. I don't think Jason is going to tell the FreeBSD project that they can't use the name "wireguard" for their implementation of "wireguard" just because Netgate put out shoddy code. It's not the FreeBSD project's fault.

https://www.wireguard.com/trademark-policy/

[CameronNemo]

I think he should exactly tell the FreeBSD project to not use the mark if they cannot meet the quality requirements (especially if the quality issues were known prior to shipping). That is assuming they actually shipped the known-problematic code, though. Which they did not do.

[kaliszad]

I don't think this is an accurate description of the responsibility hierarchy. If in fact the code by Netgate or associates/ contractors/ employees of Netgate is not of professional quality, it has no place in the FreeBSD codebase ready for a stable release. Ultimately, the FreeBSD core community/ developers are together responsible for the codebase even if you cannot hold them legally accountable because of the license. They together hand out and take back (deny) the commit rights (bit, whatever). If a highly sought after component _in the kernel_ is (at least to Jason's account) not even up to the lowest security and code quality standard it has no place in the code base in preparation for a stable or probably even a beta release. Other developers (and that's where Jason is completely in the right if his account is correct) should protest the inclusion of such possibly very bad code into the codebase more or less in late preparations for a release as I understand it.

So it is first and foremost on Netgate, if Jason is right but right after that it is on the other responsible FreeBSD co-developers. I mean having so obviously bad code in any kernel of a modern operating system release would be really, really bad. There are many people and companies dependent on it that cannot really influence anything but have to endure the consequences either way. I mean, if you buy a storage appliance, a router or a firewall you trust the quality of the product to a degree and cannot really audit much even if you had the skill. You have to take the word for it and make some reasonable accomodations. No insurance is going the replace the full damage due to lost data to an attacker or a bug. Peoples lives sometimes indirectly depend on the full chain of competence and no insurance can resurrect the dead or right the good name of anybody. Remember, most of the time when you have to update anything for security reasons, somebody didn't understand the system fully or just plain messed up. The only exception is when the problem or times / requirements have changed (e.g. the computers got so fast, we have to transition to longer keys/ passwords whatever).

So yeah, if Jason's account is accurate it is bad the code landed in the codebase at all and raises questions about the quality and security of FreeBSD. I mean, it is code directly meant for a secure-as-possible VPN and something that often directly interacts with the open internet. Surely such code should experience extra scrutiny.

From the short personal interaction with Jason he came across as quite thoughtful and knowledgeable. Over the years, he and his supporters were able to convince many not so easy to convince people about the quality of Wireguard and some of its implementations. He and the supporters have shown a long term commitment and I am for these reasons inclined to trust Jason's judgement as well.

[jamal-kumar]

I think this is my favorite comment in the whole thread. The reasons you outline are exactly how I feel when it comes to priorities here, and how I feel his conduct was -- Despite everything maintaining friendliness while being attacked for making technical criticisms was incredibly commendable.

[tomxor]

I duno, if true about the code I find it very difficult to empathize with Netgate

From what has been said it's not like they found and fixed a subtle and cryptic vulnerability in an otherwise reasonable implementation and then failed to disclose it properly. It's more like they turned over a rock and found a murder victim. The guy from Netgate is also coming across as very inward looking and seems to assume everyone else's motivations are also purely selfish (referring to his comment implying a "shower of contracts" they might receive for the publicity). His focus should be on how to prevent this mistake from happening in future.

[Jonnax]

Well if it's true, then they were trying to put flawed code into freebsd which they would then ship to customers in their security product.

They're not some random person but are representing their company with their code.

If there was a security exploit with their Wireguard implementation, would Netgate get blamed or Wireguard?

[1vuio0pswjnm7]

Reminded me of the type of statements he made last year on another set of mailing lists:

https://news.ycombinator.com/item?id=24430424

https://mail-index.netbsd.org/tech-net/2020/08/22/msg007842....

https://mail-index.NetBSD.org/current-users/2020/08/22/msg03...

https://mail-index.NetBSD.org/tech-kern/2020/08/23/msg026693...

[secondcoming]

I don't think he intends to be patronising, but he certainly comes off that way.

[kaliszad]

I mean, Wireguard in these cases is a piece of code _in the kernel_, is meant to be directly in the path of a packet coming from an untrusted network and potentially does cryptography/ security critical stuff. Besides maybe memory management and such things, with what other pieces of code do you care more about quality and security than basically a VPN? People and companies that don't even know what Wireguard or a VPN for that matter is directly or indirectly depend on this and similar code being nearly perfect sometimes with their reputations or even lives. I mean, it is not a game that is taking too long to load or something, which is quite upsetting but I cannot imagine how that would be critical. The security of a kernel or a VPN is a very different story and we should all stop pretending this isn't the case. I hope (and am quite convinced) Jason and others want only the best for the respective kernels and the Wireguard implementation from the standpoint of quality and security.

[axaxs]

Yeah, same.

Even if all of the above is true, it reads like an elaborate insult. And that's fine if that what the author set out to do for some reason. Pretending it wasn't after the fact isn't being honest, in my opinion.

A more professional and neutral announcement could just talk about code that needs to be refactored due to some incompleteness and vulnerabilities.

[tptacek]

It's not an elaborate insult.

To a much greater extent than in other security protocols, implementation security is a goal of WireGuard. The protocol itself was designed to support secure kernel implementations; for instance, it's designed in such a way as to not require on-demand dynamic memory allocation.

It's part of the premise of the security model of WireGuard that it has secure kernel implementations. If you're building a kernel WireGuard implementation for a major open source OS without taking advantage of the WireGuard implementation design concepts, you're not really building WireGuard; you're building a compatible fork and calling it "WireGuard".

The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.

Instead, the WireGuard project seems to have been put into a position where they had to scramble to fix up an implementation that was being pushed into FreeBSD, as WireGuard qua WireGuard. I can imagine that being a frustrating experience. It certainly didn't generate the most political response ever, but I think you'd be reaching to call it a deliberate insult.

[kbenson]

> It's not an elaborate insult.

My read on it wasn't that it was an elaborate insult, but more that it was far more denigrating than it needed to be, if he was trying to be professional. That doesn't mean it was purposeful, sometimes people just don't really associate the statements they make with how it may be perceived.

I think it could have been communicated clearly and succinctly with something along the lines of: "The first step was assessing the current state of the code the previous developer had dumped into the tree. We noticed some quality problems, some unimplemented protocol sections and more concerning, security issues with the code. Given these issues, we considered asking they remove the code, but instead Matt convinced me that we should rework it slowly and carefully for the next release cycle."

Notably, I think omission of the following inflammatory statements would have prevented a lot of problems:

- "It was not pretty."

- "I imagined strange Internet voices jeering, “this is what gives C a bad name!”"

- "the most spectacular buffer overflows"

- "the whole litany of awful things that go wrong when people aren’t careful when they write C."

Whether those entirely subjective statements are accurate, they are not the things you say about someone else's work output when you expect a useful dialogue with them, which is exactly why they are considered unprofessional.

I'm not defending Netgate's code here, or even the vehemence of their reaction and how they went about it, but merely noting that not only can I see how it devolved into this, I would go so far as to say it's obvious that this is why that type of language is avoided by most people trying to work professionally. Jason wrote some very unkind things, and Netgate blew up about it. There's enough blame here that they can both share some.

> The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.

Err, wasn't that actually not the ask, because he thought they wouldn't do so, so instead they worked it over in a short time-frame, only for it then to be removed when this argument broke out and it came to light?

[zx2c4]

I get your point about perceptions, but there's also another aspect of why I found it important and necessary to describe just how poor the code was:

When you're talking about replacing and rewriting the implementation on the eve of release, you better have a good reason for doing so. Stuffing a rewrite of security critical code into the kernel at the last minute is a big red flag. The main question that immediately comes up in that context is, "how is it possible that having a last minute rewrite would be better than the code that was there before? You've only looked at this for a week." And that's a really good and important question.

That much code churn is not something I wanted when I set out to get started with this, but it's ultimately where things wound up. Why? For exactly the reasons I described in my email. The idea wasn't to be _insulting_, but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite. I see how perceptions could view that instead as denigrating, but that wasn't really the motivation. And it's not as though anybody really is rushing to defend that code either; it doesn't take a lot to look at that and make up your mind that it was probably unfinished stuff, not coded with much love, that was committed prematurely.

It also had the, I think, positive effect of leading to more scrutiny of the review process. A few people have piped up and mentioned to me that their concerns during that review weren't addressed. And as a consequence of everything, all of the code, including the rewrite, is being removed from FreeBSD until it can be carefully examined and completed, which is really the best of conclusions.

[jamal-kumar]

You did good, Jason. Honestly after this streissand effect from them taking technical criticism personally and threatening you, I'm probably just going to avoid anything using code they might have written... that's on them. Responding to a perceived non-professionalism by talking like that to you -- from their COMPANY EMAIL at that? If I were their boss I'd definitely start making some considerations regarding the irony of this.

[donmcronald]

I can see how someone could be insulted by that one paragraph on a personal level, especially if it had been the person that wrote the code. I can't think of a nice way to say it though.

However, on a professional level, when a core maintainer of a protocol tells you your code is bad, it's time to sit down, eat some humble pie, and start taking notes. I think that's especially true if you're offering help or are willing to address online communities with positive messaging to help them save face.

I think you handled it really well. I almost feel sorry for Netgate that they lashed out at someone so good at managing the technical (code), social (community), and political (bullshit) sides of a software project. Almost.

[kaliszad]

May I ask, how is it possible the previous code was in the code base at all when it was of so poor quality?

Btw. thank you (and all the people that probably help you here and there) for Wireguard. I put a lot of trust into You/ Wireguard and I am hardly the only one. So I am all for quality and security even if it takes as long as it takes. It is really a huge progress we have a useable and _simple_ VPN protocol and widespread implementations of it.

[kbenson]

> The idea wasn't to be _insulting_

Sure, I didn't really interpret it as you attempting to be insulting, more that you were accidentally insulting through your explanation of what you found.

> but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite

Sure, but is any of that really needed beyond "there were numerous security problems we had to address"? When talking about shipping crypto, I think most involved would agree not shipping it is better than shipping something possibly exploitable.

I think the core of what I was trying to express is that words should be crafted with care when expected to be read in a public forum like this, just like any code expected to be used by many should be crafted with care. For the same reason it's useful to remove quadratic algorithms from places where the input is somewhat not entirely vetted, it's useful to take care with words to reduce the chance of misinterpretation.

That doesn't mean scour your statements for the smallest possible misinterpretation, but there's a lot of room to improve things like "I imagined strange Internet voices jeering, “this is what gives C a bad name!”" while still expressing your point constructively. The low hanging fruit is easy to pick, so you might as well pick it.

To be clear, I feel for you with regards to this situation. Nobody really expects weird accusations like you got from simple emails, and that's on Netgate, but a less extreme response that also publicly notes the soured relationship would also be a negative outcome from this in my opinion, if one of lower magnitude.

[tptacek]

Sure, but it's easy to clinically examine any communication and refine it with the benefit of both hindsight and low cortisol levels. My read of this situation is that everyone involved was stuck in a shitty situation; it got very briefly heated, and ended up where it should have: with another dev cycle to iterate on FreeBSD WireGuard.

[kbenson]

I agree on both counts, but I think (constructive) criticism is warranted in a mistake. To absolve Jason of all responsibility would be to possibly not provide that useful feedback of why not to do this the same way next time.

Hopefully I accurately expressed that as what I was trying to convey. I don't think Jason is close to even half the problem in this case, just the small spark that allowed it to continue and explode (continue because is started with a substandard implementation to begin with). At the same time, he's also the one easier to critique constructively because the other party is hard to relate to (I'm not one to jump to conspiracy theories about implicit efforts to defame).

[justaj]

If the criticism is about the code instead of the person, it should never be interpreted as an insult. Proper developers have learned to split their ideas from their ego, and as such are able to receive harsh yet justified criticism concerning their code without being offended.

[ksec]

Similar reaction here. My first impression was Netgate being an arse. But then when you read the announcement I kind of understand why Scott is angry. Because while the post may have been in "good faith" in an Open Development and Open Source world, it surely isn't in a professional and business world especially when the work is sponsored ( being paid ).

Jason should have informed Netgate the quality of the code is shit in private and FreeBSD dev should have told Netgate will not be shipping any of it in Rel 13.

It is then up to Netgate to decide What to do with their Rel 2.5

[tw04]

> it surely isn't in a professional and business world especially when the work is sponsored ( being paid ).

To play devil's advocate: Netgate isn't paying Jason, and they're taking his open source code to create a proprietary commercial project. I'd say Jason owes them exactly nothing in the way of courtesy or consideration. Could he have been more polite for the sake of being polite and community goodwill? Probably.

[ksec]

>and they're taking his open source code to create a proprietary commercial project.

I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.

That is judging from the two pieces of information here. Jason doesn't need to be of consideration for Netgate. There could be other communication we dont know about. I can certainly understand why Scott is frustrated.

[tw04]

>I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.

Their sponsored work was based off of the Linux and OpenBSD code that Jason and others wrote. And even if it didn't utilize that code, you literally can't write a wireguard client without building on Jason's work.

[tptacek]

WireGuard is an open-source project, and an important one. It seems to me that if you want to push to create the authoritative WireGuard implementation for a major open source OS, the commercial norms need to take a back seat.

[geofft]

It is unprofessional and bad business to deliberately sell your customers insecure code, and you should not expect anyone to support you in doing so. Jason would have veen within his rights to warn Netgate's customers about security holes in pfSense - but he didn't even do that, he made a comment on his own project's mailing list, intentionally not naming the company, about what he was doing and why, which is entirely within the realm of "professional."

(And there is, of course, the question of whether Netgate is in any way "professional" by building on top of an open project and not following its norms.)

[stonogo]

NetGate spends a lot on FreeBSD development, which is great, but they also spend a lot of time running smear campaigns against people who offend them, which is ridiculous. They even started /r/opnsense on Reddit just to post shit-talking memes, and camp on the namespace to this day.