[ansk]

You're clearly disillusioned with the general accessibility of ML research, but I don't think your cynicism is warranted here. Take a look at their prior works[1], and I think you'll agree they go above and beyond in making their work accessible and reproducible. There is no reason to doubt the open-source release of this work will be any different. As to why the release is delayed, I'd speculate it's because they put a significant additional amount of work into releases and because releasing code in a large corporation is a bureaucratic hassle.

[1] https://nvlabs.github.io/stylegan2/versions.html

[sillysaurusx]

There is no reason to doubt the open-source release of this work will be any different.

Then this is not a scientific contribution yet.

We must wait and see.

The most important tenet of science, is to doubt. I didn’t even read the name on the paper before I wrote my comment. Yes, I know this group. They’re why I got into ML, along with the group from OpenAI who published GPT-2. Because A+ science.

Their claims here are likely wrong unless and until proven otherwise. This isn’t a hardline position. It’s been my experience across many codebases, during my two years of trying to reproduce many ideas.

I agree that that is an example of A+ science. But why do you think they’re punishing this now, today? Either because conference deadline or because nVidia pressure. Neither of those are related to helping me achieve the scientific method: reproducing the idea in the paper, to verify their claims.

All I can do is kind of try to reverse engineer some vague claims in a pdf, without those things.

--

Let me tell you a little bit about my job, because my time with my job may soon come to an end. I think that might clear up some confusion.

My job, as an ML researcher, is to learn techniques that may or may not be true, combine them in novel ways, and present results to others.

Knowledge, Contribution, Presentation, in that order.

The first step is to obtain knowledge. Let's set aside the question of why, because why is a question for me personally, which is unrelated.

Scientific knowledge comes when Knowledge, Contribution, and Presentation are all achieved in a rigorous way. The rigor allows people like me to verify that I have knowledge. Without this, I have mistaken knowledge, which is worse than useless. It's an illusion – I'm fooling myself.

When I got into ML two years ago, I thought that knowledge would come from reading scientific papers. I was wrong.

Most papers, are wrong. That's been my experience for the past two years. My experience may be wrong. Maybe others obtain rigorous scientific knowledge through the paper alone.

But researchers happen to obtain a dangerous thing: prestige. Unfortunately, prestige doesn't come from helping others obtain knowledge. It comes from that last step -- presentation.

The presentation on this thread is excellent. It's another Karras release. I agree; there's no reason to doubt they'll be just as rigorous with this release as they are with stylegan2.

But knowledge doesn't come from presentation. Only prestige.

Prestige makes a lot of new researchers try very hard to obtain the wrong things.

If all of these were small concerns, or curious quirks, they'd be a footnote in my field guide. But I submit that these things are front and center to the current state of affairs in 2021. Every time a release like this happens, it generates a lot of fanfare and we come together in celebration because ML Is Happening, Yay!

And then I try to obtain the Knowledge in the fanfare, and discover that either it's absent or mistaken. Because there are no tools for me to verify their claims -- and when I do, I often see that they don't work!

That's right. I kept finding out that these things being claimed, just aren't true. No matter how enticing the claim is, or whether it sounds like "Foobars are Aligned in the Convolution Digit," the claim, from where I was sitting, seemed to be wrong. It contained mistaken knowledge -- worse than useless.

Unfortunately, two years with no salary takes a toll. I could spend another few years doing this if I wanted to. But I wound up so disgusted with discovering that we're all just chasing prestige, not knowledge, that I'd rather ship production-grade software for the world's most boring commercial work, as long as the work seems useful and the team seems interesting. Because at least I'd be doing something useful.

[phreeza]

Expecting fully executable code to accompany every publication is kind of unique to the modern ML research Scene. As someone from a very different computational research field, where zero code is the norm, not the exception, this reads as a somewhat entitled rant. Reimplementation of a paper is actually a test of the robustness of the results. If you download the code of a previous paper, there may be some assumptions hidden in the implementation that aren't obvious. So I would argue that simply downloading and re-executing the author's implementation does not constitute reproducible research. I know it is costly, but for actual reproduction, reimplementation is needed.

[sillysaurusx]

I wasn't sure whether to post my edit as a separate comment or not, but I significantly expanded my comment just now, that helps explain my position.

I'd be very interested in your thoughts on that position, because if it's mistaken, I shouldn't be saying it. It represents whatever small contribution I can make to fellow new ML researchers, which is roughly: "watch out."

In short, for two years, I kept trying to implement stated claims -- to reproduce them in exactly the way you say here -- and they simply didn't work as stated.

It might sound confusing that the claims were "simply wrong" or "didn't work." But every time I tried, achieving anything remotely close to "success" was the exception, not the norm.

And I don't think it was because I failed to implement what they were saying in the paper. I agree that that's the most likely thing. But I was careful. It's very easy to make mistakes, and I tried to make none, as both someone with over a decade of experience (https://shawnpresser.blogspot.com/) and someone who cares deeply about the things I'm talking about here.

It takes hard work to reproduce the technique the way you're saying. I put all my heart and soul into trying to. And I kept getting dismayed, because people kept trying to convince me of things that either I couldn't verify (because verification is extremely hard, as you well know) or were simply wrong.

So if I sound entitled, I agree. When I got into this job, as an ML researcher, I thought I was entitled to the scientific method. Or anything vaguely resembling "careful, distilled, correct knowledge that I can build on."

[msapaydin]

I think that not being able to reproduce the results claimed in a paper is not specific to ML research. While working as a post-doc at a top university research lab, i spent years trying to understand how it can be that some software that was supposed to corresponds to the well cited paper did not even come close to reproducing the results of the said paper, and that the primary author went on to become a prof at a top university in the US. In short, scientific fraud is also quite common, in most academic papers.

[sillysaurusx]

Thank you!!

i spent years trying to understand how it can be that some software that was supposed to corresponds to the well cited paper did not even come close to reproducing the results of the said paper,

This was my exact experience. I didn’t understand why I kept having it, and kept blaming myself for not being careful enough. My code must be wrong, or the data, or something.

Nah. It was the idea.

Kept feeling like a kick in the gut, until here we are today, when I’m warning everyone that Karras, of all people, might publish such a thing.

I really appreciate that you posted this, because I’m so happy I wasn’t alone in the feeling of “what’s going on, here…?”

[gonehome]

That seems like a worthwhile thing to publicize in and of itself?

The replication crisis in psychology threw out 50% or so of supposed scientific results.

If this (or just straight fraud) is common elsewhere, it seems like knowing about that would be a good thing for science.

[phreeza]

I get your frustrations with this state of affairs, but for the reasons I mentioned above, I don't think providing the model and code is a panacea here. Maybe the last few years have also set an unrealistic expectation for the pace of progress. In my (former) field of theoretical neuroscience, if a paper was not reproducible, this knowledge kind of slowly diffused through the community, mostly through informal conversations with people who tried to reproduce or extend a given approach. But this takes several years, not the kind of timescale that modern ML research operates on.

Fwiw I think actual knowledge is there in the ML literature, but it's not in these Benchmark-chasing highly tuned papers. It's more high level stuff, like basic architecture building blocks etc. GANs and Transformers for example. They undeniably work, and the knowledge needed to implement them can probably be conveyed in a few pages maximum. No need for an implementation to be provided by the author, really.

[skybrian]

I have no particular expertise here, but I wonder if you've learned to accept a mostly-broken process? We have the Internet, so why settle for slow diffusion over years instead of rapid communication?

Why should graduate students have to spend years trying to reproduce stuff that turns out to be no good? Nobody should have to put up with getting their time wasted like that.

[phreeza]

I think it is a social problem, not a technical one. A healthy research field should have some level of cooperation between participants. If you go ahead and publish a "this does not reproduce" paper, you can easily ruin someone's career, so in most cases you don't. I know this is not the platonic ideal of science, but it is the reality, especially of smaller research communities. I agree this is not ideal, but not sure if I would call the process broken though.

[godelski]

To be honest I don't think the StyleGAN papers are benchmark chasing. If you read StyleGAN[0], StyleGAN2[1], StyleGAN2-ADA[2], and this paper there is a clear story. They call out the mistakes in the previous papers and resolve them. The papers themselves even admit to where they fall short. But it is research. Problems don't get solved all at once. But if you pay attention to these 4 papers it is very clear Kerras has a very well defined research focus and direction. He's showing his progress over time, sharing it with the community, and learning from the community as well. This is how research should happen.

[0] https://arxiv.org/abs/1812.04948

[1] https://arxiv.org/abs/1912.04958

[2] https://arxiv.org/abs/2006.06676

[phreeza]

Good point, I wanted to make a more general point about the state of ML literature, I am not super familiar with this particular sequence of papers.

[xksteven]

Sometimes reimplantation is impossible without the code and the paper goes on to win awards because it's by a famous scientist. Then if the reimplantation doesn't work most of the time the graduate students are blamed instead of the original work.

There are always assumptions. At least with public code and models those assumptions are laid bare for all to see and potentially expose any bad assumptions.

[nerdponx]

It's not an entitled rant, other fields just have dismally low standards.

[phreeza]

I tried to give reasons why publishing code may actually not be universally desirable.

[areeh]

I think the discrepancy between what actually happens in the code and the main ideas in the paper is a great point, and it touches on the parent commenter's goal of attaining knowledge. For instance, even if the results are reproducible, maybe the proposed key idea in the paper is not the piece of code with the largest impact on performance. Though, with the code available you might be able to discover exactly what the discrepancy is if there is one and you might be stuck at "it doesn't reproduce" otherwise

[xwolfi]

Yeah but look at the first comment which said the opposite of what you say is normal. The first comment was praising the fact the code was published, the second that no it's not, followed by an autistic rant.

To his defense, the spirit of his rant was valid, the letter made it sound entitled.

[godelski]

> When I got into ML two years ago, I thought that knowledge would come from reading scientific papers. I was wrong.

I'm in the middle of a PhD and this is always an issue. It takes awhile to learn how to read papers and to gather enough background knowledge that you can read between the lines (publications are limited, you can't put everything in a paper. This is why having code is so great, it accelerates the process). You're two years into your journey, this is often when things _start_ turning the other direction. There's a reason PhDs take so long, and that's with experts (hopefully) helping you learn how to read papers, telling you which papers to read (which is a challenge in of itself), having the ability to spend full time on learning, and learning how to build background knowledge on a subject while learning the state of the art. There's a reason ML pays the big bucks. It takes a long time to learn/gather expertise, it is fucking difficult, and it has direct applications that can lead to useful products today (a big component of why you get paid big bucks). It is also easy to lose track of your progress. I remember the first research paper I read was complete gibberish to me. I'm 3 years into my PhD and now I can understand papers in my niche. But for a long time a lot of stuff didn't click. This is normal. It takes time to learn and 2 years isn't that much (especially when you have a full time job). Making contributions in your first year of a PhD is atypical, even in your second year. It only happens at top universities where people have a lot of help and resources.

Research it hard. It takes years to become an expert and learn how to read papers. Don't give up, but calm down and recognize that given more time things will make more sense.

[varispeed]

When I was writing a paper I had to include all source code in a state to be published, otherwise it wouldn't be accepted. I guess today the bar is much lower.

[godelski]

What was released was a pre-print, not a publication. Every top ML conference requires releasing of code, but this is not the norm for CS research nor for research in general.

The bar isn't low, this is just a pre-print.