[Animats]

The paper is "3D-based video recognition acceleration by leveraging temporal locality". Haven't found a copy online. I'd expect a paper like that to come with examples and a link to Github. Why fake something like that? Either the paper is ignored, or someone implements it and finds it doesn't work.

[izacus]

I spent a lot of time in my career implementing and comparing different algorithms based on published papers. This might surprise you but:

- Most of them don't publish any kind of code or only snippets.

- Most of the algorithms are incomplete (think "we add constant M here into equation tuned by an expert"). The chosen constants aren't documented and they're tweaked before publication to maximize results for the given dataset.

- Most results are only good for the chosen dataset and based on very fine tuned constants (which is the reason they're not published). As soon as you apply them to a slightly different dataset they fall apart.

- Even if you get code for a given paper, it's usually a disatrous mess of quality and runs only on the given researchers computer with given version of Windows and a weird patched version of Python they found somewhere in the internet.

There ARE better written papers out there. But most of them are just made to publish "something" and get the publishing metrics up. That means minimum effort for actual research and maximum effort to tweak and tune results to make them look good.

[TheOtherHobbes]

It's tragic that academics have no metric for quality in published work.

The only metrics are journal status and citation count, which is partly why we're in this mess.

There should be some form of public post-review attached to all public work, with explicit requirements for rigour and reproducibility.

Some domains - like math - do at least attempt this, more or less. But CS doesn't seem to.

Aside from motivations - now skewed towards business outcomes - the underlying problem is that paper publishing as the gold standard for disseminating new research is a 17th century process and badly needs an update.

[black_puppydog]

> I'd expect a paper like that to come with examples and a link to Github

You'd expect that, but you'd probably be surprised in a lot of cases. One of the strong reasons why good conferences mandate code nowadays. The quality of the code, and whether it actually reproduces what you claim, isn't checked systematically though. That's down to the reviewers' enthusiasm, and whether they actually have the technical means to reproduce. That's not a given for more niche robotics or high-profile AI applications where hardware can be an issue.

[raverbashing]

> Why fake something like that? Either the paper is ignored, or someone implements it and finds it doesn't work.

The "formal" academic world does not care one bit about stuff working or not. It's all about self-promotion, papers and meaningless academic rituals and procedures.

(And it's even worse outside of the US let me put it that way)

[kvathupo]

As someone with a little experience in ML research, I feel like outside of the top 5-10 conferences, there rarely is a code repo [1]. I think an even bigger issue is that even when a repo is provided, it's often poorly documented, contains dead links, or _hard to reproduce on other machines_ without code-wrangling.

Tangentially related, even popular repos can have implementations of models that differ from the paper cited. So, who knows how far errors can propagate ¯\_(ツ)_/¯

[1] - https://www.guide2research.com/topconf/

[buildbot]

Somethings that’s personally bothered me for a long time is that as far as I know, there’s now current way to check the test accuracy on imagenet, as the test server has been gone for awhile. Yet somehow people still report a test accuracy…

[rosstex]

https://dl.acm.org/doi/10.1145/3307650.3322260

[alfu]

Here: https://sci-hub.se/10.1145/3307650.3322260