This is very cool! My first thought is: can this be applied to converting raster graphics to vector graphics (eg PNG to SVG)? Seems like a very similar problem, though probably much more computationally expensive.
> We apply our approach to inverse graphics tasks, where our model learns to convert images into programs that produce those images.
I would argue that at least on a philosophical level, this is, definitionally, the process of converting raster graphics to vector graphics, as long as you by the premise that the difference between the two is simply that vector gfx is a programmatic/imperative representation of image generation, while raster is a data structure/declarative representation of images.
In other words, simply put, raster images are just arrays, vector images are sequences of instructions. Raster images are naturally much closer to the “raw” data needed by the output mechanism, while vector images require a much more complex interpreter.
Or, raster and vector images are philosophically the same thing. The only difference is that vector has more operations than raster. Raster just has "draw unit square at integer coordinates".
Though I agree with the point that paper makes (it makes a good case that the little square mental model of a pixel is mostly inappropriate) it does seem focused on imaging and does not mention places where the little square mental model is appropriate.
Pictures of cats, subpixel rendered text, or company logo SVGs as displayed on a web page are point samples and not little squares.
User interfaces are good examples of often being little squares. Calling HN's beige background a point sampled discrete representation of an underlying continuous function seems pretty tortured — to me it seems like a bunch of beige little squares.
> vector gfx is a programmatic/imperative representation of image generation, while raster is a data structure/declarative representation of images.
This seems a bit off to me.
Aside from oddities such as Postscript, most vector formats are canonical examples of declarative code. The distinction is more about what is being represented rather than how it is represented.
I would argue that at least on a philosophical level, this is, definitionally, the process of converting raster graphics to vector graphics, as long as you by the premise that the difference between the two is simply that vector gfx is a programmatic/imperative representation of image generation, while raster is a data structure/declarative representation of images.
In other words, simply put, raster images are just arrays, vector images are sequences of instructions. Raster images are naturally much closer to the “raw” data needed by the output mechanism, while vector images require a much more complex interpreter.