[singularity2001]

is there any progress on combining object detection with object segmentation? So instead of boundary boxes we get the true shape of objects? I know segmentation exists, just wondering about integration with Yolo or similar.

[Q6T46nT668w6i3m]

Yes, look into instance or panoptic segmentation. The most popular method is a region-based network that jointly regresses bounding box coordinates alongside an object mask and class label.

[singularity2001]

Thanks. The next step would be combining it with text-image foundation models such as clip https://github.com/openai/CLIP so that the model no longer depends on a limited set of predefined labels (coco…), right?

Also occlusion inference would be fantastic, so that we can select between the visible parts of the object and the whole shape (behind trees etc).

Exciting decade.

[yeldarb]

Yes, this is called "instance segementation". There's a YOLO-based instance segmentation model called YOLACT.

[singularity2001]

Wonderful, thanks.

It says backbone: Resnet101-FPN in https://github.com/dbolya/yolact ?

Anyone else looking for a pip installable solution: I found https://github.com/ayoolaolafenwa/PixelLib

And most current: https://github.com/yeliudev/catnet

[dimatura]

This task is called instance segmentation and is an active research topic. Mask-RCNN is relatively old these days, but still might be the most popular approach. There also happens to be a few approaches for the task taking inspiration in methodology from YOLO, e.g. YOLACT (which clearly also pays homage in name).

[genewitch]

Is this "rotoscoping"?

[klysm]

In real time?