[krisoft]

I have only access to the pre-paywall excerpt, but what I can read does not endear me to pay for more.

> This understanding, of “object permanence”, is a normal developmental milestone, as well as a basic tenet of reality. It is also something that self-driving cars do not have.

That is just simply false. Nearly every team we heard technical details from has a “tracking subsystem” which integrates observations across time and sensor modalities. You cannot do that without object permanence.

How good is their object permanence? That is up for debate. Maybe there are situations particular versions from particular companies fail at. But then you should talk about these observed failures.

After all just because a healthy adult flunks a shell game we won’t conclude that they must lack object permanence.

Also how arogant it is from the writer to assume that out of the thousands and thousands of self-driving car engineers across many companies none of them thought that object permanance could be a trick worth implementing? What kind of ego one needs to write down a sweeping statement like that?

[Closi]

> Nearly every team we heard technical details from has a “tracking subsystem” which integrates observations across time and sensor modalities.

That’s not object permanence, that’s tracking the same object across multiple frames and tracking it as a single object. The below is part of the abstract for a paper “Learning to Track with Object Permanence” released this year that describes the difference between current tracking and the concept of object permenance:

> Tracking by detection, the dominant approach for on- line multi-object tracking, alternates between localization and re-identification steps. As a result, it strongly depends on the quality of instantaneous observations, often failing when objects are not fully visible. In contrast, tracking in humans is underlined by the notion of object permanence: once an object is recognized, we are aware of its physical existence and can approximately localize it even under full occlusions.

Not sure if Tesla has it or not, but there is a difference between object permenance and tracking objects across frames.

[em500]

I don't know how the various car teams implement them currently, but I find it hard to believe that this is some novel or SOTA approach for them. Kalman filtering and state space models are explicit object permanence models for tracking objects (relying on the physical location/velocity model to track object locations when observations are missing), used since the 1960s, and that pretty standard material in engineering courses on navigation.

[krisoft]

I hear that definition, but in terms of computer science this is not a categorical difference but one on a scale.

Clearly every tracking has to be able to re-localise among frames, otherwise it is not tracking. If you want to make a robust tracking you aim that it won't loose track even if the object is lost or obscured for a few second. These are all tuning parameters and questions of scenario. If you have a strong track of a vehicle with lots of evidence and you have strong priors about the road layout then the vehicle can disappear behind a bus for hours and you will still maintain the information that it is there. If you have a fleeting noisy observation about one pedestrian, and you don't really have a strong motion model about them (Because oh horror, pedestrians sometimes enter unmapped buildings and don't follow strict lanes!) then you might delete their track within seconds after they disappear.

So tracking creates information about objects, and how permanent they are is a tuning parameter. Some companies under some circumstances can choose to make the objects very permanent, some different companies or the same company under different circumstances can have very fleeting objects.

Given this, how much information would you need about the state of every single self driving system to write down a sentence like this confidently: "For a self-driving car, a bicycle that is momentarily hidden by a passing van is a bicycle that has ceased to exist."

I would be cautious writing such a sweeping generalisation even about bakeries and bread making, and that's a technology which has been practiced for thousands of years.

Here is what the author of the article could have done: Pick a specific failure of a specific self driving project and say "sometimes self driving cars struggle with object permanence. " It's not like you have to go far for an example. One of the root causes of the Elaine Herzberg accident was the car's inability to match her track among observations.

[NicoJuicy]

It was even deeply discussed on Tesla's AI day 3 weeks ago.