[boldslogan]

anyone care to talk about this part "This feature is currently operating in “shadow mode” in the fleet, which compares our software algorithm to real-world driver behavior across tens of millions of instances around the world."

that seems very juicy?

edit: it seems people put a disclaimer on their ownership of tesla stock on hn about tesla threads. i dont own any.

[dahdum]

> We are making progress towards stopping at stop signs and traffic lights. This feature is currently operating in “shadow mode” in the fleet, which compares our software algorithm to real-world driver behavior across tens of millions of instances around the world.

It's cool that they can collect that data and no doubt will help refine the feature, but otherwise marketing speak. Says nothing about how accurate their algorithm is or how long it may take. There's nothing stopping them from rolling out an algo for anything and running it in "shadow mode".

[m463]

They run autopilot in all the cars whether it is turned on or not.

If it is not turned on, they still send the inputs to autopilot, without controlling the car. I think they can also compare what the driver does against what the autopilot would have done.

They can also ask their "fleet" to match specific conditions, and collect data if they are met.

The autonomy day video showed some interesting images, for example "animals" the fleet had seen.

I think this is the video https://youtu.be/tbgtGQIygZQ

[leesec]

This is a known feature unveiled, I believe, on Tesla's autonomy day. It just means they're able to test Autopilot on cars even if they aren't running autopilot.

[11thEarlOfMar]

Or, it means that the feature is running and the neural nets exercising, but not outputting actual control. The results are captured, but not acted on.

[ceejayoz]

That's how I took it. They can then compare to what the human driver actually did, and find false positives (driver continued when the AI would've stopped) and false negatives (driver stopped when the AI would've continued).

[abledon]

ugh! the amount of supervised training data that would generate is staggering.

[kolbe]

Hence why many investors don't mind that Tesla isn't generating wonderful cash flow. If there's a reasonable expectation that this data collection will lead to full autonomy, then Tesla should price their cars at whatever level is necessary to move their maximum production into consumers' hands, even if it were a loss (which it's not).

And even if they don't use the data to train, they can use it as an argument for how much better they are than humans. That's one hurdle that Waymo could have trouble with, whereas Tesla could point to tens of millions of miles where their sensors would have avoided crashes that humans failed to avoid.

[abledon]

Do you think they keep all that data on premise or use AWS/Azure/GCP?

edit: probably AWS [1]

[1] https://www.theinquirer.net/inquirer/news/3027077/hackers-hi...

[Angostura]

Competitors buying hundreds of cars and continuously driving over mannequins.

[boldslogan]

neat. I would say im optomistic on this actual self driving stuff. But coming from almost no machine learning knowledge, doesn't this make tesla way ahead of google/waymo and others? Because they can simply check when they failed and tweak.

I'm just curious if it does make them way ahead, and if so how much more ahead? Is this driving a car vs horse wagon levels?

[Klathmon]

I think the real answer is nobody knows.

It's obviously got some benefits, and I think most people in ML would agree that real world data from the "average person" (as in "not a Dev or paid to do this") is worlds better than simulations.

Still, "ahead" and "behind" isn't really a thing before you reach the finish line here. Many think Tesla will never reach full self driving or won't reach it without additional sensors, others think it's possible but decades or more away, still others think it's right around the corner and can be done with what is on a Tesla right now.

Until someone reaches that finish line, we won't have anything to compare it to, so we don't have any idea how close or far anyone really is.

[jameslevy]

Is there any public research or data regarding the value or predicted delta value of long tail "average person" driving data of this kind versus the kind of data captured by driving around the same small portions of SF or MV? I've looked around but haven't found anything. Granted, Tesla and its competitors don't have an incentive to publish this kind of data...but are they even in a position to be able to calculate it currently?

[Animats]

Probably not. Ordinary staying-on-road is pretty much solved. There's years of dashcam imagery available.

The last minute of video before each crash would probably be more useful. Training on dashcam data of people doing stupid stuff from Youtube might be useful. You don't need steering, braking, and accel data; you can run a 3D SLAM algorithm to extract the path and get that.

[xbmcuser]

My 2 problems with Musk is that he has dismissed lidar in favor of just cameras because lidar is expensive. So far most of the Tesla auto pilot accidents would have been avoided with lidar. Even 2 very basic lidar sensors on the front and back could make a huge difference. And he calls a driver assistance system autopilot when it hasn't reached the level of autonomy which the word autopilot implies.

[repolfx]

Tesla's approach means there isn't a finishing line. Just a long stream of over-the-air updates that incrementally enable more and more complex scenarios. They could be at it years. One day people will cross a psychological threshold where manual intervention is needed rarely enough that we talk about it as fully self-driving rather than limited driving assistance.

[leesec]

Which company is in the lead is a very contentious topic it seems. This one specific feature doesn't seem to push it one way or the other. But IMO Tesla has the best approach and has a huge lead on the data collection pipeline. No other self driving company has even close to as many cars to test on / gather data from.

[mrep]

> But IMO Tesla has the best approach and has a huge lead on the data collection pipeline.

Not only that, but by using cheap hardware that consumers can reasonably afford, they really only need to get to level 3 (eyes off so you can work/entertain but driver still in the seat) for it to be massively beneficial for everyone who commutes to work.

Waymo pretty much requires level 4 as it's too expensive to be sold as a consumer car and nobody will be in a taxi when it is driving between customers.

[bllguo]

contentious? Genuinely curious for a source here. everything ive ever seen says Waymo is far in the lead, though I dont work in this space.

[leesec]

The contentious part is that Waymo doesn't have a product yet. It's easy to imagine something is better when you can't actually use it.

In truth there are only 3 self driving products that I know of available today. Tesla's Autopilot, GM's Supercruise (only on CT6), and commma.ai's openpilot.

Of those Autopilot is easily the best.

I never know why a source would matter in these cases as the writers of these sources usually have no idea and the answer isn't clear anyways. But if you want you can use me as a source because I'm an engineer who worked in automotive and has driven all 3 of aforementiond products. I also am not sure Waymo will ever release a feasible product other than their LIDAR.

[icelancer]

>> It's easy to imagine something is better when you can't actually use it.

Yup. Carmack illustrated this very well in his interviews when he would not talk about unreleased games vs. Doom/Quake/whatever. He always said: "It's really easy to be the best graphics engine when it isn't available to the public."

[bllguo]

That is a great point. Would you agree that the _narrative_ is that Waymo has the superior tech right now? Or did I misjudge that?

I suppose I used the wrong word. Any expert or well-informed opinion is what I was after.

[dkonofalski]

Probably not way ahead but definitely ahead. Waymo has hundreds, if not thousands of fleet vehicles driving all day, every day around various parts of the country too and they can also check when people took over or when the brakes were hit. I think Tesla's approach is a little neater and nicer but that's just an opinion.

[leesec]

Tesla has hundreds of thousands driving and growing rapidly?

[dkonofalski]

There are a lot of Teslas that do not have the hardware necessary to do the training required. Additionally, Waymo cars are driving all day long while most Teslas are still commuter vehicles.

[MegaButts]

Disclaimer: I own no financial stake in Tesla, and while I love the goal of converting the world towards electric vehicles as whole I am not a fan of Tesla or Elon Musk.

There are people much more informed than myself that believe Shadow Mode isn't real, and that Tesla is outright lying about its existence.

https://twitter.com/greentheonly/status/1096322810694287361

[ajross]

That twitter thread (which is great, btw, everyone interested in this stuff should be following @greentheonly) doesn't remotely substantiate "Shadow Mode isn't real, and that Tesla is outright lying".

I don't know the extent of public statements about it, but comparing collected data like the stuff detailed there to driver behavior is certainly feasible and almost certainly being done. It's also likely true that the marketing department had their way with the announcement and made it sound more like an on-vehicle entity, but.. meh.

I just don't see how you got from that (again, really great) analysis to such an outraged conclusion.

[MegaButts]

I was stating his opinion, but I'll just quote him directly:

> We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose

I really did not mean for my comment to be inflammatory (hence the disclaimer and including a source), but considering how quickly it was downvoted I guess people do not approve.

[ricardobeat]

That quote has no substance still. In the thread he proves that he can inspect some data logging routines in his Tesla, how does that lead to “shadow mode does not exist” is a mystery.

It is an obvious thing to do and the ‘shadowing’ aspect would not be particularly challenging tech, why the disbelief?

[jklinger410]

The thread then goes on to not support this statement very well. Including a Tesla employee disagreeing with him directly.

[MegaButts]

If we're starting with the assumption that Tesla is lying about shadow mode, which is the entire premise of the claim, it would not be surprising for a Tesla employee to disagree. Obviously this is one of those you-can't-win scenarios, so I fully understand the Tesla employee could be acting in good faith, but that alone is no more evidence than the original hacker's claim that it doesn't exist considering the context.

I consider it a very interesting thread where someone went looking for proof of shadow mode and couldn't find it. Maybe his conclusion is a little extreme, and I guess people thought I was trying to derail a Tesla thread with FUD when really I was trying to share an interesting piece and someone else's better-informed opinion (which is why I worded it the way I did). I sort of consider it similar to saying "expert A believes that..." but I guess I communicated that poorly.

For what it's worth, I'm glad people are actually looking at the source and critiquing it.

[RandallBrown]

How does that person know they are running the latest software with every feature turned on? Just because their car doesn't run in shadow mode doesn't seem to mean that no cars are.

[rndmize]

Sounds like they're running their autopilot software on some (all?) of the cars in the wild, where they can compare what the autopilot would do with the actions taken by the human driving the car, and use this to analyze/refine their model/system.

In casual talks with friends, this is something I've expected would be the competitive edge Tesla has over Waymo and the others; driving millions of "virtual miles" strikes me as far less useful than having a human baseline to compare to along with data capture from a fleet in all manner of real situations/locations.

[boldslogan]

Which makes me imagine a crazy marketing video where they demonstrate their system is driving the right path and the human crashes.

In huge black font, then they just show the statistics of how often that happens.

[OliverJones]

They do this a lot when they're trying out new automation. It's part of the payback for all the investment they've put into field updates and telemetry. I think this data-gathering is part of the reason they continue to absorb the costs of LTE communications.

It's an excellent way to prove (or disprove) the effectiveness of new safety-related features.

As for non-safety-related features, my 3-yr-old granddaughter calls my Model S the "fartmobile" because of a recent electronic whoopee-cushion easter egg.

[new_realist]

Doubt the model will coverage as the actions are from human drivers, not generated by the model.