This is a known feature unveiled, I believe, on Tesla's autonomy day. It just means they're able to test Autopilot on cars even if they aren't running autopilot.
Or, it means that the feature is running and the neural nets exercising, but not outputting actual control. The results are captured, but not acted on.
That's how I took it. They can then compare to what the human driver actually did, and find false positives (driver continued when the AI would've stopped) and false negatives (driver stopped when the AI would've continued).
Hence why many investors don't mind that Tesla isn't generating wonderful cash flow. If there's a reasonable expectation that this data collection will lead to full autonomy, then Tesla should price their cars at whatever level is necessary to move their maximum production into consumers' hands, even if it were a loss (which it's not).
And even if they don't use the data to train, they can use it as an argument for how much better they are than humans. That's one hurdle that Waymo could have trouble with, whereas Tesla could point to tens of millions of miles where their sensors would have avoided crashes that humans failed to avoid.
For compute, during the autonomy day video they mention they use their own clusters, because they've designed their own "full self driving computer" with a custom set of neural network accelerator ASICs. So to do test runs on their own driving computers, of course, they need to build their own machine clusters.
Custom built clusters can be much cheaper than the cloud anyway.
For storage, they say they actually pull data on demand from the fleet. They have NNs that can detect similar-looking stuff to input samples, so if they want more videos of construction sites they just ask the fleet to send them more videos of construction sites. Actual storage of all video isn't required except for their test suites.
neat. I would say im optomistic on this actual self driving stuff. But coming from almost no machine learning knowledge, doesn't this make tesla way ahead of google/waymo and others? Because they can simply check when they failed and tweak.
I'm just curious if it does make them way ahead, and if so how much more ahead? Is this driving a car vs horse wagon levels?
It's obviously got some benefits, and I think most people in ML would agree that real world data from the "average person" (as in "not a Dev or paid to do this") is worlds better than simulations.
Still, "ahead" and "behind" isn't really a thing before you reach the finish line here. Many think Tesla will never reach full self driving or won't reach it without additional sensors, others think it's possible but decades or more away, still others think it's right around the corner and can be done with what is on a Tesla right now.
Until someone reaches that finish line, we won't have anything to compare it to, so we don't have any idea how close or far anyone really is.
Is there any public research or data regarding the value or predicted delta value of long tail "average person" driving data of this kind versus the kind of data captured by driving around the same small portions of SF or MV? I've looked around but haven't found anything. Granted, Tesla and its competitors don't have an incentive to publish this kind of data...but are they even in a position to be able to calculate it currently?
Probably not. Ordinary staying-on-road is pretty much solved.
There's years of dashcam imagery available.
The last minute of video before each crash would probably be more useful. Training on dashcam data of people doing stupid stuff from Youtube might be useful. You don't need steering, braking, and accel data; you can run a 3D SLAM algorithm to extract the path and get that.
My 2 problems with Musk is that he has dismissed lidar in favor of just cameras because lidar is expensive. So far most of the Tesla auto pilot accidents would have been avoided with lidar. Even 2 very basic lidar sensors on the front and back could make a huge difference. And he calls a driver assistance system autopilot when it hasn't reached the level of autonomy which the word autopilot implies.
Tesla's approach means there isn't a finishing line. Just a long stream of over-the-air updates that incrementally enable more and more complex scenarios. They could be at it years. One day people will cross a psychological threshold where manual intervention is needed rarely enough that we talk about it as fully self-driving rather than limited driving assistance.
Which company is in the lead is a very contentious topic it seems. This one specific feature doesn't seem to push it one way or the other. But IMO Tesla has the best approach and has a huge lead on the data collection pipeline. No other self driving company has even close to as many cars to test on / gather data from.
> But IMO Tesla has the best approach and has a huge lead on the data collection pipeline.
Not only that, but by using cheap hardware that consumers can reasonably afford, they really only need to get to level 3 (eyes off so you can work/entertain but driver still in the seat) for it to be massively beneficial for everyone who commutes to work.
Waymo pretty much requires level 4 as it's too expensive to be sold as a consumer car and nobody will be in a taxi when it is driving between customers.
The contentious part is that Waymo doesn't have a product yet. It's easy to imagine something is better when you can't actually use it.
In truth there are only 3 self driving products that I know of available today. Tesla's Autopilot, GM's Supercruise (only on CT6), and commma.ai's openpilot.
Of those Autopilot is easily the best.
I never know why a source would matter in these cases as the writers of these sources usually have no idea and the answer isn't clear anyways. But if you want you can use me as a source because I'm an engineer who worked in automotive and has driven all 3 of aforementiond products. I also am not sure Waymo will ever release a feasible product other than their LIDAR.
>> It's easy to imagine something is better when you can't actually use it.
Yup. Carmack illustrated this very well in his interviews when he would not talk about unreleased games vs. Doom/Quake/whatever. He always said: "It's really easy to be the best graphics engine when it isn't available to the public."
Yes I would agree Waymo owns the narrative and a lot of it is based on their disengagement stats. I wouldn't say those tell the whole story.
They are also completely different approaches, i.e. Waymo using LIDAR and Tesla relying on cameras as the primary sensor.
But the major question is can LIDAR _ever_ be financially feasible, and if so can it become feasible faster than camera gets good enough at vision. Right now the sensors on the Waymo van probably cost more than the van itself.
That's either a very expensive purchase for the consumer or a very large amount of cap-ex Waymo would have to put up for a fleet of self-driving taxis.
And if it turns out cameras were the right choice, or visa versa, that means the opposite company would have spent a LOT of their R&D funding on the wrong tech, and would be way behind.
Probably not way ahead but definitely ahead. Waymo has hundreds, if not thousands of fleet vehicles driving all day, every day around various parts of the country too and they can also check when people took over or when the brakes were hit. I think Tesla's approach is a little neater and nicer but that's just an opinion.
There are a lot of Teslas that do not have the hardware necessary to do the training required. Additionally, Waymo cars are driving all day long while most Teslas are still commuter vehicles.
Disclaimer: I own no financial stake in Tesla, and while I love the goal of converting the world towards electric vehicles as whole I am not a fan of Tesla or Elon Musk.
There are people much more informed than myself that believe Shadow Mode isn't real, and that Tesla is outright lying about its existence.
That twitter thread (which is great, btw, everyone interested in this stuff should be following @greentheonly) doesn't remotely substantiate "Shadow Mode isn't real, and that Tesla is outright lying".
I don't know the extent of public statements about it, but comparing collected data like the stuff detailed there to driver behavior is certainly feasible and almost certainly being done. It's also likely true that the marketing department had their way with the announcement and made it sound more like an on-vehicle entity, but.. meh.
I just don't see how you got from that (again, really great) analysis to such an outraged conclusion.
I was stating his opinion, but I'll just quote him directly:
> We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose
I really did not mean for my comment to be inflammatory (hence the disclaimer and including a source), but considering how quickly it was downvoted I guess people do not approve.
That quote has no substance still. In the thread he proves that he can inspect some data logging routines in his Tesla, how does that lead to “shadow mode does not exist” is a mystery.
It is an obvious thing to do and the ‘shadowing’ aspect would not be particularly challenging tech, why the disbelief?
If we're starting with the assumption that Tesla is lying about shadow mode, which is the entire premise of the claim, it would not be surprising for a Tesla employee to disagree. Obviously this is one of those you-can't-win scenarios, so I fully understand the Tesla employee could be acting in good faith, but that alone is no more evidence than the original hacker's claim that it doesn't exist considering the context.
I consider it a very interesting thread where someone went looking for proof of shadow mode and couldn't find it. Maybe his conclusion is a little extreme, and I guess people thought I was trying to derail a Tesla thread with FUD when really I was trying to share an interesting piece and someone else's better-informed opinion (which is why I worded it the way I did). I sort of consider it similar to saying "expert A believes that..." but I guess I communicated that poorly.
For what it's worth, I'm glad people are actually looking at the source and critiquing it.
> I consider it a very interesting thread where someone went looking for proof of shadow mode and couldn't find it.
This seems to be assuming that it's something that exists entirely within the car, whereas the more sensible implementation is that the car uploads the sensor data and driver choices to a server in a datacenter somewhere and "shadow mode" happens there by running the autopilot code against the data.
That way you can continue to use the data even with newer or experimental versions of the autopilot code that didn't exist when some of the data was collected, compare different versions to see which ones do better and when, etc.
How does that person know they are running the latest software with every feature turned on? Just because their car doesn't run in shadow mode doesn't seem to mean that no cars are.