It's interesting that an academic conference now feels like a marketing op for industrial research labs more than anything. His claims about how accurate their vision system is and how it is exceeding other sensors is not verifiable in any way to the public. Given how well qualified he is I am sure he is not wrong! Andrej is brilliant. But this is an academic conference right? This isn't open science, it's a discussion about an engineered system. I'm afraid this is the future of ML research (which CV is so heavily dependent on now). Long gone are the days of reading a paper and understanding the approach. Now you need the data and model which may not even be computationally feasible without millions of dollars in hardware. This isn't Tesla's fault or anything, it just makes me sad.
He made a very good argument for vision-only, but it seems like training actually uses radar data to help calibrate vision measurements, so it seems to me there’s value in making some vehicles still contain radar (say, one out of 10) even if it’s not used for controlling the vehicle directly at drive time.<p>Also, the sensor resolution issue he mentioned could be addressed by using a higher resolution radar sensor.<p>I find the list of 221 triggers to be interesting. In principle, the NHTSA or NTSB could help contribute lists of triggers to companies to validate their training sets on.<p>Every time there is a fatal airliner accident, the NTSB does a safety investigation and airliners get a little bit safer each time. In the same way, each fatal accident in a vehicle with this kind of autonomy could end up being captured by these triggers, improving safety over time in a sort of mixture between expert human analysis and ML.<p>(Nobody does this for all regular car crashes because fatal car crashes happen every day! And you’re not going to retrain human drivers about some new edge case every day, although you can for vehicles like this.)
Tesla's decision not to use the LIDAR as a safety feature (i.e. having reliable high-resolution data about things the car can collide with) is so incredibly indefensible, since solving the last 1% of this using only vision likely requires a general artificial intelligence<p>Prediction: Tesla will be the last of all major auto manufacturers to get to L5 autonomy. Time interval between when Tesla L5 FSD is finally available and when humanity is destroyed by the general AI it runs on will be very awesome and also very short
I see a lot of people here are stuck on the perception side of things. There's a lot more to self driving than just the sensor suite and perception. There's a lot of work that needs to be done in the planning and controls department prior to the time we get full vehicle autonomy. Andrej's work is impressive, but I wish we'd see more research into the latter. Then again this is CVPR so...
I think the more interesting question is how much human context is necessary in decreasing accident rates. The signal question and tunnel answer hinted at that. Some context is very local and some context is general at the level of humans.<p>Examples: human eyes will have trouble adjusting to the sudden darkness of tunnels so some people will tend to brake suddenly; that person looks old and will probably have slower reaction times so watch out for the upcoming sharp turn; that person looks like they are on their phone and may cross the lane suddenly; watch out for this intersection because young humans cross it after school without looking so slow down below the speed limit.<p>This human understanding doesn’t seem to be directly represented by the system without explicit architecting on their part. A more general intelligence would begin to automatically learn these. A human intelligence would automatically model these or learn from experience or read about it.<p>As mentioned, the current system has some advantages over humans: more eyes, doesn’t get tired or distracted, faster reaction time. I guess we shall see when these advantages cover up the disadvantages.
tldr: Tesla uses vision alone, and has dropped radar and the other sensor. He makes a very decent argument why.<p>(Surprisingly, he basically ignores night driving.)
I resonantly had an argument on here where somebody insistent that breaking because of over-passes were issues with vision system. Seem pretty clear that it is the resolution of the radar, not the shadow of the bridge that causes the issue. Good to get some more insight into this.<p>This is the right thing to focus on, as it is by far the largest issue with Autopilot on the highway. Multiple people who do testing of these system that false positives on some highway overpasses are the biggest usability issue.
Per "it's unscalable to get HD 3D maps of all the roads on earth", it's interesting to consider that Google/Waymo has been growing this for years with street view and the sensors on each car. Curious to see how that plays out
Even if FSD takes longer, I sure am glad about the active safety features to prevent dumb incidents. Hope that trickles to every other manufacturer petrol/electric
What an annoying charlatan. Karpathy is a brilliant computer vision engineer, but he has let his expertise in that subfield cloud his judgement on achieving the overall goal of autonomous driving.<p>Musk and Karpathy have been dead wrong about LIDAR for years. Remember Musk making the absurd claim of a million Tesla robotaxis by 2020? I think most hilarious is that both Karpathy and Musk claim the LIDAR systems are too expensive. Yet, in the same 2019 Autonomy Day they simultaneously claimed that Teslas would be able to drive themselves and operate as robotaxis, earning their owners passive income and therefore justifying significantly increased MSRPs. So, the $7k LIDAR system (that accelerates safe autonomous driving) is not worth the cost, yet stumbling towards autonomy on vision only is? If the car becomes an money-earner, you should use all of the systems available. The 2019 Autonomy Day was an utter embarrassment. I'm sure 2021 will be more of the same.<p>So now it seems that they've realized their folly in logic. So what's the solution? Well, you can't just complain about COST of non-vision perception systems. Because, as noted above, that doesn't make sense if you're going to simultaneously claim that your car will be able to earn you money (augmenting any extra hardware cost that gets you to that point faster). No, now you have to smear all non-vision perception systems. You have to say that their data is worthless and detrimental to the overall effort.<p>The entire claim from the 2019 Autonomy Day that "vision is what humans use to drive" is also completely bogus. Humans use many senses to drive. They feel the pedals and steering wheel. They use their equilibrio sense to sense motion. And they use their hearing to hear other vehicles, sirens, and issues with their own car (driving with headphones in is illegal for a reason). Any modern car, even a Tesla, is also using far more than just vision when attempting autonomy. Forget about radar and LIDAR for a moment. There are endless sensors in the drivetrain. Steering angle sensors and multiple IMUs for the electronic stability control. Brake and wheel sensors for the ABS. Temperature sensors everywhere. And countless other ECUs. The notion that vision is getting you there exclusively is nonsense. There's no good argument against LIDAR today other than perpetuating a lie to sell cars that are cheaper to produce. And, Karpathy has a massive professional conflict of interest in making CV the main player -- he's a CV expert. He was never a fusion expert before his hire. If CV is the pathway forward, he's gets to remain "the guy". It certainly behooves HIM to make that claim.<p>Autonomous driving will not be achieved in this decade. Perhaps ever. Ask yourself honestly: if you were tasked with building an autonomous commercial aircraft OR an autonomous car, which would you choose? Most would say aircraft -- nothing to really hit in the air, fully mapped airport and runway systems, and far fewer variables. Yet autonomous aircraft still do not exist. Perhaps the edge cases always rule the roost. Ask yourself why driving would be any different...
I like Andrej from his PhD research days and awesome blog posts but this is a series of disasters in the making, that is until FTC steps in after more people die from “self-driving” accidents under interesting and unexpected circumstances.<p>The whole vision vs. LIDAR stuff is a distraction as long as Tesla “AI” doesn’t have common sense.<p>It literally doesn’t know what it’s doing, and the tail of edge cases to "fit" the models is infinitely long. ANNs are fundamentally backwards looking and cannot adapt to unforeseen or even slightly unusual combination of circumstances. It will go fine for n miles and will dramatically fail at mile n+1 where a new situation requires understanding of ones surroundings, and n is arbitrary number.<p>It would be more honest to show the cases where it missed, thankfully there is no lack of them in “FSD beta“ videos on YouTube.
I've read claims that they are desperately trying to hire.<p><a href="https://mobile.twitter.com/TaylorOgan/status/1407051918317395968" rel="nofollow">https://mobile.twitter.com/TaylorOgan/status/140705191831739...</a>