What about Tesla Autopilot’s Rebuilding from Elon Mask’s Twitter info?

5 min readAug 31, 2020

Elon wrote on Twitter: “Tesla is developing a NN training computer called Dojo to process truly vast amounts of video data. It’s a beast! Please consider joining our AI or computer/chip teams if this sounds interesting.”
Elon also wrote: “The FSD improvement will come as a quantum leap, because it’s a fundamental architectural rewrite, not an incremental tweak. I drive the bleeding edge alpha build in my car personally. Almost at zero interventions between home & work. Limited public release in 6 to 10 weeks.”
He wrote: “Autopilot was trapped in a local maximum, labeling single camera images uncorrelated in time. Now, it is not.”

Tesla Autopilot from Elon Musk’s Twitter

Here it is guessed from my point of view, what gonna happen with the new Tesla Autopilot program/platform:

Dojo perhaps maybe have used the server version of FSD chip, i.e. supporting NN training, not only inference at the vehicle; Also it is reported Broadcom has codesigned with Tesla the new FSD HPC chip, crafted using TSMC’s advanced 7nm process;

Unsupervised learning is a hot point in recent deep learning community, while the video data, compared with the image data, provides more chances or possibilities for self supervised/unsupervised learning;

**Andrej Karpathy’s talk at ICML (June 2019)**

Andrej Karpathy’s talk at Scaled ML conference, 2020

“If artificial intelligence is a cake, self-supervised learning is the bulk of the cake,” LeCun says. “The next revolution in AI will not be supervised, nor purely reinforced.”

From Andrej Kaparthy’s presentation at Scaled ML ’20 and CVPR’20, it loosk the crowded sourcing HD map solution has been on the way；1) Detection and fusion of road lanes and boundaries/curbs could support lane-level map building (at least similar to the style in the mapbox), such as intersections/roundabouts, also with detection and recognition of traffic signs/lights，especially holes/bumps on the road saved in the map, similar to construction zone and cone-based lane; 2) Compared with Mobileye‘s ’REM, the Tesla fleet looks more powerful with 700,000 vehicles on the road driving around the world; 3) For the specific vehicle driven by Elon Mask, it could be reckoned the HD map is fully built for his use on the every-day commute route; 4) Fully Self Driving (FSD), with or without HD Map support, is totally different (not the traditional navigation map);

Andrej Karpathy’s demo at Scaled ML conference, 2020

4D, i.e. spatial 3D plus 1D time dimension, is not new for the computer vision or the autonomous driving field; however the 3D accuracy and reliability is not comparable to LiDAR’s point cloud, which is though relatively sparser; It depends on how good the Autopilot team works on that; Elon said the local maximum exists for 2.5D, actually we understand a driving problem is described by a spatial-temporal domain, rather than a image-depth domain;

The software architecture could be improved: 1) Since the existing Autopilot is defined autonomous driving L3 system (regarded as L2+ by the community), the on-research FSD is at least defined as L4，so their algorithm architectures are different，and it is time to rebuild now; for example, the vehicle planning module, will add more functions, especially with the HD map support，i.e. behavior planning; 2) 3D annotation needs more geometric information, the existing data structure is not good to be “remedied” on the 2.5D data base; 3) it is not sure like Andrej‘s SW2.0, but the NN model change should cause big change in modular input/output;

Change of visualization：Compared with Mobileye L4 demo and also visualization GUI of Google WayMo/GM Cruise/Ford Argo, the new Tesla Autopilot visualization tool is expected, especially with more info. shown given by the built HD map.

What about Tesla Autopilot’s Rebuilding from Elon Mask’s Twitter info?

Reference

Written by Yu Huang