Nvidia’s neural network taught itself to drive cars

Self-driving cars are quickly becoming one of the hottest things in the automotive industry. But is driving easy enough for a neural network to learn with little human input? Well, Nvidia has done just this.

The tech company fed data from the steering wheel and a front-facing camera in the car to “convolutional neural networks” (CNNs). The neural networks then conducted steering commands in reaction to said data.

“This powerful end-to-end approach means that with minimum training data from humans, the system learns to steer, with or without lane markings, on both local roads and highways. The system can also operate in areas with unclear visual guidance such as parking lots or unpaved roads,” Nvidia wrote in a blog post.

The company also shed light on how it trained the system.

“We never explicitly trained it to detect, for example, the outline of roads. In contrast to methods using explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimises all processing steps simultaneously.”

The training data was collected by driving on a variety of roads, such as residential roads, two-lane roadways (with and without markings), tunnels and more. The data was also collected by driving in a variety of weather conditions and times of day.

Feeding data to the system

Once the data has been collected, selected frames are fed to the neural network, Nvidia explained.

“Our collected data is labeled with road type, weather condition, and the driver’s activity (staying in a lane, switching lanes, turning, and so forth). To train a CNN to do lane following, we simply select data where the driver is staying in a lane, and discard the rest.”

The company also adds artificial “shifts and rotations” to the frames, in a bid to teach the network how to recover from poor positions.

So what were the initial results then?

“For a typical drive in Monmouth County NJ from our office in Holmdel to Atlantic Highlands, we are autonomous approximately 98% of the time. We also drove 10 miles on the Garden State Parkway (a multi-lane divided highway with on and off ramps) with zero intercepts,” Nvidia explained.