Technical Developments and Verification of Connected CarsNTT Digital Twin Computing Research Center
NTT Software Innovation Center

Atsushi Isomura, Kazuya Matsuo, Yuki Yokohata, Yoshikazu Hirabayashi, and Atsushi Koizumi

After the collaboration agreement was concluded ^[1], Toyota Motor Corporation and the NTT Group have been developing an information and communication technology (ICT) platform for connected cars by taking future automated driving into consideration. To achieve safe and secure automated driving, various data sent from vehicles must be collected and used in a cloud. The ICT platform is being developed as a means to achieve this.

A connected car is a car that also functions as an ICT terminal. Since a vehicle holds many items of information, it is possible to understand the vehicle and its surroundings from the data it sends. If we are to solve social problems, such as traffic accidents and traffic jams, and upgrade mobility services, it is essential to use data collected from connected cars via a network.

Many key technologies are required to support connected cars: technology to collect and store a large volume of vehicle data sent by numerous vehicles; technologies to analyze the collected large volume of data in real time and reproduce the surroundings on a cloud; artificial intelligence (AI) algorithms to determine the behavior of vehicles; technology to collect and distribute a large volume of data; 5G (fifth-generation mobile communication system); edge computing; and communication technology to collect and distribute a large volume of data reliably and securely using Internet of Things (IoT) networks.

Specifically, a large volume of data from electronic control units (ECUs) installed in vehicles on roads and sensor data, including images from cameras, need be collected and stored in a cloud reliably and efficiently. From the collected data, the status of vehicles and their surroundings need be reproduced in cyberspace in real time. These processes make it possible to generate maps to be used in next-generation traffic systems and allow various services and automated driving. The ICT platform for connected cars will support the reproduction of the status of vehicles and their surroundings in cyberspace (Fig. 1).

The development of the ICT platform for connected cars is being undertaken through short cycles of studying individual key technologies and a demonstration experiment in which these technologies are combined and verified on an end-to-end connection, including terminals, a network, and datacenter. In the demonstration experiment, data from multiple experiment vehicles running on roads are collected via a network and stored in a cloud. The collected data are analyzed to confirm whether technical issues identified for use cases of connected cars and automated driving have been solved. In addition, vehicle data equivalent to millions of vehicles are generated to evaluate the feasibility of communication and datacenter technologies.

Specific uses cases being studied are: (1) generation of a high-precision map, (2) detection and notification of obstacles, and (3) traffic-jam detection in each lane. The extent to which the technical requirements of each use case are satisfied is being examined.

(1) Generation of a high-precision map

Whether video data captured with vehicle-mounted cameras can be sent to the cloud and whether information about vehicle positions and about the surroundings can be reproduced in cyberspace are being examined (Fig. 2). In particular, we are developing a technology for collecting video data and location information from vehicles and estimating the positions of road signs, road marks, and other structures from the collected video data captured with vehicle-mounted monocular cameras with a high degree of precision. Currently, changes in road surfaces are examined by driving a dedicated measurement vehicle over the road to be examined. This method is costly and cannot produce examination data in real time. Instead, our technology can be used to produce a high-precision map from image data captured with an ordinary vehicle-mounted camera.

(2) Detection and notification of obstacles

We are examining whether data sent by a vehicle can be shared with other vehicles in real time (Fig. 3). Specifically, when a vehicle-mounted monocular camera captures image data of an obstacle, the platform receives its data from the vehicle and estimates the position of the obstacle. The information about the obstacle is registered on a dynamic map. The platform quickly searches for vehicles that can be affected by the obstacle. Finally, the information about the obstacle is transmitted to these vehicles. We are developing a technology that accelerates the processing of these steps in less than 10 seconds.

(3) Traffic-jam detection in each lane

We are verifying whether the level of congestion in neighboring lanes can be reproduced from video data captured with a vehicle-mounted monocular camera. In particular, we are developing a technology for identifying the lane in which the vehicle runs and accurately estimating the level of congestion in neighboring lanes from the video data.

Technical targets have been defined for the key technologies being used in the demonstration experiment by assuming several use cases. The targets are the ability to connect tens of millions of vehicles to the system and perform the above steps within just a few seconds. This demonstration experiment began in 2018. It has been gradually expanded and will continue until the end of fiscal year 2020. We have already confirmed that 5 million vehicles can be connected to the system and that all the required steps can be performed in less than 20 seconds.

The ICT platform for connected cars involves a number of technical challenges. The main technologies being developed to meet these challenges are described below (Fig. 4).

(1) Object-position-estimation technology

This technology estimates the positions of display objects (road signs, signboards, etc.) in the road surroundings from vehicle status data and video data captured with vehicle-mounted cameras. It can even process data from low-end dashcams. It improves positional accuracy by selecting high-quality data from a large volume of data and using statistical methods. Accurate recognition of three-dimensional objects, such as road signs and traffic lights, on roads and high-precision estimation of the positions of these objects make it possible to map the objects on a map. This technology is expected to be used for generating maps for automated driving and automatically updating them.

(2) Algorithm for selective collection of vehicle data

Data collected from connected cars are both voluminous and varied, ranging from vehicle data to video and sensor data. Since these data are collected consecutively from a large number of vehicles, the volume of collected data is enormous. This imposes a huge load on the network, and data processing capacity is stretched to the limit. To solve these problems, we developed a technology for selecting data collected from vehicles based on meta-information, such as location information, time information, sensor type, resolution, observational range, and observation impairment caused by the presence of surrounding vehicles. This technology makes it possible to efficiently collect data with minimum duplication of observational ranges and adjust the volume of collected data in accordance with the load on communication links and the analysis platform.

(3) Real-time spatio-temporal data-management technology

This technology enables a large volume of data to be stored instantaneously and searches a specified range (time and space) of data at high speed. It is a spatio-temporal database that manages time and location information generated by moving objects, such as vehicles and smartphones, as one-dimensional data using a unique method and enables the searching for those moving objects within a specified time and rectangular space range. It also searches within a space with a complex shape, such as a road or parking lot. Therefore, it enables the storing of sensor information generated by a large number of moving objects in real space, and time and space information associated with each item of sensor information, as well as searching for this information within a specified time and space range.

(4) Vertically distributed computing technology

This technology processes applications at high speed by offloading computation tasks done at datacenter servers to edge servers. Since the processing capacity of the system server is limited, it is sometimes not possible for the system to respond to all devices instantly. In reality, the required response time varies depending on the status of each vehicle (vehicle speed, moving direction, time of day, etc.). For this reason, vertically distributed computing technology dynamically distributes applications between edge servers and datacenter servers based on the status of vehicles. This technology can use limited server resources effectively while ensuring that the system responds instantly to those vehicles that require a quick response. This technology enables the system to serve a large number of vehicles and to collect and respond to a large volume of data. Thus, limited distributed computing resources can be used efficiently.

(5) Traffic-jam detection in each lane

This technology detects traffic jams in each lane from vehicle-status data and video data captured with vehicle-mounted cameras. It detects a queue of vehicles from the relative locations of vehicles moving along the surrounding lanes. It can also recognize the cause of a queue by identifying the first and last vehicles and combining this information with the surrounding facility information. Currently, information about traffic jams by road is provided by the Vehicle Information and Communication System (VICS) and smartphone map apps. When the above technology has been developed and in wide use, more valuable traffic-jam information will become available. As with the object-position-estimation technology described above, this technology can detect a traffic jam using video data captured with even low-end dashcams.

It is said that the automobile industry is facing a once-in-a-hundred-year transformation. The environment surrounding automobiles has been changing dramatically over the last few years. One of these changes is the spread of connected cars, resulting in the emergence of various services for drivers, fellow travelers, automakers, insurance companies, and service garages.

When it will become possible to process a large volume of data on vehicles quickly and at low cost, information that cannot be captured using current sensors will become available, and faster and more reliable services than today will be provided. These developments will not only enhance convenience and efficiency but also provide safety and security to the coming age of automated driving.

1. [1]Press release issued by NTT on Mar. 27, 2017 (in Japanese),
   https://group.ntt/jp/newsrelease/2017/03/27/170327a.html