03/08/2022

    Direction for Supporting Torchbearers × Swarm Communication Control TechnologyNTT Human Informatics Laboratories

    Direction for Supporting Torchbearers × Swarm Communication Control Technology

    • Olympic Torch Relay
    • supporting torchbearers
    • spatial expression using robots

    In connection with the Tokyo 2020 Olympic Torch Relay, NTT held NTT Presents Tokyo 2020 Olympic Torch Relay Celebration events in Yokohama and Osaka and presented a torchbearer-support program that welcomes torchbearers exchanging torch kisses using spatial-expression technology with robots. In this article, we introduce the control technologies used for the display bots that enabled the torchbearer-support program.

    Masafumi Suzuki, Hiroshi Chigira, Hitoshi Yamaguchi, Takuya Indou, and Shingo Kinoshita
    NTT Human Informatics Laboratories

    Overview

    NTT is working to create new experiences by combining art and technology, and one of the results of our research in this area is “Art of Swarms,” a spatial-expression technology using display bots. Spatial expression refers to the creation of an organically changing visual space through the coordinated above-ground movements of a swarm of robots equipped with multiple displays. Unlike projection through the entire floor surface, display bots, which have physical mass, move as a swarm to create a visual space that enables an unconventional and tangible visual experience.
    NTT introduced the concept of spatial expression at the 2018 NTT R&D Forum. Since then, we have expanded the scale and examined the theatrical effects of using display bots, including through demonstrations at the world’s largest art festival in Linz, Austria in 2018 and exhibits at the Sports Viewing Re-Imagined Exhibition in July 2019. At the Sports Viewing Re-Imagined Exhibition, we presented a collaboration of a swarm of more than 30 display bots with prominent artists Akiko Nakayama [1] and Ei Wada [2]. We also used a swarm of display bots for sports expression, which received high praise from many people who watched the exhibit.
    Using the knowledge gained from these demonstration experiments, we applied Art of Swarms to the Tokyo 2020 Olympic Torch Relay. We wanted many people to have a more dramatic experience of the Torch Relay and enable them to witness the possibilities of the fusion of NTT’s communication technologies with art. We directed a festival to welcome the torchbearers arriving at the venue during the NTT Presents Tokyo 2020 Olympic Torch Relay Celebration events (Figs. 1 and 2).

    Fig. 1. Torch kiss scene at the Osaka event.
    Fig. 2. Torch kiss scene at the Yokohama event.

    NTT Presents Tokyo 2020 Olympic Torch Relay Celebration

    Osaka: April 13, 2021, in front of the Tower of the Sun Plaza at the Expo ’70 Commemorative Park
    Yokohama: June 30, 2021, at the Red Brick Warehouse Event Square
    We originally planned for many local people to watch these two events. However, due to the spread of the novel coronavirus (COVID-19), onsite viewing through general admission was canceled; thus, we presented the event online.
    We collaborated with Ars Electronica, a world-class media art research institute, to create dramatic scenes for the stage presentation. The original plan was for members of Ars Electronica to come to Japan to take part in local operations with members of NTT. However, they could not travel due to COVID-19, so instead they provided support online from Austria.

    System configuration and technologies used

    Display bots are robots that move on the ground and equipped with hexagonal displays (Fig. 3). They are composed of a display component at the top and a driving function component at the bottom, each operating independently. This configuration enables the robots to move in any direction while keeping the display surface in a fixed direction, thereby increasing the breadth of the spatial-image expression. In addition to the display, the top part includes devices for communicating, positioning, and controlling the movement of the display bot. The bottom part includes the driving mechanism and battery.
    Figure 4 shows an overview of the entire system, including the display bots. The editing system is used for generating scenarios in which movements of multiple display bots and video-content assignments are programmed on the basis of the desired final image expression for the entire space. The control system then delivers and executes the specific scenarios and video content to the display bots. This enables display bots to move organically as a swarm.
    The position of each display bot must be measured accurately to enable coordinated movements while they move close to each other at a distance of around 10 centimeters. We had used motion capture with infrared cameras and reflective markers for positioning during previous demonstration experiments since they were conducted indoors. The same method, however, could not be applied since the torch relay event was going to be held outdoors. Although Global Positioning System (GPS) is commonly used for outdoor environments, the normal positioning accuracy of GPS exceeds a few meters, making it impossible to apply it as is. We therefore adopted a high-precision positioning system called real-time kinematic global navigation satellite system (RTK-GNSS). However, RTK-GNSS had not been widely used, and using it for simultaneous control of 60 robots had never been attempted. RTK-GNSS enables high accuracy (error of a few centimeters) even outdoors by using correction information generated in real time from observation data at different locations called base stations in addition to the positional information obtained from regular GPS antennas. For this project, we applied NTT DOCOMO’s “DOCOMO IoT high-precision GNSS positional information service” [3] to RTK-GNSS, a first-of-its kind initiative. Using RTK-GNSS enables accurate location information to be obtained, but it does not enable measurement of the exact direction. If the exact direction could not be determined, the direction of each display bot’s display would be different from one other, and the entire image would be become corrupt. We therefore installed two antennas diagonally on the display bot and obtained the direction information using the differential information from the two antennas.
    We used proprietary standard Laird communications (2.4 GHz) and Wi-Fi communications (2.4/5.2 GHz) for communication between the display bots and control system and fourth-generation (4G) Long-Term Evolution (LTE) communications via mobile routers for communication with the DOCOMO IoT high-precision GNSS positional information service.
    Before the display bots start moving, the scenarios are communicated simultaneously to each bot via radio signals from the control system upon the operator’s command. Each display bot then begins to move in accordance with the transmitted scenarios. The bots move in accordance with the programmed scenario, but they can also be controlled in real time by the operator, such as to stop display bots simultaneously or decrease their speed, since they always operate in concert with the control system.

    Fig. 3. Display bots.
    Fig. 4. Overview of system configuration.

    Testing

    Testing with 100 display bots requires a space that is large enough to store, charge, and operate them and that is relatively free from obstructions to enable reception of the satellite signals for RTK-GNSS. We therefore conducted tests at the NTT Tsukuba R&D Center (Tsukuba City, Ibaraki Prefecture), which meets these requirements. After the decision to postpone the Tokyo 2020 Games, at the time of test resumption, the plan was to hold the event in collaboration with members from Ars Electronica. However, COVID-19 infections did not subside as we had expected, increasing the likelihood that they could not come on the day of the event. We therefore promptly changed our policies as follows (Figs. 5 and 6).

    • Control and preparations for actual event and tests at Tsukuba was carried out by only Japanese members mainly from NTT.
    • Ars Electronica members did not come to Japan but supported the operations online.

    As a result, NTT members had to deal with many unfamiliar tasks, such as handling hardware issues, troubleshooting and analyzing problems, and other tasks that were originally supposed to be handled by members of Ars Electronica. Many hours were therefore spent on remote correspondence with members in Austria. Due to the time difference with Austria, there were many time constraints; thus, it was necessary to efficiently carry out testing, problem analysis, and response.

    Fig. 5. Remote collaboration with members of Ars Electronica.
    Fig. 6. Testing at NTT Tsukuba R&D Center.

    Live performance

    We held the first live event in front of the Tower of the Sun Plaza at the Expo ’70 Commemorative Park on April 13, 2021. The final torch kiss with the last torchbearer of the day would have been welcomed through a spatial-expression show by a swarm of display bots, with the Tower of the Sun, the symbol of the Expo ’70 Commemorative Park, in the background.
    Around April 7, when we traveled to Osaka for the preparations, the number of new COVID-19 infections in Osaka Prefecture began to increase sharply, and the declaration of the third state of emergency became imminent. Eventually, on April 7, an announcement was made that the Torch Relay along public roads would be canceled and held at the Expo ’70 Commemorative Park. Although we were able to avoid having the event on April 13 canceled, it was held with no spectators. In addition, the Osaka prefectural government requested us to avoid presenting in front of the Tower of the Sun, which was illuminated in red after sunset during the state of emergency. To comply with this request, we needed to start the event before sunset. However, since the display bots create the magical spatial expression by moving as a swarm of images in the dark, holding the event before sunset meant that the presentation would lose much of its charm. Also, given that many display-bot hardware failures had occurred since we arrived in Osaka and that GPS reception in the field was poor and positioning failed during preliminary rehearsals, to be safe, we decided to carry out the live performance in Osaka with 48 display bots in a stationary state. During the day of the event, while it was raining heavily, the stationary display bots welcomed the torchbearers and rendered a coordinated display of lights along with the torch kiss.
    After the completion of the Osaka event, we were faced with two major issues to be resolved in preparation for the Yokohama event scheduled on June 30. The first issue was the repair of the broken display bots. More than half of the bots had some sort of problem, so we needed to quickly carry out troubleshooting and repair to ensure that as many display bots as possible were operational before the next event. For the repair of the display bots, we determined which ones had to be returned to Austria for repair and which ones could be repaired in Japan. Considering all possible factors, including the number of units necessary for the entire production, number of days required for repair, and number of days required for testing, we created a repair plan and changed it in accordance with the daily situation. As a result, we were able to secure 48 operational display bots before the live event.
    The second issue was the stabilization of their positioning. The location of GPS satellites on the day of the event and the surrounding environment had a significant impact on positioning. Even if we conducted prior testing in the same location and obtained good results, there was no assurance that we could obtain the same results during the actual event. We therefore used two methods to increase the reliability of positioning.
    The first was improving the antenna-installation method. From our experience in Osaka, we found that raising the position of the antenna could lead to more stable positioning. However, installing the antennas too high leads to many issues, such as difficulty in fixing the antenna in place or an unsightly appearance, so it was necessary to find a height that balances functionality with aesthetics. After various tests, we determined that the optimum height was around 15 cm from the surface of the display. The second was achieving failsafe positioning. We needed to ensure that positioning would continue even if the RTK-GNSS positioning fails. To achieve this, we used an odometer to estimate the position and direction using the amount of rotation, etc. of the wheel of the display bots.
    The second event was held at Yokohama Red Brick Warehouse on June 30, 2021. Due to the COVID-19 situation, like in Osaka, onsite viewing through general admission was canceled, and the event was conducted exclusively through online streaming. From rehearsals to the live performance, the display bots did not fail, and their positioning was stable and almost never failed. In the area between Buildings No. 1 and No. 2 of the Red Brick Warehouse, the display-bot program showcased a magical display of lights through coordinated movements of 48 display bots as the last torch kiss of the day was exchanged and the final torchbearer was sent off to ignite the cauldron (Fig. 7).

    Fig. 7. Direction support for the Torch Relay in Yokohama.

    Summary

    Onsite viewing through general admission was canceled for both the Osaka and Yokohama events, preventing the general public from experiencing the display-bot presentation in person. Nevertheless, we consider it a major achievement to have enabled many people to enjoy via online streaming the presentation of lights and movements by the swarm of display bots around the torchbearers and for the show to be featured in various media events, including the opening ceremony presentation, as an iconic scene of the Tokyo 2020 Olympic Torch Relay.
    Going forward, we will continue to study further possibilities of the fusion of art and technology.

    Acknowledgments

    The implementation of the display-bot torchbearer-support program was made possible with the cooperation of many people. We would like to express our sincere gratitude to the people of Osaka and Kanagawa Prefectures for supporting the events, members of Dentsu and Dentsu Live, members of Ars Electronica, and many others who were involved in the preliminary rehearsals, tests, as well as during the live performance.

    References

    1. [1]Akiko Nakayama,
      http://akiko.co.jp/akikoweb
    2. [2]Ei Wada,
      https://eiwada.com
    3. [3]DOCOMO IoT high-precision GNSS positional information service (in Japanese),
      https://www.nttdocomo.co.jp/biz/service/highprecision_gnss_positioning/
    Masafumi Suzuki, Hiroshi Chigira, Hitoshi Yamaguchi,Takuya Indou, and Shingo Kinoshita

    Masafumi Suzuki, Hiroshi Chigira, Hitoshi Yamaguchi,Takuya Indou, and Shingo Kinoshita

    Related content