03/08/2022

    Torch Relay Commemorative Photography × Ultra-realistic Communication Technology Kirari!NTT Human Informatics Laboratories

    Torch Relay Commemorative Photography × Ultra-realistic Communication Technology Kirari!

    • Olympic and Paralympic Games
    • Torch Relay
    • ultra-realistic communication technology Kirari!

    Yoshiyuki Mihara, Masami Nagata, Keisuke Hasegawa,Taiji Nakamura, Junji Watanabe, Tatsuya Matsui,Hideaki Iwamoto, and Shingo Kinoshita
    NTT Human Informatics Laboratories

    Overview

    To foster the momentum of the Olympic Games Tokyo 2020, we wanted as many people as possible to experience the Olympic Torch Relay. We therefore created three experiential exhibits that enabled participants to take commemorative photos with a torch as a personal experience of the Olympic Torch Relay. The first exhibit was the Virtual Torch Kiss, Torch Relay Commemorative Photo Corner: Kirari! Version, where participants could carry an actual torch and virtually experience a torch kiss using the Kirari! ultra-realistic communication technology. Kirari!, which conventionally needs to be set up in a laboratory and include large devices, was packaged into a truck, making it possible to hold the exhibit anywhere around the country. The second exhibit was the Torch Relay Commemorative Photo Corner: Optical Illusion Version. By projecting a special light that elicits phantasmagoric optical illusion onto a still image, we created the mysterious effect of making a supposedly unmoving image of a flame appear to move. Unlike shooting in front of a regular panel, it was possible to provide a more dynamic experience despite using still images. The third exhibit was the Torch Relay Commemorative Photo Corner: Kabuki Version. This exhibit combined kabuki, which has a long history in Japan, and the Olympic Torch Relay, which connected the regions of Japan, by enabling participants to take a commemorative photo with a torch while having kabuki makeup virtually applied to their faces.

    Torch Relay Commemorative Photo Corner: Kirari! Version

    We modified a truck that enabled participants to do a virtual torch kiss with the professional tennis player Kei Nishikori, who was holographically projected using Kirari! inside the truck (Fig. 1). The holographic display was based on a two-dimensional (2D) aerial image display [1]. We set up the display on the floor of the truck and mounted a transparent screen at a 45-degree angle from the display. The image of Kei Nishikori shown on the display was reflected and transmitted to the transparent screen, making it appear as if he were standing on the stage.
    Setting up the stage for this purpose usually requires time for installing truss parts and setting the transparent screen diagonally. For this celebration event, since we had to move and install the transparent screen on the truck every day, we implemented a structure that made it easy to store and install the transparent screen. Thus, we were able to complete the stage setup, which would usually take a day, in only about two hours. Because also we had to travel throughout the country for 121 days, we needed to display the exhibit in a bright environment during the day, depending on the actual venue and the time of the event. Therefore, we used a light-emitting diode (LED) panel instead of a projector to improve the brightness of the original image and enhance outdoor visibility. We installed these pieces of equipment on the loading deck of a 10-ton truck.

    Fig. 1. Kirari! truck exterior view.

    Flow of experience

    Participants first climbed onto the stage on the truck’s loading deck from the rear entrance of the vehicle and received a torch from the staff. We also installed a lift to enable wheelchair users to experience the exhibit.
    Once the participant bearing the torch was in the proper position, a holographic image of Kei Nishikori appeared on the stage in such a way that the virtual Kei Nishikori and real-life participant appear to be standing side-by-side (Fig. 2). We prepared two patterns for displaying the holographic image of Kei Nishikori: one where he was standing up and one where he was kneeling. This made it possible for participants to make a torch kiss regardless of their height.
    Since participants could not see Kei Nishikori directly on the transparent screen, we prepared a separate monitor through which the participants could view the movement of Kei Nishikori’s torch as they moved their torch toward his torch and do a virtual torch kiss. The torch carried by the participant was equipped with an infrared sensor to detect its position. As the participant’s torch approached Kei Nishikori’s, its tip lit up with an image of the Olympic flame.
    The experience was captured using a camera installed off stage, and the video was instantly uploaded to an Internet server. We provided a means for issuing a 2D code of the video’s URL (uniform resource locator) to the participant after the experience so that they could immediately view the recorded video on their smartphone and post it on social networking services (SNS).

    Fig. 2. Torch Relay Commemorative Photo Corner: Kirari! Version experience.

    Results

    The novel coronavirus (COVID-19) continued to spread during the Olympic Torch Relay period, and many municipalities canceled local torch relay celebration events, so the number of exhibitions was less than originally planned. We were able to hold the torch commemorative photo corner using the Kirari! truck at 34 venues in 25 prefectures, enabling a total of 4498 people to experience the exhibit. Many people were surprised at the realness of the life-size holographic image of Kei Nishikori appearing on stage, with many participants saying that they were able to gain a realistic experience of being an Olympic torchbearer.
    To improve the realness of the Torch Relay experience, we added a function to simulate the tactile sensation of the fire being lit. This allowed people with visual impairments to also enjoy the experience (Fig. 3). Since the torch kiss experience with Kei Nishikori’s holographic image was visual based, visually impaired individuals had no way of knowing when and how it was done. Therefore, by adding a function that vibrated the torch simultaneously with the torch kiss, we were able to deliver a haptic experience for the passing of the flame from Kei Nishikori’s torch to the participant’s.

    Fig. 3. Visually impaired person trying the virtual torch kiss experience.

    Torch Relay Commemorative Photo Corner: Optical Illusion Version

    In this exhibit, a special light was projected onto a panel showing an image of Kei Nishikori bearing a torch to trigger the illusion that the flames on the panel and the flurry of cherry blossoms in the background are moving (Fig. 4). Position matching was important for this exhibit because light was applied on the feature points on the panel. To accurately carry out alignment at each venue, we accurately projected the light to a specific location on the panel by linking the projector with the camera. This made it possible for the local staff to easily set up the exhibit.
    A total of 5616 people in 3373 groups experienced this exhibit. All participants were surprised at the movement of still images that were not supposed to move.

    Fig. 4. Torch Relay Commemorative Photo Corner: Optical Illusion Version experience.

    Torch Relay Commemorative Photo Corner: Kabuki Version

    NTT began its initiatives to create new experiences by combining traditional culture with information and communication technology (ICT) in 2016. As part of these initiatives, we have been focusing on kumadori, the distinctive kabuki style of makeup, and created the unusual experience of having your chosen kumadori mask virtually adhere to your face using ICT. Packaging this experience exhibit in a container, we have traveled to Fukushima and Kumamoto prefectures to provide a fun experience through ICT. Building on these initiatives, we provided the Torch Relay Commemorative Photo Corner: Kabuki Version as an experience with the theme of “connecting,” as exemplified by the Olympic Torch Relay that connected the regions (Fig. 5).

    Fig. 5. Torch Relay Commemorative Photo Corner: Kabuki Version booth.

    Flow of experience

    First, participants chose their preferred mask from among the many painted with kumadori makeup (Fig. 6). We also projected the same special light used in the optical illusion version to make it appear that the kumadori masks hanging on the wall were casually changing their expressions. Once the participant stood in front of the monitor and placed their chosen mask on their face, the system automatically recognized the kumadori mask and overlaid it on the face of the participant. The kumadori mask had augmented reality (AR) features, i.e., it followed the person’s facial expressions and movements in all directions on the monitor in real time while the mask remained attached to the person’s face. The background in the monitor was also rendered to match the selected kumadori mask. We also prepared a background specifically inspired by the Olympic Torch Relay (Fig. 7). With their chosen mask attached to their faces, participants could pose any way they wanted while carrying the torch. Going to the next room, participants then found their faces being projected onto a gigantic 1.2-meter-high, three-dimensional (3D) face, to give them an element of surprise (Fig. 8). Photos of the experience were later posted on SNS and made available for download.

    Fig. 6. Selection of kumadori mask.
    Fig. 7. Torch Relay Commemorative Photo Corner: Kabuki Version experience.
    Fig. 8. A gigantic 3D face with a height of 1.2 m.

    Results

    This experience was intended to be offered at the Olympic Torch Relay expanded celebration events at two locations, namely, the Expo ’70 Commemorative Park in Osaka and the Yokohama Red Brick Warehouse Square in Yokohama. However, due to the spread of COVID-19, it was only held at the Yokohama Red Brick Warehouse Square. A total of 82 people, including Olympic torchbearers and other people involved, experienced the exhibit. Participants laughed once they saw their chosen kumadori masks attached to their faces upon holding the masks up, and many got into character as kabuki actors while taking commemorative photos with the torch in hand.

    Summary

    Due to the spread of COVID-19, the number of people who experienced the exhibits was less than planned. Nevertheless, we were able to deliver surprises and smiles to the participants at the venues where we could provide the experiences. We believe that these interactions with the advanced technologies of our laboratories, which are not normally accessible to the public, enabled many people to experience firsthand NTT’s innovativeness.
    Going forward, we will continue to work on creating new experiences using ICT and connect people with each other.

    Acknowledgments

    We would like to express our gratitude to the members of the Tokyo Organising Committee of the Olympic and Paralympic Games and the representatives of the local governments who were involved in the operations. We also would like to thank the members who handled the operation of the Kirari! truck at the celebration venues.

    Reference

    1. [1]S. Uchida, E. Ashikaga, M. Imoto, M. Wagatsuma, and K. Hidaka, “Concept of Immersive Telepresence Technology ‘Kirari!’,” Proc. of the Media Computing Conference 2015, p. 39, 2015 (in Japanese).
    Yoshiyuki Mihara, Masami Nagata, Keisuke Hasegawa,Taiji Nakamura, Junji Watanabe, Tatsuya Matsui,Hideaki Iwamoto, and Shingo Kinoshita

    Yoshiyuki Mihara, Masami Nagata, Keisuke Hasegawa,Taiji Nakamura, Junji Watanabe, Tatsuya Matsui,Hideaki Iwamoto, and Shingo Kinoshita

    Related content