Technique for Modulating the Tactile Sensation of Objects Using an IllusionNTT Communication Science Laboratories

Takumi Yokosaka, Scinob Kuroki, and Shin’ya Nishida

Compared with visual displays (monitors) and auditory displays (loudspeakers), it may seem that tactile displays have not reached a practical level, but their ability to represent tactile sensation is steadily improving. For example, the controller of a video-game console can simulate the tactile texture (how something feels to the touch) of a variety of objects by controlling a built-in oscillator. However, a technique capable of eliciting the perception of a tactile sensation, i.e., giving someone the impression that she/he is physically touching the object in question rather than receiving a tactile sensation via a controller, is yet to be established. When someone touches an object directly, his/her perception of the texture of that object is based not only on the kinetic sensation on the skin but also on various other physical properties, such as shape, irregularities on the surface, elasticity, heat conductivity, and moisture content. This is why it is difficult to fabricate a device that can flexibly represent the complex properties and states of an object. A clue to solving this problem is using a tactile illusion.
Visual illusions are well known, such as an object that ought to appear stationary looks as if it is moving. However, there are also tactile illusions, which are perceived via touch on the skin of the hands, for example. Tactile illusions can take a variety of forms depending on the properties that someone feels are different from the actual properties, such as shape, weight, movements, and tactile texture [1, 2]. Tactile illusions are important because they provide a clue as to how we estimate the properties of an object when we touch it. This article focuses on the velvet hand illusion, which is a tactile texture illusion. This illusion occurs when wires, such as tennis racket strings, are held between two hands and either the hands or the wires are moved back and forth, resulting in a tactile sensation that is strange and different from that of the wires (Fig. 1) [3]. The reason it is called the “velvet” illusion is due to the fact that the strange tactile sensation produced is similar to the sensation of touching velvet. This sensation provides a significant hint about tactile illusions. That is, in this illusion, a sensation like that of touching velvet, while completely at odds with the real object being touched, is produced directly on the skin of the palms. This illusion can be induced simply by moving the wires despite the fact none of the complex characteristics of the wires, such as shape, irregularities on the surface, and elasticity, are controlled. We surmised that a close examination of this phenomenon could provide a clue as to how to create a variety of tactile but illusory sensations directly on the skin of the hands.

When someone experiences the velvet hand illusion, she/he finds it difficult to describe precisely the strange tactile sensation she/he feels when the wires are moved. This makes it difficult to define the question that the study of the velvet hand illusion needs to focus on. That is, it is difficult to understand qualitatively and quantitatively what type of illusion the velvet hand illusion is and how it is caused. Therefore, we conducted an experiment to gain a clearer understanding of what kind of phenomenon the velvet hand illusion is [4]. In this experiment, the participants compared the tactile sensation when wires were moved between the two palms, namely, what they felt while the velvet hand illusion was occurring (Fig. 2(a)) and the tactile sensation produced with various commonly used materials (Fig. 2(b)). To identify how the tactile sensation varies when the intensity of the velvet hand illusion changes, we controlled the intensity of this illusion by presenting a variety of wire-related conditions, i.e., manipulating the distance between the wires, moving them vertically or horizontally, and touching them only with one hand (Fig. 2(c)). We also asked the participants to indicate the intensity of the velvet hand illusion for each condition.

The degree of similarity between the tactile sensation in the velvet hand illusion and the real texture of various materials was visualized on a two-dimensional (2D) space, as shown in Fig. 3. In other experiments in which the textures of various materials and those in the velvet hand illusion were evaluated in terms of roughness and hardness, we found that the horizontal axis of the 2D space corresponds to softness and warmth and the vertical axis to smoothness. The intensities of the illusory sensations evaluated under various wire-related conditions were not used in this analysis. Nonetheless, the different tactile sensations were plotted linearly from the condition of weak illusion intensity to that of strong illusion intensity. The correlation coefficient between the illusion intensity and horizontal axis was 0.75 and that between the illusion intensity and vertical axis was –0.93. It was found that the illusion was the weakest when the wires were touched with one hand (bottom right in Fig. 3). In this case, what was felt when touched most closely resembled the actual texture of a wire mesh. This seems a reasonable result considering that both the wires and a wire mesh are hard and linear materials. It was also found that the stronger the illusion, the more the tactile sensation shifts towards the upper left in Fig. 3, indicating that the tactile sensation gets softer and smoother. The illusion was the strongest when the wires were 75 mm apart, sandwiched between both hands, and moved back and forth. The tactile sensation was similar to the real texture of cloth or leather. These experimental results revealed that the velvet hand illusion is a phenomenon in which humans perceive that the texture of wires, which are inherently hard and rough, changes into something soft and smooth such as cloth or leather when the wires are moved between the hands.

It has become clear that the velvet hand illusion is a phenomenon in which the perceived texture of directly touched wires changes. Can we apply this illusion to other materials besides wires? This is an important question to ask when studying potential techniques of presenting various tactile sensations. One obvious technique is to sandwich the wires between one hand and an object other than the other hand. However, there are several problems with this technique. For example, when wires are moved back and forth between a hand and object, the close contact between the hand and object can be lost, making the tactile sensation produced by the wires dominant. In addition, if the object has a rough surface, the wires can snag, making the movement bumpy. To solve these problems, we have taken into account the fact that the velvet hand illusion can also be produced by moving a thin board with a hole in it, instead of wires, between two finger pads [5]. This technique also allows the board to be sandwiched between two hands (Fig. 4(a)). It has also been found that the velvet hand illusion can be produced not only by moving the hole back and forth but also by rotating it (Fig. 4(b)). These findings suggest that it is not necessary to move the hole across the entire palm to produce the illusion. This led us to hypothesize that rubbing the hole with the outer part of the palm could produce the illusion in the central part of the palm confined by the edge. We have thus discovered a technique of manipulating the tactile sensation of a touched object by rotating a sheet of heavy paper held between a hand and object (Fig. 4(c)). We refer to this technique as the frame-rotation technique. This technique does not require anything to move a long distance between a palm and object. Also, because few edges exist in the rotating direction, they do not get caught on the object; thus, the paper can be moved smoothly.

We conducted experiments to ascertain whether this technique can make a range of objects feel soft and smooth [6]. Participants were asked to evaluate the textures of various materials and how similar the textures of the same materials were when the frame-rotation technique was applied. As in Fig. 3, the experimental results were visualized in a 2D space, as shown in Fig. 5. In other experiments in which the textures of these materials were evaluated in terms of roughness and hardness, we found that the horizontal axis in the 2D space is related to softness and warmth and the vertical axis to smoothness. For example, it was found that when the frame-rotation technique was applied to a rough and hard carpet, the perception of the carpet’s texture changed to being softer and smoother. Thus, this technique makes the texture of various objects seem softer than they actually are.

As discussed above, we developed and evaluated a technique using the phenomenon in which the texture of an object can be made to seem softer and smoother, similar to the velvet hand illusion. The study of the brain mechanism that causes this illusion may enable us to discover hitherto unknown tactile-sensation processing mechanisms. It would also allow us to develop a technique for making the perception of an object’s texture seem harder and rougher and one that allows the perception of softness and smoothness to be changed independently. A characteristic of this newly developed technique is that it requires no special device. Anyone can easily implement it using items found at home. Thus, anyone without specialist knowledge on the operation of a vibration-presentation or power-generation device will be able to easily control the perception of texture. For example, this technique would be easy to use when a designer wants to convey the tactile texture of a package to his/her clients or a sales clerk wants to demonstrate the effect of a product such as a softening agent.

1. [1]M. Delcroix, K. Zmolikova, K. Kinoshita, S. Araki, A. Ogawa, and T. Nakatani, “SpeakerBeam: A New Deep Learning Technology for Extracting Speech of a Target Speaker Based on the Speaker’s Voice Characteristics,” NTT Technical Review, Vol. 16, No. 11, pp. 19–24, Nov. 2018.
   https://ntt-review.jp/archive/ntttechnical.php?contents=ntr201811fa2.html
2. [2]K. Zmolikova, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. Cernocky, “SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures,” IEEE JSTSP, Vol. 13, No. 4, pp. 800–814, Aug. 2019.
3. [3]A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to Listen at the Cocktail Party: A Speaker-independent Audio-visual Model for Speech Separation,” ACM Trans. Graph., Vol. 37, No. 4, Article 112, pp. 1–11, Aug. 2018.
4. [4]T. Ochiai, M. Delcroix, K. Kinoshita, A. Ogawa, and T. Nakatani, “Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-visual Speaker Clues,” Proc. of INTERSPEECH 2019, Graz, Austria, Sept. 2019.
5. [5]H. Sato, T. Ochiai, K. Kinoshita, M. Delcroix, T. Nakatani, and S. Araki, “Multimodal Attention Fusion for Target Speaker Extraction,” Proc. of IEEE Spoken Language Technology Workshop (SLT) 2021, pp. 778–784, Jan. 2021.
6. [6]Demonstration page of the paper [5],
   http://www.kecl.ntt.co.jp/icl/signal/member/demo/audio_visual_speakerbeam.html
7. [7]I. Medennikov, M. Korenevsky, T. Prisyach, Y. Khokhlov, M. Korenevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, “Target-Speaker Voice Activity Detection: A Novel Approach for Multi-speaker Diarization in a Dinner Party Scenario,” Proc. of INTERSPEECH 2020, Shanghai, China, Oct. 2020.
8. [8]M. Delcroix, S. Watanabe, T. Ochiai, K. Kinoshita, S. Karita, A. Ogawa, and T. Nakatani, “End-to-end SpeakerBeam for Single Channel Target Speech Recognition,” Proc. of INTERSPEECH 2019, Graz, Austria, Sept. 2019.
9. [9]T. Ochiai, M. Delcroix, Y. Koizumi, H. Ito, K. Kinoshita, and S. Araki, “Listen to What You Want: Neural Network-based Universal Sound Selector,” Proc. of INTERSPEECH 2020, Shanghai, China, Oct. 2020.
10. [10]Demonstration page of the paper [9],
    http://www.kecl.ntt.co.jp/icl/signal/member/tochiai/demos/universal_sound_selector/index.html
11. [11]Y. Ohishi, A. Kimura, T. Kawanishi, K. Kashino, D. Harwath, and J. Glass, “Pair Expansion for Learning Multilingual Semantic Embeddings Using Disjoint Visually-grounded Speech Audio Datasets,” Proc. of INTERSPEECH 2020, Shanghai, China, Oct. 2020.

• Reach Out and Touch Someone’s Heart: Exploring the Essence of Communication to Create a Spiritually Rich Society
• The Day a System Becomes a Conversation Partner—Exploring New Horizons in Social Dialogue Systems with Large-scale Deep Learning
• Looking More, Acting Better
• Developing AI that Pays Attention to Who You Want to Listen to: Deep-learning-based Selective Hearing with SpeakerBeam
• Technique for Modulating the Tactile Sensation of Objects Using an Illusion