08/06/2021

    New Technology for Measurement and Analysis of Biological Sounds and Electrocardiographic Signals—Toward Early Detection of Heart Disease and Rehabilitation by Using Personal Heart ModelingNTT Basic Research Laboratories

    Feature Articles: NTT’s Medical and Health Vision toward Creation of Bio-digital Twin

    Masahiro Nakano, Ryohei Shibue, Kunio Kashino, Shingo Tsukada, and Takayuki Ogasawara

    1. Estimating the state of mind and body through biological sounds

    The biological information obtained from measuring the living body reflects the function, morphology, and dynamic characteristics of the living body. We are researching and developing the concept of bio-digital twins to improve people’s well-being by, for example, early detection of diseases. With a bio-digital twin, various types of biological information as signals are captured, and signal-processing and machine-learning technologies are used to model the physical and mental states of each person on the basis of the observed signals. Initiatives that focus on the sounds generated by living organisms are first introduced.

    1.1 Telestethoscope

    Doctors and nurses use auscultation to determine the presence or absence of abnormalities and the degree of urgency. Skilled medical practitioners can visualize in their minds the generation and transmission of biological sounds such as breathing, movement of heart valves, and blood flow. Inspired by such auscultation by medical professionals, we are conducting research and development on a wearable device called the telestethoscope (see Fig. 1(a)).
    The telestethoscope is used for acquiring biological sounds (acoustic signals) from multi-channel microphones and electrocardiograms (ECGs). The acquired acoustic signals and ECG can be sent to a remote terminal via a network. Sounds from various points on the wearer’s body can be heard by touching the screen of that terminal. An overview of the telestethoscope is shown in Fig. 1.
    The telestethoscope will allow medical personnel to listen to acoustic signals of patients remotely without having to come into contact with patients; therefore, it will be useful for examining patients with infectious diseases and assessing the urgency of patient care from a distance (e.g., when the patient is at home or in a rural area) (Fig. 1(b)). Since it is possible to record and share biological sounds, it is also possible to repeatedly auscultate offline or chronologically study the status by comparing current and past sounds. In addition to the above use by medical professionals, we are also researching its use by the general public for self-healthcare, as described below.

    1.2 Adding descriptive text to heart sounds by using the telestethoscope

    It is not easy for non-specialists to understand the information about living organisms by simply listening to biological sounds. However, signal processing and machine-learning technology makes it possible to convert biological sounds into a form that is easy for many people to understand. Therefore, we are investigating converting (translating) biological sounds directly into simple wording (Fig. 1(c)).
    The most-straightforward means of explaining biological sounds would be to classify them as “normal” or “abnormal.” However, if a person’s sounds are classified as “abnormal,” a more-detailed explanation may be needed. For example, it would be useful to know what abnormalities are present and whether the person in question should go to the hospital immediately. To address this issue, we previously proposed a method called conditional sequence-to-sequence caption generation, which allows the user to specify a numerical value called specificity that represents the level of detail of the description to be generated [1]. For example, if the heart sounds and a large number as a desired level of detail are input into the system, a detailed explanation can be output.

    1.3 Estimating health status via video images from heart sounds acquired from the telestethoscope

    Another means of converting biological sounds into an easy-to-understand form is to generate moving images that represent biological functions in real time (Fig. 1(d)). We have attempted to reconstruct the movement of the heart from the sound of the heart. A normal heart moves in a cyclic fashion in which the transitions occur in the following order: atrial systole, isovolumic systole, ejection phase, isovolumic diastole, and filling phase. We therefore aim to estimate the transitional state of the heart from measuring heart sounds then estimate and reconstruct the movement of the heart in three dimensions (3D). Since the heart pumps blood around the body, a specific physical relationship between its pressure and volume is maintained through the state transitions. Focusing on this physical relationship, we devised a method of constructing a 3D moving image. This method uses the physical relationship to improve the accuracy of cardiac motion estimation by restraining it in accordance with the state transition, which is estimated from the heart sounds [2].
    Accurately estimating and predicting the physical and psychological state of an individual from easily observable biometric information, such as sound, will necessitate many other innovations. We will continue to research and develop signal-processing and machine-learning technologies for measuring biological functions and dynamics from multiple perspectives, such as sound information and electrocardiographic information, as described below.

    Fig. 1. System overview of telestethoscope. (a) Telestethoscope wearable device, (b) remote auscultation, (c) automatic sound-description generation, and (d) visualizing estimated 3D cardiac motion in accordance with the estimated heart states.
    Fig. 1. System overview of telestethoscope. (a) Telestethoscope wearable device, (b) remote auscultation, (c) automatic sound-description generation, and (d) visualizing estimated 3D cardiac motion in accordance with the estimated heart states.

    2. Wearable 3D ECG device using new lead system and ECG analysis method called tensor ECG guided by the new system

    To determine the physiological function of the heart, ECGs are widely used in, for example, diagnosis at medical institutions, vital-sign monitoring, medical examinations, and automated external defibrillators. With advances in information and communications technology, such as the Internet and smartphones, as well as in information-processing technologies, such as machine learning, ECGs are being applied in new fields such as artificial-intelligence-guided healthcare. Due to the increase in cardiac diseases in the super-aging society, a society whose population aged 65 or older is over 21%, the need for home care and telemedicine using electrocardiography is rapidly increasing. Taking advantage of our experience in developing a wearable ECG device using hitoeTM, we are developing a wearable device for continuous measurement of ECGs and an ECG analysis method by combining medical knowledge on clinical ECGs with recent information-processing technology [3].

    2.1 3D ECG induction suitable for long-term recording in wearable device

    ECGs are analyzed on the basis of their potential difference (scalar quantity) and shape (pattern), which are obtained from multiple bioelectrodes placed at specified points (standard 12 leads) on the surfaces of the extremities and chest. As a method of recording electrocardiographic potential more three-dimensionally, a vector ECG (which uses a three-axis Cartesian-coordinate system) involves using electrodes on the chest, head, and lower extremities and a resistance-correction circuit. Since both standard-12-lead and vector ECGs involve using electrodes on the extremities, they are easily affected by body movements, so they are basically recorded while the patient is in a resting state. However, electrodes attached to the chest are less affected by body motion and can produce relatively large cardiac potentials, so they are used for Holter ECG monitors (portable devices for cardiac monitoring), exercise-load, and sports ECGs.
    For long-term stable recording of a 3D ECG, we devised a wearable devise for measuring ECG with its reference point set in the apex region of the heart (i.e., on the left anterior wall, where the heart is closest to the thorax) and with counter poles in three linearly independent directions (Fig. 2(a)) [4]. By bipolar induction*1 generated by placing two horizontal leads on opposite sides of the thorax and vertical leads on the upper right precordium in line with the electromotive-force base axis*2, 3D electrocardiographic potential can be stably recorded (Fig. 2(b)). The electrodes and wires are integrated into an elastic belt that can be easily attached by simply tightening the shoulder and waist parts of the belt. We have also developed a non-invasive cardiac polygraph with hemodynamics for simultaneously measuring cardiac output and deep vascular pulse waves (Fig. 2(c)).

    2.2 New method for analyzing ECGs: tensor ECG

    Abnormalities in ECGs are shown as not only a standardized abnormal waveform (pattern) but also as a slight distortion of the shape or change in potential. A method of quantitatively evaluating such atypical ECG abnormalities has not been developed. We are currently designing individual criteria for specific diseases and special patterns to identify them. Regarding the collective action potential of the myocardium, the electromotive force of the cardiac potential has three plateau phases with continuous depolarization, whereas the cardiac potential at the body surface does not usually show such a plateau. We use the molecular-biology and physiological constraints of the membrane potential of each period (from phases 0 to 4) of the action potential to estimate the collective action potential of the myocardium from the electrocardiographic potential on the body surface (i.e., an inverse problem) (Fig. 2(d)). From the conversion of electrocardiographic potential to action potential, atypical distortions are magnified and made clearly visible (Fig. 2(e)). We are examining whether the parameters (tensor[s]) obtained during the conversion to action potential of the myocardium can be used as indicators for quantifying and uniformly evaluating complex abnormal patterns and minute distortions in ECGs. We anticipate that tensor ECG for multi-mode data measurement and analysis will be useful in diagnosing arrhythmias associated with heart failure, ischemic heart disease, and sudden cardiac death.

    Fig. 2. Overview of tensor ECG: a new induction and analysis method.
    Fig. 2. Overview of tensor ECG: a new induction and analysis method.
    1. *1Bipolar induction: An ECG records the flow of electricity in the heart, and induction is the process of creating a flow of electricity from a negative electrode to a positive electrode by attaching electrodes across the heart. Bipolar induction is a method of acquiring an ECG by using two electrodes.
    2. *2Base axis of electromotive force: Since the heart is a 3D structure, its electrical charge continuously changes three-dimensionally. An ECG can be regarded as a recording of the electromotive force generated throughout the heart, and the direction of the electromotive-force vector is called the base axis.

    3. Application to rehabilitation using wearable devices

    Wearable devices are useful not only for early detection and accurate diagnosis of diseases but also after the onset of disease. One example of a wearable device is one for improving rehabilitation quality we have been developing and medically validating with Fujita Health University and Toray Industries, Inc. since 2017 [5]. Taking stroke rehabilitation as an example, the more opportunities for rehabilitation (exercise training) after the occurrence of a stroke, the better the outcome can be expected [6]. However, rehabilitation with therapists is limited, so stroke patients are encouraged to be as active as possible in their daily lives by getting out of bed or doing rehabilitation exercises on their own. However, it is difficult for a patient with a one-side-paralyzed (hemiparetic) body to exert vigorous effort on their own, so they need support to achieve a better recovery. Against this background, we developed a wearable-device that enables precise monitoring of activities of a stroke patient on a 24-hour basis (Fig. 3). For example, a hemiparetic patient wears our device, which he/she can put on and take off by him/herself, and data are automatically collected on a server through a relay such as a smartphone or gateway. The device includes an algorithm for processing data for (i) advanced downsampling that avoids overloading the network of hospital facilities while preserving the characteristics of the patient’s activity and (ii) imputation processing assuming partial data loss. Medical validation of the index obtained by these processes is also underway.

    Fig. 3. Overview of the rehabilitation support system.
    Fig. 3. Overview of the rehabilitation support system.

    This device is used on the basis of three types of outputs. The first is a report for the patient and their family. The recovery process can be clearly conveyed through numerical records of the gradual increase in the time spent in active movement. The illustrations change as the patient progresses, so the patient and his/her therapist can discuss the progress made after each session, which will lead the patient to the next exercise. The second output is the detailed measurement results used in medical-team meetings. Medical-care providers working with a large number of patients cannot keep track of specific individuals. However, on viewing the 24-hour record output from the device, the entire team can share changes in a certain patient’s daily activities. The third output is listed information based on measurement data.
    Regarding medical research, facts are revealed by verifying statistical certainty derived from a large number of measurement results. However, the amount of activity data over a 24-hour period is large; therefore, we designed a system for automated preprocessing so that it can be easily used in medical research. The system also supports the 6-minute walk test, which is widely used in rehabilitation studies. This allows the results of 24-hour activity monitoring to be easily compared with previous studies. We expect this will lead to new insights in rehabilitation research. The system has been commercialized and is being used for medical research. We believe that it will further contribute to better patient recovery through combining it with advanced technologies such as machine learning.

    References

    1. [1]S. Ikawa and K. Kashino, “Neural Audio Captioning Based on Conditional Sequence-to-sequence Model,” Proc. of the Detection and Classification of Acoustic Scenes and Events 2019 (DCASE 2019) Workshop, pp. 99–103, New York, USA, Oct. 2019.
    2. [2]M. Nakano, R. Shibue, K. Kashino, S. Tsukada, and H. Tomoike, “Gaussian Process with Physical Laws for 3D Cardiac Modeling,” Proc. of the 28th European Signal Processing Conference (EUSIPCO 2020), pp. 1452–1456, Amsterdam, Nederland, Jan. 2020.
    3. [3]Y. Tsukada, M. Tokita, H. Murata, Y. Hirasawa, K. Yodogawa, Y. Iwasaki, K. Asai, W. Shimizu, N. Kasai, H. Nakashima, and S. Tsukada, “Validation of Wearable Textile Electrodes for ECG Monitoring,” Heart Vessels, Vol. 34, No. 7, pp. 1203–1211, 2019.
    4. [4]S. Tsukada, “Wearable Textile Electrodes for Long-term Vector ECG Monitoring ‘Tensor Cardiography,’” 14th International Symposium on Medical ICT (ISMICT 2020), Nara, Japan, May 2020.
    5. [5]T. Ogasawara, K. Matsunaga, H. Ito, and M. Mukaino, “Application for Rehabilitation Medicine Using Wearable Textile ‘hitoe’,” NTT Technical Review, Vol. 16, No. 9, pp. 6–12, 2018.
      https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201809fa2.html
    6. [6]M. Yagi, H. Yasunaga, H. Matsui, K. Morita, K. Fushimi, M. Fujimoto, T. Koyama, and J. Fujitani, “Impact of Rehabilitation on Outcomes in Patients with Ischemic Stroke: A Nationwide Retrospective Cohort Study in Japan,” Stroke, Vol. 48, No. 3, pp. 740–746, 2017.

    Related content