Telemedicine supported by Augmented Reality: an interactive guide for untrained people in performing an ECG test

Telemedicine refers to the use of telecommunications and information technologies for the delivery of medical services where is needed [1, 2, 3]. For many applications, the correct usage of medical device at the point of need is essential to provide an appropriate service, but there are some practical situations that may require untrained or inexpert people to interact with medical devices and patients. Some examples are telemedicine services on transportation (e.g. aircrafts [4, 5, 6], boats [7, 8], trains, etc.), application during military actions [9], on islands or remote areas [10, 11], some emergency applications [12, 13], but also home care telemedicine supported by family members [14, 15, 16], elderly care [17, 18], operators training and so on.

In these cases, untrained or improvised (but necessary) actors can, involuntarily, use medical instruments in an inappropriate manner and/or make improper connection between the patient and the medical device seriously invalidating the telemedicine service.

In these scenarios, tailored applications of Augmented Reality (AR) can offer a valid support by guiding non-trained people to a correct usage of medical devices at the point of need. Augmented reality basically consists of a live view of the real-world in which some elements of the scene are “augmented” (enriched, enhanced) by computer-generated information such as graphics, texts and sounds. The application domains for AR are numerous and extend in different fields such as training and support, design, medicine, entertainment and cultural heritage [19, 20, 21].

Recently, there is a growing interest about AR in medicine. The main applications of AR are in the field of surgery, rehabilitation and teaching/training. Interventional medicine, surgery [22, 23], laparoscopy and other procedures (e.g. needle biopsy [24]) can be assisted by integrating preoperative and intraoperative anatomic and functional data improving the visual perception of the surgeon [20, 25, 26, 27]. Obviously, surgery AR applications require very accurate registration and camera calibration [28]. AR and virtual reality have long since found use in rehabilitation and particularly in neurorehabilitation, by guiding and aiding the patient to perform therapy [29, 30]. Teaching and training of students or physicians can take great advantage by AR, which can be further enriched with direct haptic and auditory feedback [31, 32, 33]. Superimposition in real time of images from US, CT or MR scans can also help in learning [20].

There are only very few examples of AR applications in telemedicine, among these there are systems of virtual reality supporting distance teaching of minimally invasive surgery and systems for interactive telemedicine in the operating theatre [34, 35]; some low-cost peripherals to support telehealth, visualization, education and collaborative systems [36]; some applications of distance training for the restoration of motor function, supported by virtual reality [37].

The capability of AR to provide live support to users in operating on instrumentations is also of interest for this study. AR can support and guide workers in operating, actuate, disassembling and maintaining complex devices or systems. As example, it was proposed a mixed reality environment aimed to improve the effectiveness of servicing and repair procedures in mission critical systems, while reducing the time required for the intervention. Also technicians with no previous experience on specific, complex devices were able to perform an assigned maintenance task when supported by the AR application. In maintenance activities there are well-defined sequences of procedures to be done in a relatively static environment. These features allow a defined design space, supporting a wide variety of systems and technologies. It was proved effective in supporting workers for maintenance activities [38].

As feasibility study for new possible telemedicine services, an AR application was built to support untrained users in performing an ECG-test. In particular, the objective of this study was to analytical assess the benefits of exploiting augmented reality principles in order to estimate how untrained users with limited or no knowledge can properly interact with an ECG device and properly placing ECG electrodes on patient’s chest. The proposed AR application was assessed in terms of efficacy and clinical acceptability.

In many clinical activities, 12-lead electrocardiogram is an essential medical investigation. On the other hand, it should carefully carried out. Misplaced ECG electrodes can cause changes in ECG recordings, which could have an impact on clinical decisions [39]. Incorrect electrode cable connections, reversal of electrodes, inadequate placement of the electrodes are common source of error (changes the true ECG morphology) and can conceal or simulate different pathology such as, myocardial ischemia or infarction, arrhythmias, ventricular hypertrophy [40, 41, 42, 43]. It is also worth mentioning that untrained service providers are one of the key barriers to implementation of telemedicine services.

Each of the untrained tester was able to carry out in an appropriate manner the ECG-test with the only support of the AR application. All the testers reported that they had not encountered any particular difficulty in interacting with the AR application and in carrying out all the requested actions. Some of them reported only minor problems in perception of distances in the direction perpendicular to their plane of view. This is due to the use of a single camera, which obviously fails to accurately render the depth of the scene. The AR application was perceived as intuitive and easy to use, the opportunity to interact with vocal commands was particularly appreciated.

The execution time over both tests was on average of 8 minutes (including 3 minutes of ECG recording in the automatic mode) that can be considered acceptable for practical purposes.Bar plots in Figure 5 show the averages and standard deviations of the x, y and z components of errors (computed on all testers) in placement of the precordial electrodes on the mannequin (Figure 5a) and patient’s chest (Figure 5b). Averages can be considered as an indication of the accuracy, while standard deviations as a measure of the precision (repeatability) achieved during the trials.

For the first test, the average errors in electrode positioning on the mannequin resulted less than or equal to 3 mm, while the standard deviation less than 5 mm.

Involvement of the real patient instead of the mannequin led to a slight increase of the errors committed. This can be reasonably due to thorax motion due to breathing or other little movements of the patient. Nevertheless, even in the test of the real patient the mispositioning of the electrodes resulted less than 7 mm on average and reached a maximum of 16 mm on V6, these data support the effectiveness of the AR procedure and the clinical acceptability of the recorded ECG. As a matter of fact, electrode malposition exceeding 25 mm is associated with potentially significant ECG changes [39]. Taking into account this threshold value and observing the results achieved during both tests (Figure 5a-b and Table 1), the average errors in electrode positioning resulted reasonably acceptable and comparable with placement errors usually made by technicians and nurses in an emergency care department [52]. This supports the clinical validity of the acquired ECG waveforms by means of the developed AR system.

It is worth noting that in both tests only for V5 and V6 significant variations of positioning were registered on the z-axis. This is inherent to the specific positions of these two electrodes and their relative positioning with respect to the markers on the thorax. When the user directs his gaze towards V5 or V6, the plane on which lies the marker (x- and y-axes represented in Figure 2c) results significantly angled and then the 3D pose errors increase [48]. This mainly occurs because of the natural curvature of the human thorax. Indeed, many studies have shown that the accuracy of positioning virtual object on the real scene mainly depends on camera distance and viewing angle with respect to the markers and also on other factors (e.g. size of the marker, focal length, field of view, pixel resolution, etc.) [48, 53, 54].

However, in the specific case, the distance and the relative angle between camera and markers are actually limited. Indeed, when placing the electrodes onto a patient or when manually operating on the electrocardiograph, the distance between the camera and markers cannot be greater than the length of operator’s arm, nor the angle of gaze be particularly tilted. Hence, the need for manual interaction with objects marked severely restricts the space in which augmented reality operates.

Due to the perspective of the scene taken by the camera, the greatest pose error of an AR marker lies in the direction that connects the marker with the camera. However, in this specific application (e.g. in placing the electrodes V1 to V4) the error made on the z-axis of Figure 2c (coincident with the most probable line of sight of user) results in practice negligible for the precordial electrodes V1 to V4: the user’s hand must stops on patient’s skin. This, obviously, does not hold for the electrodes V5 and V6 because of their intrinsic positions. Consequently, the placement of these electrodes suffers from greater errors (an higher component on z axis, see Figure 5a-b, and consequently an increase in the positioning error, see Table 1).

Lastly, it is also interesting to note that the errors are consistent with data reported by the developers of the system ARToolKit [48]. Indeed, if we consider that usually the operator is located at approximately 30-40 cm from the patient (and therefore from the markers) with an angle between 0° and 45° the errors predicted by previous studies [48, 53] are contained within +/- 5 mm; while for a greater angle (e.g. 45°-80°) they increase to approximately +/-12 mm. In the extreme case in which the inclination of the user’s gaze line reaches or exceeds 90 degrees with respect to the marker plane, the markers are no longer in sight making impossible to coherently augment the scene. Even if this event did not occur during the trials, it is possible and should be taken into account.

This study has highlighted the possibility of using Augmented Reality to support untrained user while performing an ECG-test. The developed AR application is at the proof-of-concept stage and can certainly be improved. Furthermore, it can be easily generalized to the use of other medical equipment.

AR can have a relevant and positive impact on various telemedicine applications. For example, telemedicine applications involving acquisition of diagnostic signals onboard of a plane or a ship by an untrained crew member or even passengers, as well as in the battlefield by untrained soldiers where it can obtain very beneficial effect. Also other home-care solutions, where a relative may be request to interact with medical device or emergency application (e.g. use of a defibrillator by an inexpert user), can take advantage of such AR applications. More appropriate and reliable usage of medical device can be obtained, but also opens new horizons for novel and/or more advanced telemedicine applications.

However, some issues and question should be fully addressed before moving to clinical applications. While no particular difficulties arose in using AR with medical device (mainly because they are rigid and their 3D geometry known and fixed), problems occur when “augmenting” a generic, alive, human body: large anatomical variations and patient’s motion can be accurately taken into account. It is worth to remember that even if some parts of the human body can be roughly considered rigid, change in proportions should be accounted, e.g., between adults of infants, but also between different body types. Some parts of the body are in continuous motion such as the chest during breathing. However, in the conducted trials, with the patient breathing normally, this phenomenon did not significantly affect the results. These considerations lead to the use of non-rigid constellations of markers (e.g. proportionally scalable in geometric proportions) to take into account different body types, or even to a marker-less recognition of the body (however, this requires the use of sophisticated software and can adversely affect the real-time operation). The final AR application should also consider and properly instruct the operator on how to face with special situation (e.g. in the case of very hairy patients). The flexibility of the XML language offers the possibility to easily enrich or to customize the set of operations and warnings. For example, a simple reminder for the operator to shave the skin or clip small patches of hair for proper sticking of electrodes, whether it is the case, can be included in the final AR application.This study, being essentially a proof of concept, does not provide solutions to these problems. Only a small variation in the markers T-structure has been designed to overcome the problem of excessive user’s slant. An improved version was obtained by adding two extra markers placed adjacently to the left marker (that indicated by the trapezoidal symbol) and the lower marker (that indicated by the C-like symbol), but arranged at 90 degree with respect to the T-structure plane (i.e. laying in anatomical sagittal planes). Figure 6 shows the new marker constellation for the patient’s body. Very preliminary results seem to confirm the expectations.Further developments of this study are oriented towards the realization of this AR application on mobile devices, i.e. tablets or smartphones (see Figure 7). These devices intrinsically incorporate a camera, a relatively large screen, a processor capable of running the software, and other multimedia interfaces such as speakers and microphone. Possibly, the software can be also downloaded, at the very moment, from the Internet. The use of tablets or smartphones, today widely disseminated, could bring further benefits.

Written informed consent was obtained from the patient for the publication of this report and any accompanying images.