The digitalization of modern imaging has led radiologists to become very familiar with computers and their user interfaces (UI). New options for display and command offer expanded possibilities, but the mouse and keyboard remain the most commonly utilized, for usability reasons. In this work, we review and discuss different UI and their possible application in radiology. We consider two-dimensional and three-dimensional imaging displays in the context of interventional radiology, and discuss interest in touchscreens, kinetic sensors, eye detection, and augmented or virtual reality. We show that UI design specifically for radiologists is key for future use and adoption of such new interfaces. Next-generation UI must fulfil professional needs, while considering contextual constraints.
• The mouse and keyboard remain the most utilized user interfaces for radiologists.
• Touchscreen, holographic, kinetic sensors and eye tracking offer new possibilities for interaction.
• 3D and 2D imaging require specific user interfaces.
• Holographic display and augmented reality provide a third dimension to volume imaging.
• Good usability is essential for adoption of new user interfaces by radiologists.
The digitalization of modern imaging facilitates exchange and archiving, and enables the application of advanced image analysis solutions such as computer-aided detection (CAD) for identification of small lesions in several organs. The shift from analog to digital imaging should have led to an increase in efficiency among radiologists by reducing the time for interpretation and image manipulation. This has not been clearly demonstrated, and one limiting factor is represented by what is called the computer user interface. Advances in recent years have enabled the availability of touchscreens and new sensor devices for eye, kinetic or voice commands at low cost, offering expanded possibilities for this human–computer interaction.
Terminology and concepts
The user interface (UI), also known as the human–machine interface, is defined as all the mechanisms (hardware or software) that supply information and commands to a user in order to accomplish a specific task within an interactive system. All machines (e.g. cars, phones, hair dryers) have a UI. The computer has a global UI called an operating system, such as Windows or Mac OS X. A web browser has a specific UI, and a web site itself has a specific UI. In practice, the UI is the link between the machine and the operator. In informatics, the UI includes inputs and outputs. Inputs communicate a user’s needs to the machine, and the most common are the keyboard, mouse, touch interface and voice recognition. New sensor devices for eye and kinetic commands that have recently been developed offer greater sophistication at low cost, enhancing the potential of this human–computer interaction . Outputs communicate the results of a computer’s calculation to the user. The most common UI is a display screen, but sound and haptic feedbacks are sometimes used.
The user experience (UX) is the outcome of the UI. The objective is to provide the best usability to achieve a good UX. This depends on both user specificity and the specific usage context. For these reasons, the UX is evaluated on the basis of both psychology and ergonomics. Ergonomic requirements are defined by International Organization for Standardization (ISO) 9241 regulatory standard, to ensure the operator’s comfort and productivity, preventing stress and accidents (Fig. 1) . Usability is high when efficacy, efficiency and satisfaction are high. Efficacy is the user's ability to complete the planned task. Efficiency is measured by the time to completion. Satisfaction is the user’s subjective evaluation of the comfort of use. The UI needs to be developed and designed specifically for the context of use. This “user-centred design” aims to maximize usability.
Computed radiology and specific needs
With regard to medical imaging, the UI is specifically constrained by human and contextual factors . The interaction occurs between a human observer and a display technology (Fig. 2). With respect to human visual ability, the eye is made of cones and rods. The cones are concentrated in the macula at the centre of the field of vision and provide the best spatial resolution. On the periphery of the visual field, the image becomes blurry. The eye's maximum power of discrimination is 0.21 mm. The physiological focal point of the eye is around 60 cm from the viewer. This creates the technical standards for the practice of radiology. The diagonal of the display should be located at 80% of the distance to the eye, which corresponds to a screen of approximately 50 cm (about 21 in.). For this size, the resolution providing a pitch of 0.21 mm is 1500×2000 pixels . Diagnostics are performed using “macular vision”. The radiologist needs to explore the whole image by moving the eye. This justifies the need for pan and zoom tools to study a particular region of interest.
With regard to contextual constraints, we differentiate diagnostic from interventional radiology. Indeed, with regard to the former, the constraints of UX are more about managing the workflow for maximum productivity. One challenge is the integration of different commands and information in a common UI. Regarding the latter, the limit is clearly in maintaining operator sterility.
In this paper, we review and discuss different UI tools available for radiology over time. We also try to provide an outlook for the future and suggestions for improvements.
Review of the literature
Interfaces for 2D images
Imaging devices have largely provided two-dimensional images: X-ray planar imaging at the beginning of the twentieth century, and then computed tomography (CT) or magnetic resonance (MR) sliced imaging in the 1970s. Initially, the image was printed like a photo, and there was no machine interaction. Today, if we look at any interpretation room around the world, chances are good that we will find the same setup, combining a chair, a desk and one or more computers with a keyboard, a mouse and a screen. The picture of modern radiology can be understood through the evolution of the computer UI, as the image has become digital and is displayed on computers.
In the 1960s, the command-line interface (CLI) was the only way to communicate with computers. The keyboard was the only input, and a strict computer language had to be known to operate the system. In 1966, Douglas Engelbart invented the computer mouse. Together with Xerox, and then Apple's Mac OS X or Microsoft Windows, they participated in developing a graphical user interface (GUI), known as “WIMP” (windows, icons, menus and pointing device) , which vastly improved the user experience. This system made computers accessible to everyone, with minimal skill required. Today, the WIMP UI remains nearly unchanged, and it is the most commonly used UI for personal computers. In 2007, the post-WIMP era exploded with the introduction of the “natural user interface (NUI)” using touchscreens and speech recognition introduced by Apple iOS, followed by Google Android, used mainly for tablet personal computers (PC) and smartphones (Fig. 3) .
Digital radiology and the current workstation were introduced during the WIMP era. The specific UI was designed with a keyboard and a mouse, and this setup has remained in use for approximately 30 years. Resistance to change and the “chasm” or delay in the new technology adoption curve explains the UI stagnation globally in the field of radiology .
However, is there really a better alternative to a mouse and a keyboard? Weiss et al. tried to answer this question, and compared five different setups of IU devices for six different PACS users during a 2-week period . The study did not include post-WIMP UI. The authors concluded that no one device was able to replace the pairing of the keyboard and mouse. The study also revealed that the use of both hands was thought to be a good combination. However, the evaluation focused on image manipulation and did not consider single-handed control needed for microphone use in reporting.
The authors proposed an interesting marker of efficacy for radiologic IU as the highest ratio of “eyes-to-image” versus “eyes-to-interface” device time.
Some solutions for improving the WIMP-UI have been tested. They combine an “eye tracking” technology with the pointing technique . The objective is to eliminate a large portion of cursor movement by warping the cursor to the eye gaze area . Manual pointing is still used for fine image manipulation and selection. Manual and gaze input cascaded (MAGIC) pointing can be adapted to computer operating systems using a single device (Fig. 4).
Regarding post-WIMP UI, and especially touchscreens, there is abundant literature, most of which deals with emergency setup involving non-radiologist readers as well . Indeed, the tablet PC offers greater portability and teleradiology possibilities. Tewes et al. showed no diagnostic difference between the use of high-resolution tablets and PACS reading for interpreting emergency CT scans . However, touchscreen adoption is not evident at the moment, even if full high-definition screens fulfil quality assurance guidelines. Users have found the windowing function less efficient than the mouse, and have also noted screen degradation due to iterative manipulations. Technically, portable tablet size and hardware specifications are not powerful enough for image post-processing. However, cloud computing and streaming can provide processor power similar to a stand-alone workstation (Fig. 5). Their portability makes them more adaptable for teleradiology and non-radiology departments. One possible solution discussed recently is a hybrid type of professional tablet PC for imaging professionals . The interface is designed to enable direct interaction on the screen using a stylet and another wheel device. Microsoft and Dell are currently proposing design solutions for specific use with photo and painting software. These desktops could be used for radiology workstations with a few UX-specific design modifications (Fig. 6).
Interventional radiology is a specific process with specific needs, the most important of which is maintaining the sterility of the operating site while manipulating the images. Ideally, the operation has to be autonomic for at least basic features such as selecting series, reformatting, slicing, and pan and zoom manipulation. Some have proposed taking a mouse or trackpad inside the sterile protected area, or even using a tablet PC to visualize images. However, the most efficient setup in these conditions is touchless interaction , which will minimize the risk of contamination.
Iannessi et al. developed and tested a touchless UI for interventional radiology . Unlike previous efforts, the authors worked on redesigning a specific IU adapted to the kinetic recognition sensor without a pointer (Fig. 7). The user experience has been clearly improved with respect to simple control of the mouse pointer . This is also a good example of environment constraints and user-centred design. Indeed, the amplitude of the arm movements had to be reduced to a minimum, considering the high risk of contamination inside a narrow operating room.
Interfaces for 3D images
Three-dimensional imaging volumes began to be routinely produced in the 1990s. They were originally acquired on MRI or reconstructed from multi-slice helical CT acquisitions, and volume acquisition later became available from rotational angiography or ultrasound as well . With the exception of basic X-ray study, medical imaging examination rarely does not include 3D images.
Volume acquisition can now be printed in three dimensions, similar to the case with 2D medical films . Obviously, this option can be considered only for selected cases such as preoperative planning, prosthesis or education. It is expensive and absolutely not conducive to productive workflow .
Some authors dispute the added value of 3D representations. Indeed, radiology explores the inside of organs, and except in rare situations, 2D slices give more information than a 3D representation of the surfaces.
However, mental transformation from 2D to 3D can be difficult. For example, when scoliosis needs to be understood and measured, 3D UI appears to be more efficient . Some orthopedic visualization, cardiovascular diagnoses and virtual colonoscopic evaluations are also improved by 3D UI [21,22,23]. For the same reasons, 3D volume representations are appreciated by surgeons and interventional radiologists, as they help to guide complex surgery or endovascular procedures [24,25,26]. Preoperative images improve surgical success . Moreover, advanced volume rendering provides more realistic representations, transforming medical images into a powerful communication tool with patients (Fig. 8) [28, 29].
However, both use and usability of such acquisition volumes remain poor. There are many reasons for the non-use of 3D images, including the absence of full automation of the required post-treatment. Also, exploitation of 3D volume is hindered by the lack of adapted display and command UI . By displaying 3D images on 2D screens, we lose part of the added information provided by the 3D volume .
With regard to inputs, touchless interfaces have been demonstrated as one interesting option. A kinetic sensor placed in front of a screen senses 3D directional movements in order to manipulate the virtual object with almost natural gestures [14, 32].
For displays, some authors have explored the use of holographic imaging in radiology, especially in the field of orthopedic diagnostic imaging [23, 33, 34]. In 2015, the first holographic medical display received FDA approval. This includes 3D glasses and a stylet for manipulation (Fig. 9).
Another possibility for displaying 3D volume is the use of augmented reality. The principle is to show the images with a real-time adjustment to observe cephalogyric motion. This can be done using a head-mounted device such as Google Glass, a handheld device such as a smartphone, or a fixed device. Nakata et al. studied the latest developments in 3D medical imaging manipulation. The authors demonstrated improved efficiency of such UI compared to a two-button mouse interaction . Augmented reality and 3D images have also been used in surgical practice for image navigation [36, 37]. Conventional registration requires a specific acquisition, and the process is time-consuming . Sugimoto et al. proposed a marker-less surface registration which may improve the user experience and encourage the use of 3D medical images (Fig. 10) . Recent promotion of the HoloLens (Microsoft, Redmond, WA, USA), a headset mixed-reality device including an efficient UI controlled by voice, eye and gesture, may help to accelerate radiological applications of augmented reality, especially for surgery (Fig. 10) .
Another UI for displaying 3D medical images is virtual reality. In this case, it is a completely immersive experience. The operator wears a device and the environment is artificially created around him. Some authors have proposed including a 3D imaging volume inside the environment to give the user the opportunity to interact with it (Fig. 11).
Outlook for the future
We believe that UX and UI specifically designed for radiology is the key for future use and adoption of new computer interface devices. A recent survey including 336 radiologists revealed that almost one-third of the radiologists were dissatisfied with their computing workflow and setup . In addition to innovative hardware devices, efforts should focus on an efficient software interface. We are mainly concerned with PACS software in this discussion. Indeed, a powerful specific UI has to meet the radiologist's needs, and these needs are high (Fig. 12).
Regarding image manipulation, Digital Imaging and Communications in Medicine (DICOM) viewers are typically built with two blocks: the browser for study images and series selection, and the image viewer with manipulation tools. The key elements of the PACS UI architecture are hanging protocol and icons, image manipulation, computer-aided diagnosis and visualization features . The goal of a hanging protocol is to present specific types of studies in a consistent manner and to reduce the number of manual image ordering adjustments performed by the radiologist . In fact, automated scenarios should be promoted in order to present the maximum information by default at initial presentation . In addition, the hanging protocols and icons should be user-friendly, intuitive and customizable. Visualization features can be incorporated into a stand-alone facility and integrated with the workstation. The software requires expert functionality that entails more than just simple scrolling, magnification and windowing.
For diagnostic imaging, in addition to the UI for image manipulation, radiologists need a UI for workflow management that includes medical records and worklists . As teleradiology evolves, the concept of “SuperPACS” will probably drive the next UI to an integrated imaging viewer . Indeed, medical information is tedious and labor-intensive when it is not integrated on the same interface and/or computer. The interface should aggregate all needed information for the reporting task. It is the same for the reporting and the scheduling systems. Enhancing the performance of automated voice recognition should enable real-time dictation, where we can fully interact with the images .
As explained above, 3D manipulation and display must be promoted for the added value they provide. Even though the technology may be ready for robust utilization, there is an intractable delay in radiologist adoption . Radiologists, like any customer, are resistant to change, and the design of radiology-specific UI will hasten the revolution [14, 30].
Since the digitalization of radiology, UI for radiologists have followed the evolution of common interfaces in computer science. The mouse and the keyboard remain the most widely adopted UI.
We highlight the importance of designing a specific UI dedicated to the radiologist in terms of both the hardware and software in order to make the experience more efficient, especially with the evolution of teleradiology and the need for increased productivity. Touch technology (touch or stylus) is promising, but requires exact customization for good radiologist usability.
Algorithmic advances will facilitate the take-up of 3D imaging through automated detailed and informative volume rendering. However, specific UI will be needed for 3D image display. Augmented and virtual reality are promising candidates to fill this gap. With regard to image manipulation, contactless interfaces appear to be more suitable for interventional radiology units that already have a good level of usability.
Berman S, Stern H (2012) Sensors for gesture recognition systems. IEEE Trans Syst Man Cybern Part C Appl Rev 42(3):277–290
Kenn H, Bürgy C (2014) “Are we crossing the chasm in wearable AR?”: 3rd workshop on wearable Systems for Industrial Augmented Reality Applications. Paper presented at the Proceedings of the 2014 ACM International Symposium on Wearable Computers: Adjunct Program, Seattle
Weiss DL, Siddiqui KM, Scopelliti J (2006) Radiologist assessment of PACS user interface devices. J Am Coll Radiol 3(4):265–273
Tewes S, Rodt T, Marquardt S, Evangelidou E, Wacker FK, von Falck C (2013) Evaluation of the use of a tablet computer with a high-resolution display for interpreting emergency CT scans. Rofo 185(11):1063–1069
Jo J, L'Yi S, Lee B, Seo J (2017) TouchPivot: blending WIMP & post-WIMP interfaces for data exploration on tablet devices. Paper presented at the Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver
Reinschluessel AV, Teuber J, Herrlich M et al (2017) Virtual reality for user-centered design and evaluation of touch-free interaction techniques for navigating medical images in the operating room. Paper presented at the Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Denver
Drijkoningen T, Knoter R, Coerkamp EG, Koning AH, Rhemrev SJ, Beeres FJ (2016) Inter-observer agreement between 2-dimensional CT versus 3-dimensional I-space model in the diagnosis of occult scaphoid fractures. Arch Bone Joint Surg 4(4):343–347
Schnetzke M, Fuchs J, Vetter SY et al (2016) Intraoperative 3D imaging in the treatment of elbow fractures--a retrospective analysis of indications, intraoperative revision rates, and implications in 36 cases. BMC Med Imaging 16:24
Sundaramoorthy G, Higgins WE, Hoford J, Hoffman EA (1992) Graphical user interface system for automatic 3-D medical image analysis. Paper presented at the  Proceedings Fifth Annual IEEE Symposium on Computer-Based Medical Systems
Venson JE, Albiero Berni JC, Edmilson da Silva Maia C, Marques da Silva AM, Cordeiro d'Ornellas M, Maciel A (2017) A case-based study with radiologists performing diagnosis tasks in virtual reality. Stud Health Technol Inform 245:244–248
Nakata N, Suzuki N, Hattori A, Hirai N, Miyamoto Y, Fukuda K (2012) Informatics in radiology: intuitive user interface for 3D image manipulation using augmented reality and a smartphone as a remote control. Radiographics 32(4):E169–E174
Elmi-Terander A, Skulason H, Soderman M et al (2016) Surgical navigation technology based on augmented reality and integrated 3D intraoperative imaging: a spine cadaveric feasibility and accuracy study. Spine (Phila Pa 1976) 41(21):E1303–E1311
Sugimoto M, Yasuda H, Koda K et al (2010) Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci 17(5):629–636
Pratt P, Ives M, Lawton G et al (2018) Through the HoloLens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. Eur Radiol Exp 2(1):2
Iannessi Antoine is co-founder of Therapixel SA, therapixel.com. Therapixel is a medical imaging company for custom user interface dedicated to surgeons.
Clatz Olivier is CEO and co-founder of Therapixel SA, therapixel.com. Therapixel is a medical imaging company for custom user interface dedicated to surgeons.
Maki Sugimoto is COO and co-founder of Holoeyes Inc., Holoeyes.jp. Holoeyes is a medical imaging company specialized in virtual reality and 3D imaging user interface.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Iannessi, A., Marcy, PY., Clatz, O. et al. A review of existing and potential computer user interfaces for modern radiology.
Insights Imaging9, 599–609 (2018). https://doi.org/10.1007/s13244-018-0620-7