While the previous survey on AI [1] was based on the expectations of the ESR members regarding the impact of AI on radiology, the present survey intended to obtain an overview of current practical clinical experience with AI-based algorithms. Although the respondents with practical clinical experience in this survey represent only 1% of the ESR membership, their proportion among all respondents varied greatly among countries. The geographical distribution of the 276 radiologists who shared their experience with such tools in clinical practice shows that the majority was affiliated to institutions in Western and Central Europe or in Scandinavia. Half of all respondents with practical clinical experience with AI tools was affiliated to academic institutions, whereas the other half practiced radiology in regional hospitals or in private services. Since it is likely that the respondents in this survey were radiologists with a special interest in AI-based algorithms, it cannot be assumed that this survey reflects the true proportion of radiologists in the European region with practical clinical experience with AI-based tools.
Most of the respondents of this brief survey did not encounter major problems related to the integration the AI-based software tools into the local IT systems; less than 18% did have such issues. However, it must be taken into consideration that radiologists are not always directly involved in the technical process of software integration; this fact may perhaps also explain the relatively high number of respondents who did not reply to this specific question.
Today, AI-based tools for diagnostic purposes may address a large range of use case scenarios. Although this was reflected by the free text answers of the respondents of the present study, the present survey distinguished mainly between algorithms for diagnostic purposes and those for the prioritisation of workflow whereas a detailed analysis of all the different individual use case scenarios was beyond the scope of this survey. Since diagnostic tools are usually quite specific and related to organs and pathologies, even radiologists working in the same institution but in different subspecialties may have different experiences with different algorithms related to their respective fields.
In a recent survey among the members of the American College of Radiology (ACR) the most common applications for AI were intracranial haemorrhage, pulmonary embolism, and mammographic abnormalities, although it was stated that in the case of mammography, confusion must be avoided between AI-based tools and the more traditional software for computer aided diagnosis (CAD) [4]. It was estimated that AI was used by approximately 30% of radiologists, but concerns over inconsistent performance and a potential decrease in productivity were considered to be barriers limiting the use of such tools. Over 90% of respondents would not trust these tools for autonomous use. It was concluded that despite initial predictions the impact of AI on clinical practice was modest [4].
Quality assurance of algorithms that are based on machine–learning may be quite time-consuming and requires considerable resources. Effective supervision of the sensitivity and specificity of a device that adapts itself over time may be done by recording differences between the diagnosis of the radiologist and the algorithm but ideally combines regular monitoring by comparison against a final diagnosis as a gold standard—a so-called “ground truth”. Despite the enthusiasm about AI-based tools there are some barriers to be addressed when implementing this new technology in clinical practice. These include the large amount of annotated image data required for supervised learning as well as validation and quality assurance for each use case scenario of these algorithms, and, last but not least, regulatory aspects including certification [5, 6]. A recent overview of commercially available CE-marked AI products for radiological use found that scientific evidence of potential efficacy of level 3 or higher was documented in only 18 of 100 products from 54 vendors and that for most of these products evidence of clinical impact was lacking [3].
Nonetheless, as a general impression, most of the respondents of this ESR survey who used AI-based algorithms in their clinical practice considered their diagnostic findings to be reliable for the spectrum of scenarios for which they were used. It is noteworthy that 44% of the respondents recorded discrepancies occurring between the radiologists’ and the algorithms’ findings and that approximately one-third indicated that they generated ROC curves based on the radiological report or the clinical record in order to calculate the performance of algorithms in clinical practice. Details regarding the methodologies, e.g. the degree of automation used for establishing these data, were neither asked from nor provided by the respondents. However, since over one-half of the respondents worked in academic institutions, it is possible that some of the algorithms were not only evaluated in the context of clinical routine but also in the context of scientific research studies, thus explaining the relatively high level of quality supervision of the algorithms. Only a small minority of radiologists participating in this survey informed the patients about the use of AI for the diagnosis and about one-third mentioned it in their reports. This may be understandable as long as the radiologist and not the algorithm makes the final diagnosis.
However, the important question remains to what extent AI-powered tools can reduce radiologists’ workload. In the previous ESR survey conducted in 2018, 51% of respondents expected that the use of AI tools would lead to a reduced reporting workload [1]. The actual contributions of AI to the workload of diagnostic radiologists were assessed in a recent analysis based on large number of published studies. It was concluded that although there was often added value to patient care, workload was decreased in only 4% but increased in 48% and remained unchanged in 46% institutions [2]. The results of the present survey are somewhat more optimistic since almost 23% of respondents experienced a reduction of their workload when using algorithms for diagnostic assistance in clinical practice, whereas almost 70% did not. Observations with algorithms aiming at workflow prioritisation were comparable. In view of the wide range of use case scenarios for which AI- based tools can be applied, additional studies are needed in order to determine for which specific tasks and questions in which subspecialties AI-based algorithms could be helpful to reduce radiologists’ workload. Typically, this could be the case in those scenarios that address the detection of relatively simple diagnostic findings and a high volume of cases.
The previous ESR survey from 2018 included 675 participants of which 20% were already using AI-powered tools and 30% planned to do so [1]. The present ESR survey included 690 participants of which 276 (40%) had experience with such tools in clinical practice. However, when all the participants of the present survey were asked whether they intended to acquire a certified AI-based algorithm, only a minority (13.3%) answered yes, whereas the majority either answered no (52.6%) or did not answer the question (34.1%). Reasons given for the negative answers included doubts about the added value or the advertised performance or concerns regarding added workload. We must consider, however, that the answers to this particular question included not only the opinions of the respondents who had experience with practical clinical use but also of those who used these algorithms rather in the context of scientific projects including non-commercial, home-grown AI-based tools.
The results of the present ESR survey are difficult to compare with the recent ACR survey [4] not only because the questions were not identical, but also because of the existing diversity among European countries. Nonetheless, both surveys conclude that, compared with initial predictions and expectations, the overall impact of AI-based algorithms on current radiological practice is modest.
Several limitations of this brief survey need to be mentioned. Firstly, the survey data cannot reflect the true proportion of European radiologists using AI. Secondly, the answers to several questions can only provide a general overview, although some of the issues addressed by this survey would deserve a more detailed analysis. This is true, for example, regarding the differentiation of use case scenarios as well as the methodologies used for the verification of their results. Thirdly, the observations are based on the situation in 2022, and results and opinions may change rapidly in this evolving field.
In summary, this survey suggests that, compared with initial expectations, the use of AI- powered algorithms in practical clinical radiology today is limited, most importantly because the impact of these tools on the reduction of radiologists’ workload remains unproven. As more experience with AI-powered algorithms for specific scenarios is being gained and some of the barriers to their use may become mitigated in the future, a follow-up to this initial survey could provide further insights into the usefulness of these tools.
