A framework to integrate artificial intelligence training into radiology residency programs: preparing the future radiologist

Abstract

Objectives

To present a framework to develop and implement a fast-track artificial intelligence (AI) curriculum into an existing radiology residency program, with the potential to prepare a new generation of AI conscious radiologists.

Methods

The AI-curriculum framework comprises five sequential steps: (1) forming a team of AI experts, (2) assessing the residents’ knowledge level and needs, (3) defining learning objectives, (4) matching these objectives with effective teaching strategies, and finally (5) implementing and evaluating the pilot. Following these steps, a multidisciplinary team of AI engineers, radiologists, and radiology residents designed a 3-day program, including didactic lectures, hands-on laboratory sessions, and group discussions with experts to enhance AI understanding. Pre- and post-curriculum surveys were conducted to assess participants’ expectations and progress and were analyzed using a Wilcoxon rank-sum test.

Results

There was 100% response rate to the pre- and post-curriculum survey (17 and 12 respondents, respectively). Participants’ confidence in their knowledge and understanding of AI in radiology significantly increased after completing the program (pre-curriculum means 3.25 ± 1.48 (SD), post-curriculum means 6.5 ± 0.90 (SD), p-value = 0.002). A total of 75% confirmed that the course addressed topics that were applicable to their work in radiology. Lectures on the fundamentals of AI and group discussions with experts were deemed most useful.

Conclusion

Designing an AI curriculum for radiology residents and implementing it into a radiology residency program is feasible using the framework presented. The 3-day AI curriculum effectively increased participants’ perception of knowledge and skills about AI in radiology and can serve as a starting point for further customization.

Critical relevance statement

The framework provides guidance for developing and implementing an AI curriculum in radiology residency programs, educating residents on the application of AI in radiology and ultimately contributing to future high-quality, safe, and effective patient care.

Key points

• AI education is necessary to prepare a new generation of AI-conscious radiologists.

• The AI curriculum increased participants’ perception of AI knowledge and skills in radiology.

• This five-step framework can assist integrating AI education into radiology residency programs.

Graphical Abstract

Introduction

The impact of artificial intelligence (AI) on healthcare is immense, with numerous applications already transforming clinical practice [1,2,3]. In radiology departments, the use of AI can improve administrative workflow, image acquisition, interpretation, and disease detection, transforming the role of radiologists in the process [4]. For example, AI-based algorithms can optimize radiology department workflows by prioritizing chest x-rays, thereby reducing report turnaround times for critical findings [5]. They also maximize image acquisition, such as by reducing noise and artifacts in MRI scans [6], and improve early detection of breast cancer in digital mammography [7]. A demonstration of a future workflow, illustrating how AI can impact multiple steps along the imaging life cycle, integrates various examples of how AI may assist [8]. Traditionally, radiologists are expected to make management decisions and investments in medical imaging equipment, picture archiving and communication systems, and other radiology information systems. However, the emergence of AI-powered diagnostic decision-making and workflow efficiency tools presents a new challenge that requires radiologists to have a basic understanding of AI systems [9]. Educating radiologists on the capabilities and limitations of AI, empowering them to assess AI systems, is crucial to ensure they can make optimal use of AI solutions to improve and sustain high-quality, safe, and effective patient care [10, 11].

While the importance of AI is inevitable, previous studies on developing an AI curriculum for existing radiology residency programs are scarce [12, 13]. Moreover, there is a general lack of formal AI training in radiology residency programs, and most academic institutions do not yet offer such training [14]. Therefore, we aim to present a framework to develop and implement a fast-track AI curriculum into an existing radiology residency program, with the goal of preparing a new generation of AI conscious radiologists.

Methods

The Institutional Review Board of the University Medical Center Groningen (UMCG) approved the study, and informed consent was provided by the participants. Procedures followed were in accordance with the ethical standards and human regulations. This study is part of a subsidized project B3CARE (B3CARE; www.b3care.nl).

We conducted a study on how to design, implement, and evaluate an AI curriculum, specifically tailored for educating radiology residents and emphasizing the assessment of its feasibility and endorsement of the radiology residents. In accordance, we developed a framework consisting of five sequential steps (Fig. 1).

• 1. Compilation of a multidisciplinary team of AI experts

• 2. Assessment of the knowledge levels and needs of the residents

• 3. Definition of learning goals

• 4. Matching these learning goals with effective methods

• 5. Execute and evaluate the pilot

Fig. 1

A five-step framework to develop and implement an AI curriculum into an existing radiology residency program. AI, artificial intelligence

Each step is described in more detail below. Following this five-step framework, a 3-day AI curriculum was designed by a multidisciplinary team of AI engineers, radiologists, and a radiology resident.

Step 1: Compile a team of AI experts

A multidisciplinary team was formed to design and implement an AI curriculum in the radiology residency program. The team was composed of AI engineers with educational as well as scientific expertise from the Robotics and Mechatronics Group at the University of Twente (C.O.T., E.I.S.H.), data science experts as part of the Data Science Center in Health (DASH, UMCG, P.M.A.v.O., R.V.), radiologists with ample experience in the field of AI and current experience with the latest scientific advancements and practical applications from the Department of Radiology of the UMCG (D.Y., R.V., T.C.K.), radiology residency program directors (W.N., M.J.L.), and a radiology resident with special interest in AI in radiology (M.J.v.K.).

Step 2: Ask your residents

A pre-curriculum survey (Supplementary material 1) was completed anonymously by all 17 residents including junior and senior residents (experience 1–5 years) working at the Radiology Department of the University Medical Center Groningen at April 2021. The pre-curriculum survey comprised a set of eight questions to inquire about residents’ current knowledge of AI, what specific topics they would like to learn about, and which learning methods they preferred. This information was then used in the designing and evaluation process of the AI curriculum.

Step 3: Define audience and top learning goals

The optimal group size for the course was determined to be 12 participants based on various factors including residents’ preferences, practical considerations, and in order to facilitate effective interaction and group discussions. However, due to limited availability and a total of 17 radiology residents, priority was given to senior residents (3–5 years’ experience) over junior residents (1–2 years’ experience) by the radiology residency program directors and the AI expert team. Consequently, 11 senior residents with 3–5 years of experience (all senior residents working at the radiology department at October 2022) were included for participation in the course. Participation for the course was mandatory for the senior residents, and no prior skills or experience in computer science or programming were required. With one remaining vacancy, an attending radiologist with 10 years of working experience expressed interest in AI education and occupied the final seat. All participants were expected to have basic knowledge, understanding, and experience in medical imaging techniques and their application in clinical practice.

Learning goals were distilled by the AI expert team based on institutional goals and the results of the pre-curriculum survey among all residents. The top 3 learning goals were to understand the following: fundamental architecture structure of AI systems, how to exploit AI for clinical research, and how to use AI in clinical practice. These learning goals were defined in short contents: learning the fundamentals of AI, getting experience with building an algorithm, and understanding implication of AI in clinical research and clinical practice. An additional learning goal was to get familiar with billing, legal, and ethical aspects of AI use. See Table 1 for a more detailed description of the learning goals and Table 2 for a short description of the content. The learning goals of the present course were developed based on an established long-term curriculum designed for technical medicine students, which was a large-scale project supported by B3CARE. Our team of AI experts previously developed a relevant full-term course for technical medical students, providing a foundational structure that was condensed and modified to serve radiology residents in training.

Table 1 Learning goals distilled based on the institutional goals and the results of the pre-curriculum survey

Table 2 Short description of the contents of the AI curriculum divided over 3 days

Step 4: Educational format

The learning goals of the curriculum were matched with different learning methods based on the pre-curriculum survey and the input of the AI expert team. The course format included didactic lectures about the basics of the underlying AI algorithms, hands-on laboratory sessions with building an algorithm on a clinical research data adopted from Orange (https://orangedatamining.com) (e.g., defining training and test sets, adjusting hyperparameters, consequences of overfitting, interpreting results), group discussions (billing, legal and ethical issues), Q/A sessions with an AI expert, and a presentation from a representative of a commercially available vendor. We asked the vendor to focus on providing insights on the integration of commercially available AI systems in institutional workflows and the associated implementation challenges, rather than marketing their software. Vendor’s presentation tools were reviewed by the instructor (C. O. T.) prior to the presentation, and the vendor clearly disclosed his potential conflict of interest.

The length of the curriculum was determined based on residents’ preferences (pre-curriculum survey results), feasibility of achieving learning goals, and practical feasibility of incorporation into the radiology residency program: the total length of the course was 3 days (8 h a day during regular working hours). Of these 24 h, roughly 9 h were dedicated towards the first learning goal, 9 to the second, and 6 h to the third.

Step 5: Start and evaluate the pilot

A post-curriculum survey (Supplementary material 2) was completed by all 12 participants of the course (11 senior radiology residents and 1 experienced radiologist). The post-curriculum survey comprised a set of 22 questions aimed at evaluating the extent to which the participants’ expectations and learning goals were achieved, the influence of the course on their knowledge and skills pertaining to AI, and the potential impact of the course on their future career and professional work. Change in perceived confidence levels of participants before and after the course was rated retrospectively during the post-curriculum survey. Suggestions resulting from the post-curriculum survey and the insights of the AI expert team were collected for future revision of the curriculum. The development, running and evaluation of the AI curriculum utilizing the 5-step framework required a duration of approximately 20 months, with an estimated time commitment of 2 h per week in total.

Statistical analysis

All results were analyzed descriptively. A Wilcoxon rank-sum test was conducted to evaluate the self-reported levels of confidence of the participants on knowledge and understanding of AI-based approaches in radiology before and after the curriculum, using R language for statistical computing.

Results

A total of 17/17 residents completed the pre-curriculum survey (100% response rate). The survey indicated that most residents had little to no knowledge or experience with AI (Table 3). Most residents believed it was necessary to implement AI education in the radiology residency program (Fig. 2). Topics that residents wanted to be included in the curriculum were as follows: how to implement AI in the radiologist’s workflow (15 residents/88.2%), understanding machine learning and deep learning (7 residents/41.2%), and how AI can be used in clinical practice (12 residents/70.6%) and for research purposes (10 residents/58.8%) (Table 3). Residents suggested various learning methods: integrating education in AI into the existing clinical rotations (10 residents/58.8%), a separate learning course about AI (9 residents/52.9%), or an online learning module about AI (8 residents/47.1%) (Table 3).

Table 3 Results from the pre-curriculum survey consisting of eight questions

Fig. 2

Highlighted results of the pre-curriculum and post-curriculum survey. *0 values are not shown on the pie charts. a Question from the pre-curriculum survey including 17 responses. b–d Questions from the post-curriculum survey including 12 responses per question

A total of 12/12 participants completed the post-curriculum survey (100% response rate). After completion, participants’ rated confidence levels on knowledge and understanding of AI-based approaches in radiology were significantly increased (retrospectively rated pre-curriculum mean of confidence level = 3.25 (SD = 1.48) and post-curriculum mean of confidence level = 6.5 (SD = 0.90), Z-value =  − 3.06, p-value = 0.002) (Fig. 3). The group size (12 participants) and time investment (3 days) for the course were deemed efficient (Table 4). Moreover, 9 participants (75%) confirmed that the course covered topics that were applicable to their work as radiologists (Fig. 4). Lectures on the fundamentals of AI and group discussions on AI were rated most useful (Fig. 4). Only 2 participants (16.7%) found the hands-on laboratory sessions (building an algorithm on pre-defined clinical research data) useful (Table 4). Nine participants (75%) noted that they would highly recommend the course to their colleagues and future radiologists, and 9 participants (75%) were positive about including the course in the regular radiology residency program (Fig. 4). However, the course was found to cover certain areas insufficiently, including hospital management’ views on legal and insurance issues related to AI, input from radiologists that integrated AI in their workflow (for example: how they select an appropriate AI software), and cost-effectiveness strategies of AI-powered tools in healthcare in Europe (Table 4).

Fig. 3

Level of confidence of the participants about their knowledge and understanding of AI-based approaches in radiology. AI, artificial intelligence. A total of 12 responses on confidence level of participants on a scale from 0 to 10 (0 = not confident at all, 10 = very confident) that they rated during the post-curriculum survey. A Wilcoxon rank-sum test resulted in a pre-curriculum mean of 3.25 (SD = 1.48) and post-curriculum mean of 6.5 (SD = 0.90), Z-value =  − 3.06 and p-value = 0.002

Table 4 Results from the post-curriculum survey consisting of 22 questions

Fig. 4

Results on which part of the AI curriculum was evaluated most useful. AI, artificial intelligence. Question from the post-curriculum survey including 12 responses; multiple options were possible

Discussion

With this study, we demonstrated the feasibility of implementing a framework for developing and delivering a 3-day AI curriculum in a radiology residency program. The framework consisted of five key steps (composing an AI expert team, assessing knowledge and needs of residents, defining audience and learning goals, determining educational format, and staring and evaluating a pilot) that other radiology departments may find useful to prepare a similar fast-track AI curriculum. The post-curriculum survey indicated that the curriculum improved participants’ self-reported confidence in how to handle AI-based approaches in radiology practice. Participants also recommended that the curriculum should be included as a standard component in the existing radiology residency program. Lectures on the fundamentals of AI and group discussions with experts were deemed most useful, while hands-on laboratory sessions with building an algorithm were rated as less useful. Furthermore, the participants perceived a course length of 3 days as sufficient. These results serve as a marker of the curriculum’s effectiveness, underlining its practical utility within radiology residency programs.

Two previous studies [12, 13] reported on the development and implementation of AI curriculum in radiology residency programs. Shiang et al. [12] evaluated residents’ real-time experience and perception of using AI-based decision support system applications. The residents found the approach desirable and reported positive experiences. However, a major limitation of their study was the lack of generalizability because they focused on the use of one commercially available platform rather than the concepts underlying AI-based systems and their practical application. Thus, this approach may not provide a thorough overview of the potential role of AI in radiology practice, and may not be applicable or appropriate for other institutions that use different platforms or vendors. Moreover, this study did not include a detailed description of learning goals on more foundational concepts of machine learning or deep learning. Nonetheless, we do agree that their design of training will prepare the participants for future AI advancements, and a real-time experience could be a part of a more comprehensive curriculum.

Hu et al. [13] developed a 3-week AI workshop. Their results on their post-workshop surveys showed increasing confidence in understanding AI concepts by the participating residents, similar to ours. They presented a comprehensive overview of methodology with learning objectives that make it more generalizable to other institutions. Nonetheless, a major limitation of this study is the length of the course (3 weeks), which limits the feasibility of embedding this course to radiology resident training programs. In our study, we designed a curriculum of 3 days, which the participants perceived as sufficient and the radiology residency program directors found feasible for easy incorporation into the existing radiology residency schedule. However, Hu et al. delved extensively into the technical aspects of AI, whereas we focused on broader aspects such as implementation, legislation, and ethics.

Recently, Salastekar et al. [15] highlighted the need for education in AI based on a survey among 759 residents in the USA. They found that a majority of radiology residents believed that education in AI should be included in the radiology training program. They found that hands-on laboratory sessions and didactic lectures were rated as the most effective learning methods but, in our study, especially the hands-on laboratory sessions, were not evaluated as most valuable. This may be because we used an open-source, research data set (Orange, https://orangedatamining.com). Some residents found this approach to be too technical (writing and adjusting algorithms), and preferred an easier and more visual method, which might be closer to the work of a radiologist. This is in line with the findings in other previous studies were residents felt AI to be important and worth learning, but most were not very interested in learning to program [16] or simplified, and self-contained coding environments could also serve as fertile opportunity for self-exploration [17]. However, preferences may not be the sole reasoning behind avoiding the technical intricacies. The team of AI experts still find the relative in-depth technical approach to be necessary. In future courses, balancing these perspectives, a less technical and a more visual approach, could be considered based on residents’ needs and expert recommendations.

Regarding the content, in our post-curriculum survey, participants felt that some topics were missing from the course. For example, input from radiologists currently working with AI, hospital management’ views on legal and insurance issues related to AI, and cost-effectiveness strategies of AI-powered tools in health care in Europe were not covered in our curriculum. Similar findings were reported by Huisman et al. [18] who surveyed 1041 radiologists on AI in radiology and concluded that AI education should include issues related to data management, ethics, and legislation. We addressed data management and implementation challenges by inviting a representative from a commercial vendor to provide insight during group discussions. This part of the curriculum can be further improved by including more (or specific) commercial AI software (Van Leeuwen et al. provided a full list of 100 commercially available AI software for radiology [19]), although this would add to the overhead by increasing the need for external resources and time commitment.

Ethical issues were also addressed during our sessions in the form of group discussions. However, we did not include an ethics expert with experience in clinical use of AI, and did not cover legislative aspects, hospital management views, or cost-effectiveness strategies in Europe. As these are emerging topics, it would be valuable to include group discussions with hospital management, policy makers, and insurance companies to discuss future challenges.

In our curriculum, we did not use any existing learning or vendor-based platforms. While some learning platforms are open source and can facilitate learning by providing access to the latest AI tools and resources, others come with a price tag. Depending on the budget and learning goals, the use of existing learning platforms could be useful while also allowing each institution to customize learning goals according to their own specific needs and practice.

Our study did not issue certificates upon course completion due to our non-accreditation status and the current lack of standardized certification for AI in healthcare. Recognizing the involvement of global entities in standardizing AI in healthcare, we suggest considering accreditation and skills evaluation as a next step for AI courses in the medical field. A recent study identified six competencies for physicians using AI tools in healthcare: foundational knowledge, critical appraisal, medical decision-making, technical use, patient communication, and awareness of unintended consequences [20]. A framework like this one could be used in shaping assessments for AI education of physicians.

This proposal presents some limitations that should be discussed. First, our findings are somewhat context dependent as different settings may have different needs and resources available. We compiled a team of AI experts, which may not be feasible in other institutions. Nonetheless, our institution does not have access to large platforms that are integrated into daily radiology practice yet, which may in fact be comparable to many institutions worldwide. Second, our self-reported pre- and post-assessment surveys may introduce bias by inflating participants’ confidence post-curriculum due to the positive experience of completing the curriculum. The absence of objective assessments to measure participants’ proficiency in understanding and evaluating AI software before and after the course is a limitation. To enhance future curricula, we recommend integrating more objective assessments pre- and post-curriculum, such as multiple-choice questions, case-based assessments, or practical exams, ensuring a more comprehensive evaluation of participants’ AI-related competence. Besides, in our surveys, we employed the Likert scaling method to evaluate various components of the curriculum. Additionally, integrating the net promoter score (NPS) may be beneficial, especially for queries related to recommending the curriculum. Third, our curriculum was developed and executed in a single academic hospital and with a small sample size (12 participants). Therefore, our results may not fully generalize to other radiology departments. However, its independence from a commercially available platform and its practical feasibility in terms of course duration would make it easier to implement in any residency program. Lastly, the implementation of the curriculum in a singular manner, combined with the reliance on a one-time post-curriculum survey, lacks a longitudinal assessment. Integrating an annually repeating curriculum accompanied by consistent survey results and experiential feedback would enhance its value. We plan to further personalize our AI curriculum to make it available annually to our radiology residents and to use annual survey’ results to further meet the needs of the participants continuously. Investigating these aspects further through a multicenter approach for future AI educational programs would be interesting. This could potentially offer a broader perspective and deeper insight into the perceptions and experiences of residents across different institutions.

In conclusion, designing an AI curriculum for radiology residents and implementing it into a radiology residency program is feasible using the framework presented. The 3-day AI curriculum effectively increased participants’ perception of knowledge and skills about AI in radiology and can serve as a starting point for further customization.