- Original Article
- Open access
- Published:
Training on contrast-enhanced ultrasound LI-RADS classification for resident radiologists: a retrospective comparison of performance after training
Insights into Imaging volume 15, Article number: 205 (2024)
Abstract
Objectives
To evaluate the effects and benefits of training radiology residents on contrast-enhanced ultrasound (CEUS) according to the Liver Imaging Reporting and Data System (LI-RADS).
Methods
In total, 234 patients at high risk of hepatocellular carcinoma (HCC) who underwent CEUS were enrolled, including 27 lesions in the education set and 207 lesions in the test sets (a–d). Forty-five radiology residents and 4 radiology experts involved in CEUS LI-RADS training individually reviewed the test sets before, immediately after, and 3-months after training. The consistency with kappa values of the description of CEUS features, the classification of focal liver lesions (FLLs), and the diagnostic performance were evaluated.
Results
The level of agreement between the radiology experts and residents improved after training (all p < 0.05), while there were no significant differences between the post-training and 3-months post-training results (all p > 0.05). The sensitivity, specificity, positive predictive value, and area under the curve (AUC) based on the CEUS LI-RADS classification of the radiology experts in the diagnosis of HCC were 62.9%, 96.4%, 96.3%, and 0.796, respectively. The diagnostic performance of the radiology residents significantly improved after training (all p < 0.05). Misunderstanding of definitions and subjective interpretation of images were the main reasons for disagreement with multiple responses.
Conclusion
Dedicated CEUS LI-RADS training improved the performance of radiology residents in diagnosing FLLs and their agreement with radiology experts on CEUS features. Images and videos to explain typical features of the training were essential to improve agreement between the radiology experts and residents.
Critical relevance statement
Agreement on lesion descriptors between radiology experts and residents can improve with training.
Key Points
-
The diagnostic performance of less experienced radiologists for diagnosing HCC could be improved by training.
-
Images and videos to explain typical features during training were essential.
-
Agreement on lesion descriptors between radiology experts and residents improved after training.
Graphical Abstract
![](http://media.springernature.com/lw685/springer-static/image/art%3A10.1186%2Fs13244-024-01786-6/MediaObjects/13244_2024_1786_Figa_HTML.png)
Introduction
Hepatocellular carcinoma (HCC) is the most common liver malignancy and is associated with an overall 5-year survival rate of less than 12% [1]. Early diagnosis can improve patient prognosis, with a 5-year survival rate of 80% [2]. The majority of HCC cases can be diagnosed based on imaging findings alone [3], and unnecessary biopsies should be avoided. However, for some lesions classified as LR4 (probable HCC) or LR M (probably or definitely malignant but not HCC specific), a biopsy is still needed. Computed tomography (CT) and magnetic resonance imaging (MRI) have been widely used in the diagnosis of focal liver lesions (FLLs) [4], and the diagnostic performance of contrast-enhanced ultrasound (CEUS) is similar to that of CT and MRI [5]. However, compared with CT or MRI, CEUS has greater temporal resolution [6] and can more clearly reflect the arterial enhancement pattern of FLLs. As the contrast agent functions as a pure-blood pooling agent, the washout observed by CEUS is real and clear [7], which is helpful in distinguishing HCC from intrahepatic cholangiocarcinoma (ICC) [8]. The American College of Radiology released a scheme for standardising CEUS reports for FLLs at risk of HCC, which was named the Contrast-Enhanced Ultrasound Liver Imaging Reporting and Data System (CEUS LI-RADS). It is very important for radiologists to perform standardised evaluations of FLLs, effectively communicate with physicians, especially multicentre physicians, and assist physicians in making correct clinical decisions [9].
The European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB) [10] mandates in its minimum training requirements that CEUS should be performed by experienced observers. In a multicentre international study [11] based on CEUS LI-RADS, the level of agreement between experienced observers reached 0.73, which was better than that between similarly experienced CT/MRI observers; however, the level of agreement between less experienced observers was worse than that between similarly experienced CT/MRI observers [12], especially in evaluating washout. Disagreements in the assessment of FLLs based on CEUS LI-RADS have led to increased false-positive rates, resulting in unnecessary biopsies, excessive radiography, and even excessive treatment, which seriously affect the physical and mental health of patients. Many studies have shown that the level of agreement in lesion evaluation based on the Breast Imaging Reporting and Data System (BI-RADS) and Thyroid Imaging Reporting and Data System (TI-RADS) between less and more experienced observers could be improved by training [13]; however, at present, there is a lack of research on whether training could improve the level of agreement during lesion evaluations based on CEUS LI-RADS between inexperienced and experienced observers.
This study was performed to assess whether training improves diagnostic performance and the level of agreement between inexperienced and experienced observers and to provide specific educational recommendations for observers who need training. Furthermore, we included FLLs with pathological outcomes to assess the diagnostic performance of observers with varying levels of expertise using CEUS LI-RADS, and further evaluated the effectiveness of training.
Materials and methods
This retrospective study was approved by the ethics committee of our research centre.
Patients
This study included 234 FLLs from 234 patients who underwent ultrasound and CEUS at our facility between January 2019 and June 2020. For patients with multiple liver lesions, the most suspicious lesion was selected for study. The inclusion criteria were as follows: (1) cirrhosis or chronic hepatitis; (2) detection of malignant lesions and most benign lesions by ultrasound with pathological biopsy results and imaging follow-up for more than 12 months for other benign lesions; and (3) an interval of less than 4 weeks between CEUS and clinical diagnosis. The exclusion criteria were as follows: (1) cirrhosis caused by congenital liver fibrosis or vascular disease; (2) administration of local or systemic therapy; and (3) poor video image quality (except for one patient in the education set). All included lesions met the CEUS LI-RADS criteria.
CEUS examination
All CEUS examinations in the study were performed according to EFSUMB guidelines [9]. The videos included in the study were acquired by two expert radiologists. The ultrasound contrast agent (25 mg, SonoVue) was mixed with 5 mL of 0.9% saline solution, and 2.4 mL of this suspension was injected through the antecubital vein.
All videos comprised images from both ultrasound and CEUS screens. Videos were recorded continuously from the arrival of microbubbles through the first 60 s; thereafter, images were captured intermittently (every 30 s) to minimise microbubble destruction until the microbubbles had cleared completely from the circulation (4–6 min). The phase of angiographic perfusion was based on CEUS LI-RADS.
Image selection and interpretation
CEUS videos of 234 liver lesions were selected and reviewed by two expert radiologists dedicated to CEUS examinations based on CEUS LI-RADS and then reviewed by another four expert radiologists. First, four expert radiologists performed the evaluations separately, and then agreement was reached after discussion. The clinical, histopathological, and CT/MRI findings of all videos were not known by the four expert radiologists during the review to minimise bias in ultrasound interpretation. The videos were classified into four groups (a–d), as follows: (a) The education set comprised 27 videos that showed typical CEUS features or were representative of cases described in the CEUS LI-RADS classification. The education set included ultrasound images of the lesions (including the size of the lesions), dual ultrasound and CEUS videos, the consensus results of the classification and CEUS features of the lesions reached by four expert radiologists based on CEUS LI-RADS, and the pathology and CT/MRI findings. The above contents were recorded on a table. The CEUS features included arterial-phase enhancement features, washout, onset of washout, and degree of washout. (2) The three test sets (b, c, d), each containing 69 videos, were organised like the education set. The three tables (b, c, d) could be reviewed by the trainees after the test. The three test sets (b, c, d) were used for the pretraining, post-training, and 3-months post-training evaluations, respectively.
From August 2020 to June 2022, 45 resident radiologists from 9 institutions (2 academic centres and 7 community hospitals) participated in the training. Before training, 45 resident radiologists reviewed test set a. According to the CEUS LI-RADS online, the resident radiologists filled out tables with CEUS features and categories for each lesion. Then, each resident radiologist received theoretical training. Twenty-one resident radiologists participated in the online training, and 24 resident radiologists participated in the offline training. Each training session consisted of three lectures, each of which lasted more than 2 h. The content of the training mainly included the explanation and case presentation of the CEUS LI-RADS. During the training period, 45 resident radiologists were free to review the education set, and 4 expert radiologists answered questions regarding the relevant contents of CEUS LI-RADS during the months of training. Test sets c and d were reviewed, and tables were constructed with CEUS features and categories for each lesion by resident radiologists immediately after training and 3-months after training. The reasons for errors were ascertained with a questionnaire after the three tests. The questionnaire included the following questions: (1) Why did you misclassify the lesion? Please write what you think were the reasons for the classification. (2) If it was a lesion imaging feature recognition error, please state the feature you think, and analyse the reason for the error. To simulate daily diagnosis and prevent “background bias,” the lesions in the three test sets were randomly assigned [14].
Statistical analysis
All the statistical analyses were performed with SPSS software (version 27) and GraphPad Prism (version 9). Continuous variables are presented as averages ± standard deviations, and categorical variables are presented as percentages. One-way ANOVA and the χ2 test were used for comparisons between groups. Cohen’s kappa was used to analyse the consistency of the CEUS feature descriptions and categories between the resident and expert radiologists in the three tests, with kappa values ranging from 0.00 to 0.20 indicating slight agreement; 0.20 to 0.40 indicating fair agreement; 0.40 to 0.60 indicating moderate agreement; 0.60 to 0.80 indicating substantial agreement; and 0.80 to 1.00 indicating almost perfect agreement. Multiple responses were used to analyse the reasons for disagreements between the resident and expert radiologists. For the diagnosis of HCC, LI-RADS category 5 results were considered positive, and other results were considered negative. The diagnostic performance of the residents and experienced radiologists was calculated, including the sensitivity, specificity, Positive predictive value (PPV), and accuracy. The area under the receiver operating characteristic (ROC) curve (AUC) was calculated. All p-values < 0.05 were considered to indicate statistical significance.
Results
Patient and lesion characteristics
A total of 234 patients were enrolled; they included 153 men and 81 women aged between 24 and 83 years (58.3 ± 12.8 years). The exclusion flow chart is shown in Fig. 1. Patient and tumor characteristics are shown in Table 1.
Agreement on classifications and CEUS features between resident and expert radiologists
The level of agreement on the CEUS features and classifications of the three test sets between the resident and expert radiologists was evaluated, as shown in Table 2.
Agreement between the resident and expert radiologists was significantly improved post-training and 3-months post-training for CEUS features and classifications (all p < 0.05). In the post-training and 3-months post-training evaluations, most resident and expert radiologists showed almost a perfect level of agreement in terms of CEUS features and classifications, except for rim enhancement.
Kappa values did not show significant differences in the level of agreement for CEUS features or classifications between post-training and 3-months post-training (all p > 0.05). The difference in the level of agreement on FLL classification between the resident and expert radiologists in the three test sets is shown in Fig. 2.
Diagnostic performance of the resident and expert radiologists
Compared with the pathological and CT/MRI findings, the accuracy, sensitivity, specificity, PPV and AUC of LI-RADS 5 in the diagnosis of HCC were 76.3%, 62.9%, 96.4%, 96.3%, and 0.796, respectively. The classifications of the expert radiologists and comparisons with the pathology and CT/MRI findings are shown in Table 3. The accuracy, sensitivity, specificity, PPV, and AUC of the diagnosis of HCC by the resident radiologists post-training (75.8%, 69.2%, 97.8%, 99.0%, 0.834) and 3-months post-training (74.5%, 51.6%, 96.8%, 92.0%, 0.742) were greater than those of the resident radiologists pretraining (58.5%, 33.2%, 87.6%, 76.2%, 0.604) (p < 0.05), but there were not significant differences in the values between post-training and 3-months post-training (p > 0.05, Fig. 3).
Reasons for disagreements between resident and expert radiologists on CEUS features before training
A total of 32.6% of disagreements involving arterial-phase hyperenhancement (APHE) were due to subjective image interpretation errors, in which rim enhancement was confused with APHE. A total of 21.5% of disagreements regarding APHE were due to misunderstanding definitions; APHE could not be recognised when multiple enhancement patterns existed (Fig. 4). A total of 93.1% of disagreements regarding rim enhancement were due to image subjective interpretation errors, in which rim enhancement was confused with APHE and discontinuous spherical hyperenhancement. A total of 67.4% of disagreements regarding washout were due to a misunderstanding of definitions; resident radiologists believed that the degree of lesion enhancement in the portal phase was weaker than that in the arterial phase, which could be defined as a washout and partial washout was defined as no washout. A total of 25.8% of disagreements regarding the degree of washout were due to subjective interpretation errors because of differences in subjective descriptions of washout marked as “punched out” and “black” in the guidelines (Fig. 5).
CEUS LI-RADS M (hepatocellular carcinoma). Nodule in a 47-year-old man with hepatitis virus B-related cirrhosis. US image (a) showed a hypoechoic lesion sized 4.0 cm in segment VI (arrow). CEUS image (b) showed partial APHE during the arterial phase (arrow). CEUS image (c) showed early washout during the portal venous phase (arrow)
CEUS LI-RADS M (adenocarcinoma of intestinal origin). Nodule in a 68-year-old man with hepatitis virus B-related cirrhosis. US image (a) showed a hypoechoic lesion sized 2.1 cm in segment VI (arrow). CEUS image (b) showed APHE during the arterial phase (arrow). CEUS image (c) showed a marked washout during the late phase (arrow)
Discussion
A standardised report based on CEUS LI-RADS is an effective tool for promoting correct clinical decisions in patients at risk for HCC. [15, 16] showed that practitioners should be trained to achieve greater consistency. Our study showed that training could effectively improve the ability of resident radiologists to recognise CEUS features and diagnose HCC. In addition, the resident radiologists showed the same excellent ability to perform CEUS feature recognition and FLL classification in the test 3-months after training, indicating that training can significantly enhance the diagnostic capabilities of resident radiologists over the long term and that the training process is reproducible.
Our study showed that resident radiologists had moderate consistency with expert radiologists in evaluating APHE before training. There was fair and slight consistency between the resident and expert radiologists in terms of evaluating rim enhancement as well as the appearance, onset, and degree of washout and final classification, which are similar to the results of previous studies [17]. Due to the above incorrect recognition of the CEUS features of FLLs, resident radiologists have incorrectly classified FLLs and have failed to effectively diagnose them.
In view of the problems in lesion characteristic evaluation and classification by resident radiologists before training, expert radiologists implemented systematic and targeted norms in the subsequent training. First, the definitions and interpretations of the characteristics and classifications were clarified. APHE could be considered present if the APHE was shown in the entire nodule or only part of the nodule. Washout was defined as a reduction in the enhancement of part or all of the lesion with respect to the surrounding normal liver parenchyma during or after the arterial phase, even if there was no high enhancement during the arterial phase. Second, video-derived images were used to objectify some subjective descriptions. Through the comparison of rim enhancement and discontinuous spherical hyperenhancement, the characteristics of rim enhancement were defined. The image of marked washout was repeatedly clarified by replaying the marked washout videos conforming to the punched out and black features in the education set and in test set a. Finally, timely feedback based on clinical and pathological results after the test could effectively improve ultrasonic interpretation skills [18].
The specificity and PPV of the expert radiologists for HCC diagnosis were 96.4% and 96.3%, respectively, which were similar to the results reported by Terzi, once again confirming the effectiveness of CEUS LI-RADS for diagnosing HCC. In our study, no expert radiologists classified the ICC as LR-5, suggesting a lower risk of diagnosing ICC as HCC based on CEUS LI-RADS. In this study, one patient with focal nodular hyperplasia (FNH) with a background of fatty liver was classified as having LR-5 by an expert radiologist due to APHE and mild washout in the late phase. In addition, another case of haemangioma with APHE and mild washout in the late phase was classified as LR-5. When the haemangioma was accompanied by a shunt in the arteriovenous fistula, there was a mild washout, showing characteristics of malignancy [19, 20]. In the absence of other images and clinical results, expert radiologists classified the lesions as LR-5. CEUS LI-RADS 2017 requires a further standardised description of the diversity of FLLs.
The 45 resident radiologists came from 9 medical institutions of different levels, including 2 community hospitals and 2 academic centres. The diversity of trained personnel is conducive to further research on the possibility of the popularisation of CEUS LI-RADS in medical institutions of different levels for multicentre and multidisciplinary communication. The main limitation of our study was that the trainees played the video on the computer screen. Although the high-quality CEUS videos we selected were sufficient for effectively evaluating the lesions, other differences may have been caused by technical reasons.
Our study suggested that standardised scientific training could improve the performance of resident radiologists in evaluating FLLs, which is conducive to the early diagnosis of HCC. Moreover, images and videos to explain typical features were essential for improving the level of agreement between the radiology experts and residents.
Data availability
The data and materials used in this study are available. All data and materials are preserved by the corresponding author and can be obtained by contacting the corresponding author.
Abbreviations
- AUC:
-
Area under the curve
- CEUS:
-
Contrast-enhanced ultrasound
- CT:
-
Computed tomography
- EFSUMB:
-
European Federation of Societies for Ultrasound in Medicine and Biology
- FLL:
-
Focal liver lesion
- FNH:
-
Focal nodular hyperplasia
- HCC:
-
Hepatocellular carcinoma
- ICC:
-
Intrahepatic cholangiocarcinoma
- LI-RADS:
-
Liver Imaging Reporting and Data System
- MRI:
-
Magnetic resonance imaging
- PPV:
-
Positive predictive value
References
El-Serag HB (2011) Hepatocellular carcinoma. N Engl J Med 365:1118–1127. https://doi.org/10.1056/NEJMra1001683
Tang A, Hallouch O, Chernyak V et al (2018) Epidemiology of hepatocellular carcinoma: target population for surveillance and diagnosis. Abdom Radiol (NY) 43:13–25. https://doi.org/10.1007/s00261-017-1209-1
Bota S, Piscaglia F, Marinelli S et al (2012) Comparison of international guidelines for noninvasive diagnosis of hepatocellular carcinoma. Liver Cancer 1:190–200. https://doi.org/10.1159/000343833
Li J, Yang L, Ma L et al (2021) Diagnostic Accuracy of Contrast-Enhanced Ultrasound Liver Imaging Reporting and Data System (CEUS LI-RADS) for differentiating between hepatocellular carcinoma and other hepatic malignancies in high-risk patients: a meta-analysis. Ultraschall Med 42:187–193. https://doi.org/10.1055/a-1309-1568
Zheng W, Li Q, Zou XB et al (2020) Evaluation of contrast-enhanced US LI-RADS version 2017: application on 2020 liver nodules in patients with hepatitis B infection. Radiology 294:299–307. https://doi.org/10.1148/radiol.2019190878
Schellhaas B, Hammon M, Strobel D et al (2018) Interobserver and intermodality agreement of standardised algorithms for non-invasive diagnosis of hepatocellular carcinoma in high-risk patients: CEUS-LI-RADS versus MRI-LI-RADS. Eur Radiol 28:4254–4264. https://doi.org/10.1007/s00330-018-5379-1
Kim TK, Noh SY, Wilson SR et al (2017) Contrast-Enhanced Ultrasound (CEUS) Liver Imaging Reporting and Data System (LI-RADS) 2017—a review of important differences compared to the CT/MRI system. Clin Mol Hepatol 23:280–289. https://doi.org/10.3350/cmh.2017.0037
Wilson SR, Kim TK, Jang HJ et al (2007) Enhancement patterns of focal liver masses: discordance between contrast-enhanced sonography and contrast-enhanced CT and MRI. AJR Am J Roentgenol 189:W7–W12. https://doi.org/10.2214/AJR.06.1060
Bartolotta TV, Terranova MC, Gagliardo C et al (2020) CEUS LI-RADS: a pictorial review. Insights Imaging 11:9. https://doi.org/10.1186/s13244-019-0819-2
Claudon M, Dietrich CF, Choi BI et al (2013) Guidelines and good clinical practice recommendations for contrast enhanced ultrasound (CEUS) in the liver-update 2012: a WFUMB-EFSUMB initiative in cooperation with representatives of AFSUMB, AIUM, ASUM, FLAUS and ICUS. Ultraschall Med 34:11–29. https://doi.org/10.1055/s-0032-1325499
Fowler KJ, Tang A, Santillan C et al (2018) Interreader reliability of LI-RADS version 2014 algorithm and imaging features for diagnosis of hepatocellular carcinoma: a large international multireader study. Radiology 286:173–185. https://doi.org/10.1148/radiol.2017170376
Terzi E, Iavarone M, Pompili M et al (2018) Contrast ultrasound LI-RADS LR-5 identifies hepatocellular carcinoma in cirrhosis in a multicenter restropective study of 1,006 nodules. J Hepatol 68:485–492. https://doi.org/10.1016/j.jhep.2017.11.007
Barth BK, Donati OF, Fischer MA et al (2016) Reliability, validity, and reader acceptance of LI-RADS-an in-depth analysis. Acad Radiol 23:1145–1153. https://doi.org/10.1016/j.acra.2016.03.014
Egglin TK, Feinstein AR (1996) Context bias. A problem in diagnostic radiology. JAMA 276:1752–1755. https://doi.org/10.1001/jama.276.21.1752
Grani G, Lamartina L, Cantisani V et al (2018) Interobserver agreement of various thyroid imaging reporting and data systems. Endocr Connect 7:1–7. https://doi.org/10.1530/EC-17-0336
Hoang JK, Middleton WD, Farjat AE et al (2018) Interobserver variability of sonographic features used in the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am J Roentgenol 211:162–167. https://doi.org/10.2214/AJR.17.19192
Li W, Li L, Zhuang BW et al (2021) Inter-reader agreement of CEUS LI-RADS among radiologists with different levels of experience. Eur Radiol 31:6758–6767. https://doi.org/10.1007/s00330-021-07777-1
Berg WA, Blume JD, Cormack JB et al (2012) Training the ACRIN 6666 Investigators and effects of feedback on breast ultrasound interpretive performance and agreement in BI-RADS ultrasound feature analysis. AJR Am J Roentgenol 199:224–235. https://doi.org/10.2214/AJR.11.7324
Makamure J, Zhao D, Liu Y et al (2019) Hepatic hemangioma with arterioportal shunt: prevalence and lesion characteristics based on DSA, CT and MR imaging. Eur J Radiol 121:108715. https://doi.org/10.1016/j.ejrad.2019.108715
Kato H, Kanematsu M, Matsuo M et al (2001) Atypically enhancing hepatic cavernous hemangiomas: high-spatial-resolution gadolinium-enhanced triphasic dynamic gradient-recalled-echo imaging findings. Eur Radiol 11:2510–2515. https://doi.org/10.1007/s003300101110
Funding
There is no funding for the research.
Author information
Authors and Affiliations
Contributions
The conceptualisation is provided by corresponding authors Z.S. and X.M., experimental data, formal analysis, and writing draft are generated by T.D., H.Z., and M.Q.; software is operated by Y. Song and Y. Sun, and writing-review editing was performed by Z.S., X.M., and T.D. All authors read and approved the final manuscript.
Corresponding authors
Ethics declarations
Ethics approval and consent to participate
This retrospective study was approved by our institutional ethics committee.
Consent for publication
We have obtained consent for publication from the patients.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Dai, T., Zhu, H., Qiao, M. et al. Training on contrast-enhanced ultrasound LI-RADS classification for resident radiologists: a retrospective comparison of performance after training. Insights Imaging 15, 205 (2024). https://doi.org/10.1186/s13244-024-01786-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13244-024-01786-6