Automated segmentation of liver segment on portal venous phase MR images using a 3D convolutional neural network
Insights into Imaging volume 13, Article number: 26 (2022)
We aim to develop and validate a three-dimensional convolutional neural network (3D-CNN) model for automatic liver segment segmentation on MRI images.
This retrospective study evaluated an automated method using a deep neural network that was trained, validated, and tested with 367, 157, and 158 portal venous phase MR images, respectively. The Dice similarity coefficient (DSC), mean surface distance (MSD), Hausdorff distance (HD), and volume ratio (RV) were used to quantitatively measure the accuracy of segmentation. The time consumed for model and manual segmentation was also compared. In addition, the model was applied to 100 consecutive cases from real clinical scenario for a qualitative evaluation and indirect evaluation.
In quantitative evaluation, the model achieved high accuracy for DSC, MSD, HD and RV (0.920, 3.34, 3.61 and 1.01, respectively). Compared to manual segmentation, the automated method reduced the segmentation time from 26 min to 8 s. In qualitative evaluation, the segmentation quality was rated as good in 79% of the cases, moderate in 15% and poor in 6%. In indirect evaluation, 93.4% (99/106) of lesions could be assigned to the correct segment by only referring to the results from automated segmentation.
The proposed model may serve as an effective tool for automated anatomical region annotation of the liver on MRI images.
3D-CNN exhibits satisfactory performance for segmentation of liver-segment on MR images.
It is robust across different MR scanners and variable liver backgrounds.
It lays a foundation for computer-aided analyses of liver MR images.
Subdivision of the human liver into anatomical regions is part of the daily routine of radiologists, especially considering the need for precise preoperative localization of focal liver lesions. Currently, the type of hepatectomy chosen depends mainly on the segmental localization of the lesion . In addition, the identification of liver segment is also crucial to reduce the risk of liver surgery . Several quantitative parameters, such as liver fat fraction and R2* value, also need to be measured at a segment level due to the heterogeneous nature of some diseases in liver . These tasks are usually performed by radiologists via visual interpretation or manual segmentation, which are labor-intensive, time-consuming, and prone to intra- and inter-observer variations. A practicable automated liver segment segmentation tool is therefore very desirable for clinical purposes.
In the past two decades, a lot of research work has been done in computer-assisted liver segment segmentation [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21], most of them using traditional machine learning techniques [4, 7,8,9,10,11,12,13,14,15,16,17,18,19,20,21], which cannot meet the needs of clinical applications in terms of segmentation performance and efficiency. The method proposed by Lebre et al.  requires first segmentation of the hepatic vessels using the skeletonization process, and then the main direction of the largest vessels was extracted to achieve separation of different liver segments. However, the whole process took more than 8 min, and good results are highly dependent on accurate vascular segmentation, which is difficult owing to the complex intertwined vascular anatomy in the liver. To tackle this problem, in a recently published paper, Wu et al.  introduced a multiple feature-based method. Their method improved the accuracy of vessel separation and was subsequently expected to improve the segmentation of liver segments. However, the quantitative results of liver segments segmentation were not given in this paper and their method took an average of 20.8 s per case to obtain liver segments.
In recent years, the rapid development of deep learning, particularly the advent of convolutional neural network (CNNs), has pushed medical image segmentation to a new level . Tian et al.  used a 2.5-dimensional (2.5D) class-aware deep neural network with spatial adaptation to obtain Couinaud segmentation of the liver from CT images. They achieved an average dice score of 0.882 and the entire running time was quite fast (~5 s). The results demonstrated the potential of deep learning to improve segmentation efficiency and accuracy over traditional methods. For MR images, Mojtahed et al.  recently reported a Couinaud segment volume measurement tool based on deep learning, but this device is semi-automatic (needs human position eight landmark placement). To our knowledge, there is still lack of a deep learning-based fully automatic tool for liver segments segmentation on MR images. In this study, we would like to develop a 3D U-net based algorithm to automatically segment the liver into Couinaud regions on MR images.
Materials and methods
This retrospective study was approved by the institutional review board of Beijing Friendship Hospital, and informed consent was waived. All abdominal MR scans were performed as part of routine clinical care for patients. For model development, we initially obtained 744 multiphase contrast-enhanced MR scans by searching picture archiving and communication system (PACS) randomly between January 2017 and February 2019. After excluding the following cases: (1) Those with apparent liver deformation due to advanced cirrhosis or large tumors (n = 33). (2) Those with poor imaging quality or severe artifacts (n=8), and (3) those after partial hepatectomy or liver transplantation (n = 21). A total of 682 MR scans were included in the final cohort (367, 157, 158 for training, validation and testing, respectively).
For clinical evaluation, a consecutive sample of 100 patients who underwent abdominal enhanced MR in our hospital between April 1 and 24, 2021 was collected.
One 1.5T and four 3.0T MR scanners (GE, GE Health care, Boston, USA; Siemens Medical Solutions, Forchheim, Germany, Siemens Medical Solutions, Forchheim, Germany; Philips Medical Systems, Best, the Netherlands) were used in this study. 3D T1-weighted breath-hold sequences from our institutional standardized liver MR imaging protocols were used for contrast-enhanced imaging. Parameters varied across equipment and ranged from TR 3.4–4.1ms, TE 1.15–1.91ms, flip angle 10–15°, bandwidth 125 and 500 Hz, slice thickness 3–4.4 mm, image matrix 216 × 188 to 288 × 216 and field-of-view 380 × 342 mm to 400 × 369 mm, acquisition time 10 s. In this study, only portal venous phase (PVP) images acquired at ~60s postinjection were used. Anonymized MR images in DICOM format were exported from PACS.
Classification of liver segment was based on Couinaud's description , as shown in Fig. 1. The ground-truth was generated by two radiologists (Z.Y.S. and Z.X.L.) with more than 5 years of experience in abdominal imaging using Mimics software (version 19.0; Materialise, Leuven, Belgium). All the images were divided into two groups and evenly distributed to each radiologist. To ensure the consistency of the segmentation results, two radiologists went through a training session and segmented several cases together before formal segmentation.
We used 3D U-Net as the backbone, which is built with 3D conv, ReLU, BN and pooling layers . The network takes a 3D image as input, and produces the output with the same shape as the input image. The images were normalized by spacing and cropped into a fixed shape (160 × 160 × 160) in a sliding window approach before feeding into the model. As the variation in the gray distributions of MR scan images should not be ignored compared to CT scans, a grayscale normalization by subtracting the mean and divide by the standard deviation was applied after spacing normalization.
We introduced two branches in the output layer. As shown in Fig. 2, the upper segment produces the liver boundary and the other segment is the pixelwise segmentation of the liver segments. The late output serves as the final output, while the previous output helps to produce a better boundary. The model is trained end-to-end using the SGD optimizer on the training dataset. The learning rate of the training is start at 1e−3 and decay by 0.1 at every 50 epochs. We use Dice-Loss as the loss function, and batch size is set to 8. The training is end at 150 epochs.
Quantitative results were derived on the internal test dataset (158 cases not used for training). The following four parameters were used:
The Dice similarity coefficient (DSC)
DSC is used to measure the voxel overlap between the prediction (X) and the ground truth (Y). The value of this metric ranges from 0 to 1, with minimum and maximum values indicating no overlap and fully overlap, respectively.
The mean surface distance (MSD) and Hausdorff distance (HD)
MSD and HD are designed to quantify surface-based difference between X and Y. MSD calculates the mean distance between the two surfaces, whereas HD measures the maximum distance between them.
The volume ratio (RV)
RV computes the volume ratio of the liver segments from two segmentations, defined as RV (X, Y) = VX/VY, where VX and VY represent the volume of model and manual segmentations, respectively.
The value of DSC, MSD, HD and RV for each liver segment and the average of eight segments were calculate. All of these quantitative parameters by different MRI manufacturers were also calculated. In addition, time consumed for AI-based segmentation and manual segmentation was compared.
Our model was applied to a total of 100 consecutive patients who received upper abdominal contrast-enhanced MR examinations in our hospital from April 1 to 24, 2021. The clinical evaluation consists of the following two parts:
The segmentations generated by the model were evaluated independently by two radiologists (X.H. and Y.D.W.) with more than 15 years’ experience. Segmentation quality of each patient was classified as either good, moderate, or poor based on the following two main elements: a. The accuracy of separation planes placement; b. whether inter-segment segmentation fault occurs. Accuracy of separation planes placement was determined by how much they shifted off center. For the three vertical planes along the hepatic veins, a shift ≤ 5 mm was defined as slight, 5–10 mm as moderate and ≥ 10 mm as severe. For the two horizontal planes along the portal veins, a shift ≤ 2 slices was defined as slight, 2–5 slices as moderate and ≥ 5 slices as severe. Inter-segment segmentation faults are the errors that cannot be explained by misplacement of the separation planes and are usually irregular shaped. Finally, segmentations with two or less slight shifts and no other errors, or with only one moderate shift and no other errors were classified as good; Those with more than two slight shifts or more than one moderate shifts, but no severe plane shifts and no inter-segment segmentation faults, were classified as moderate. Once the segmentation had a severe shift or inter-segment segmentation fault, it was classified as poor segmentation. Differences were rerated by an abdominal subspecialty radiologist (Y.Z.H.) with more than 30 years of experience.
The accuracy of segmentation was assessed indirectly based on the localization of lesions in the liver. The steps were as follows: first, cases with uncountable lesions or no lesions were excluded. Then, for each of the remaining cases, all the focal abnormal findings (larger than 5 mm) that could be seen in PVP were identified. If a liver had more than five lesions, only the five largest lesions were selected. Finally, for each eligible lesion, the specific liver segment that it resides (by only referring to the automated segmentation results) was recorded and judged by radiologists. Detection, screening, and localization judgment of lesions were performed by two radiologists (X.H. and Y.D.W.) in consensus. One radiologist (Y.Z.H.) arbitrated in case of disagreement. Fig. 3 illustrates the indirect evaluation with two examples.
Continuous variables were expressed as mean ± standard deviation (SD) or as the median and interquartile range depending on the normality of the data. Categorical variables were summarized as frequencies and percentages. Reader agreement was tested by the linearly weighted kappa coefficient. All statistical analyses were conducted using SPSS software version 25.0 (IBM, Armonk, New York). Significance was set at p < 0.05.
Patients and image characteristics
A total of 782 patients who underwent multiphase contrast-enhanced MR scans at our hospitals were included in this study. The mean age was 59 ± 15 years, and there were slightly more males (404; 51.7%). The demographic and MR scan profiles that were employed for each dataset are summarized in Table 1.
Experimental results show that our method is capable of accurately subdividing the input liver MR images into eight anatomical regions, with an average DSC of 90.20%. The good value of other metrics including HSD, MSD, and RV further confirmed the accuracy of the proposed method. Tables 2 and 3 summarize all four metrics (DSC, MSD, HD, and RV) on the whole dataset and on the data by different MR manufacturers, respectively. As shown in Table 3, there were no significantly difference of the quantitative evaluation results determined by this segmentation model on the data from different manufacturers.
The average running time using the developed model on an GPU-accelerated computing platform with Xeon(R) Silver 4210R (CPU) and GeForce RTX 2070 (GPU) was 8.2 ± 2.4 s. This takes much less time than manual segmentation, which took 26 min on average for a set of liver MR images.
Clinical evaluation results
The concordance between the two raters was satisfactory (kappa = 0.858). 6% (6/100) of the cases were rated as poor, meaning they had more serious segmentation errors that needed to be manually corrected. Among the six cases with poor segmentation, five cases were found for the patients with cirrhosis, and 1 case had diffuse metastases in the liver. Severe changes in liver shape and vascular anomalies caused by liver diseases appeared to be the main reason for the poor segmentation. Some examples of AI segmentations from different liver conditions are presented in Figs. 4 and 5.
After excluding 46 cases with no or uncountable lesions, a total of 106 lesions from 54 patients were used to evaluate the accuracy of AI segmentation. The maximal diameters of all lesions ranged from 0.6 to 6.0 cm (mean +/− SD, 1.6 cm +/− 0.8 cm) on MRI. According to the results, 93.4% (99/106) of the lesions were assigned to the segments where they were actually located. Moreover, the proposed method not only accurately located small lesions in a single segment but was also effective for massive lesions across segments. Seven of 106 lesions in total failed to be accurately located, of which two were due to the severe segmentation errors. Of the other errors, four lesions were on the boundary between two segments, and one lesion was on the edge of the liver and was not included in the segmentation.
In this study, we proposed and validated a deep learning model for fully-automated Couinaud segmentation of the liver based on PVP MRI images. Current experimental results reveal that our model can accurately subdivide the input liver MR images into eight anatomical regions and is robust across different MR scanners. The effectiveness of the proposed method was also validated in clinical scenarios and it would be a promising tool for assisting lesion localization automatically.
Subdivision of the liver into anatomical regions is critical because it is a fundamental step in potential computer-aided diagnosis in liver imaging. Most researchers who work on liver segment segmentation present their results on CT. Although CT is more widely used, MRI has several very definite advantages and can be used as a crucial complement to CT to screen for and diagnose liver disease [25, 26]. Lebre et al.  introduced a method for liver segment segmentation that is applicable to MR images. However, it requires liver segmentation and extraction of the vascular network before the Couinaud classification can finally be obtained. The time cost of the whole process is more than 8 min. Furthermore, their model performance was inadequately validated on a very small dataset (4 cases) and no quantitative results were given. Zhang et al.  proposed a method for partitioning the liver on 4D dynamic contrast-enhanced MR (DCE-MR) imaging. They obtained a mean DSC value of 88.43%. The accuracy is acceptable considering that it is produced on DCE-MR images with low spatial resolution. However, their method is not fully automated as manual feature extraction is needed in the process. In this work, our fully-automated approach received a quantitative assessment on a larger testing dataset, and the results confirmed that it is efficient, accurate and robust on three heavyweight MR manufacturers. We believe this is a step forward toward achieving intelligent diagnosis for liver MR images.
Previously, the evaluation of liver segment segmentation was mostly performed by visual inspection  and volumetric validation [16,17,18, 20, 21], or a combination of both [10, 19]. It is a major challenge because it imposes to have actual segment specimens. Several recent studies have adopted new approaches, including quantitative assessment, qualitative assessment, and indirect assessment [4, 5, 8, 9]. However, they all have a small sample size problem. In this study, we made a thorough evaluation of the model by using three different methods. Quantitative evaluation provides exact results and allows comparison between different studies. A common metric used for this purpose is DSC. To date, the accuracy of liver segment segmentation produced by prior methods does not exceed a DSC of 88.2% . In this work, our model yielded a mean DSC of 90.2%. Good results were also obtained in HD, MSD and RV, metrics that have not yet been reported in similar research but have been widely used for segmentation evaluation. As a part of our experiment, the model was applied to real clinical scenarios for evaluation. As expected, our model generated good segmentation images in most cases, covering a wide range of liver appearances from normal to abnormal. It is worth mentioning that some cases with advanced liver cirrhosis or multiple liver metastases were also accurately segmented by our model and some of the large across-segment tumors did not affect the detection of boundary lines. Poor segmentation occurred mainly in cases with severe vascular problems, including abnormal vascular routes or vessels that were badly invaded by tumors. This problem may be solved by collecting this kind of case and retraining the model. Additionally, our approach has shown promising results in assisting lesion localization. As shown, 93.4% of the lesions were localized in the right segments simply by referring to the segmentation results without using any medical experience. This is meaningful considering the important role of automated lesion localization for future liver computer-aided diagnosis systems. Further investigating the mislocalized lesions, we found that most of them were at the edge of segment and pushed the borderline between two adjacent segments (Fig.3b). They were often mistaken for a cross-segmental lesion according to automated segmentation results. Only a few mislocalizations were caused by the real severe segmentation errors of liver segments.
Although the results are promising, our study still have several limitations. First, this is a retrospective single-institution study. Although the evaluation dataset maximally simulated real clinical scenario, more image data that collected prospectively from multiple centers are still necessary to assess the model performance. Second, manual segmentation of the qualitative evaluation dataset was produced by radiology residents rather than experienced specialists. Nonetheless, it has certain rationality because residents are often the ones who do segmentation work in clinical practice. Third, the performance of the model segmentation was validated only on PVP MR images. Future work should include extending the method to more sequences and modalities. Fourth, although the acceptable level of liver deformities by our method is good, it is still limited in cases in which the liver vasculature is severely deformed or has poor visibility. We will conduct future work on this problem.
In conclusion, we designed a 3D U-Net model by jointly adopting the pixelwise losses and boundary loss for liver segment segmentation on MRI images. A thorough evaluation from three respects was conducted, showing promising segmentation performance. We anticipate that this practical tool will assist in planning surgical strategies and enhancing computer-aided quantitative analyses of liver MR images in the future.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Convolutional neural network
Dice similarity coefficient
Inferior vena cava
Left hepatic vein
Mean surface distance
Middle hepatic vein
Picture archiving and communication system
Portal venous phase
Right hepatic vein
Cho JY, Han HS, Yoon YS, Shin SH (2008) Feasibility of laparoscopic liver resection for tumors located in the posterosuperior segments of the liver, with a special reference to overcoming current limitations on tumor location. Surgery 144(1):32–38. https://doi.org/10.1016/j.surg.2008.03.020
Clavien PA, Petrowsky H, DeOliveira ML, Graf R (2007) Strategies for safer liver surgery and partial liver transplantation. N Engl J Med 356(15):1545–1559. https://doi.org/10.1056/NEJMra065156
Hong CW, Cui JY, Batakis D et al (2021) Repeatability and accuracy of various region-of-interest sampling strategies for hepatic MRI proton density fat fraction quantification. Abdom Radiol (NY) 46(7):3105–3116. https://doi.org/10.1007/s00261-021-02965-5
Wu C, Fu T, Gao Y et al (2021) Multiple feature-based portal vein classification for liver segment extraction. Med Phys 48(5):2354–2373. https://doi.org/10.1002/mp.14745
Tian Y, Xue F, Lambo R et al (2021) Fully-automated functional region annotation of liver via a 2.5D class-aware deep neural network with spatial adaptation. Comput Methods Programs Biomed 200(105818). https://doi.org/10.1016/j.cmpb.2020.105818
Mojtahed A, Núñez L, Connell J et al (2021) Repeatability and reproducibility of deep-learning-based liver volume and Couinaud segment volume measurement tool. Abdom Radiol (NY). https://doi.org/10.1007/s00261-021-03262-x
Alirr OI, Abd Rahni AA (2020) Automatic atlas-based liver segmental anatomy identification for hepatic surgical planning. Int J Comput Assist Radiol Surg 15(2):239–248. https://doi.org/10.1007/s11548-019-02078-x
Lebre MA, Vacavant A, Grand-Brochier M et al (2019) Automatic segmentation methods for liver and hepatic vessels from CT and MRI volumes, applied to the Couinaud scheme. Comput Biol Med 110(42–51). https://doi.org/10.1016/j.compbiomed.2019.04.014
Zhang T, Wu Z, Runge J et al (2018) A hybrid segmentation method for partitioning the liver based on 4D DCE-MR images. SPIE. https://doi.org/10.1117/12.2293530
Yang X, Yang JD, Hwang HP et al (2018) Segmentation of liver and vessels from CT images and classification of liver segments for preoperative liver surgical planning in living donor liver transplantation. Comput Methods Programs Biomed 158(41–52). https://doi.org/10.1016/j.cmpb.2017.12.008
Chen Y, Yue X, Zhong C, Wang G (2016) Functional region annotation of liver CT image based on vascular tree. Biomed Res Int 2016(5428737). https://doi.org/10.1155/2016/5428737
Rodrigues FM, Silva JS, Rodrigues TM (2012) An algorithm for the surgical planning of hepatic resections. In: 2012 IEEE 2nd Portuguese meeting in bioengineering (ENBENG). IEEE, pp 1–6
Pamulapati V, Venkatesan A, Wood BJ, Linguraru MG (2011) Liver segmental anatomy and analysis from vessel and tumor segmentation via optimized graph cutsInternational. In: MICCAI workshop on computational and clinical challenges in abdominal imaging. Springer, pp 189–197
Oliveira DA, Feitosa RQ, Correia MM (2011) Segmentation of liver, its vessels and lesions from CT images for surgical planning. Biomed Eng 10(30). https://doi.org/10.1186/1475-925x-10-30
Debarba HG, Zanchet DJ, Fracaro D, Maciel A, Kalil AN (2010) Efficient liver surgery planning in 3D based on functional segment classification and volumetric information. Annu Int Conf IEEE Eng Med Biol Soc 2010(4797–4800). https://doi.org/10.1109/iembs.2010.5628026
Schenk A, Zidowitz S, Bourquain H et al (2008) Clinical relevance of model based computer-assisted diagnosis and therapy. Medical Imaging 2008: Computer-Aided Diagnosis. International Society for Optics and Photonics, pp 691502
Boltcheva D, Passat N, Agnus V, Jacob-Da Col M-A, Ronse C, Soler L (2006) Automatic anatomical segmentation of the liver by separation planes. In: Medical imaging 2006: visualization, image-guided procedures, and display. International Society for Optics and Photonics, pp 61411A
Beichel R, Pock T, Janko C et al (2004) Liver segment approximation in CT data for surgical resection planning. In: Medical imaging 2004: image processing. International Society for Optics and Photonics, pp 1435–1446
Selle D, Preim B, Schenk A, Peitgen HO (2002) Analysis of vasculature for liver surgical planning. IEEE Trans Med Imaging 21(11):1344–1357. https://doi.org/10.1109/tmi.2002.801166
Soler L, Delingette H, Malandain G et al (2001) Fully automatic anatomical, pathological, and functional segmentation from CT scans for hepatic surgery. Comput Aided Surg 6(3):131–142. https://doi.org/10.1002/igs.1016
Glombitza G, Lamadé W, Demiris AM et al (1999) Virtual planning of liver resections: image processing, visualization and volumetric evaluation. Int J Med Inform 53(2–3):225–237. https://doi.org/10.1016/s1386-5056(98)00162-2
Cardenas CE, Yang J, Anderson BM, Court LE, Brock KB (2019) Advances in auto-segmentation. Semin Radiat Oncol 29(3):185–197. https://doi.org/10.1016/j.semradonc.2019.02.001
Couinaud C (1957) Le foie: études anatomiques et chirurgicales. Masson
Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotationInternational conference on medical image computing and computer-assisted intervention. Springer, pp 424-432
Van Beers BE, Daire JL, Garteiser P (2015) New imaging techniques for liver diseases. J Hepatol 62(3):690–700. https://doi.org/10.1016/j.jhep.2014.10.014
O’Neill EK, Cogley JR, Miller FH (2015) The ins and outs of liver imaging. Clin Liver Dis 19(1):99–121. https://doi.org/10.1016/j.cld.2014.09.006
We thank Dr. Ning Guo for constructive comments and critical revision of the manuscript. We thank Xianjun Han, Hao Ren, and Boyang Ma, Yaosong Zhou and Xiaolan Zhang for the manual segmentation work in this study.
This study has received funding by the National Natural Science Foundation of China (Grants 62171298, 61871276 and 82071876), Beijing Municipal Health Commission, Special Program of Scientific Research on health development in Beijing (Grants Shoufa 2018-2-2023), Beijing Hospitals Authority Clinical Medicine Development of Special Funding (ZYLX202101).
Ethics approval and consent to participate
The study was approved by local ethical committee and written informed consent was waived due to the retrospective nature of the study.
Consent for publication
The selected pictures of sample cases have personal consent form of each patient.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Han, X., Wu, X., Wang, S. et al. Automated segmentation of liver segment on portal venous phase MR images using a 3D convolutional neural network. Insights Imaging 13, 26 (2022). https://doi.org/10.1186/s13244-022-01163-1
- Liver segment segmentation
- Couinaud classification