Fig. 2

Overview of machine learning validation techniques. a Bootstrapping is based on resampling with replacement, allowing to create n datasets from an original sample. These may include any number of copies of a specific instance from the original case, even none. b K-fold cross-validation is based on dividing the dataset in k parts, using each in turn as the validation set and the remaining as the training data. c In leave-one-out cross-validation, each instance in the dataset is used for model validation, using the remaining for model training. d In nested cross-validation, two loops of validation take place. The training data from each outer loop undergo an additional K-fold cross-validation. The figure depicts a fourfold outer loop paired with a threefold inner loop. In (e) Monte Carlo and (f) random-split cross-validation, the folds are not made up of contiguous data but from random sampling of the entire dataset. During the first, a sample may appear in multiple folds, which is not possible in random-split cross-validation. g In leave-P-out cross-validation, the K-fold cross-validation process is iterated to obtain all possible folding splits for the data