cross diagram of kfold crossvalidation with k4 crossvalidation sometimes called rotation estimation and to give an insight on how the model will generalize to an independent dataset ie an unknown dataset for instance from a real problem one round of crossvalidation involves partitioning a sample of data into complementary subsets performing the analysis on one subset called the training set and validating the analysis on the other subset called the validation set or testing set to reduce variability in most methods multiple rounds of crossvalidation are performed using different partitions and the validation results are combined eg averaged over the rounds to give an estimate of the models predictive performance in summary crossvalidation combines averages measures of fitness in prediction to derive a more accurate estimate of model prediction performance purpose of crossvalidation suppose we have a model with one or more unknown parameters and a data set to which the model can be fit the training data set the fitting process optimizes the model parameters to make the model fit the training data as well as possible if we then take an independent sample of validation data from the same population as the training data it will generally turn out that the model does not fit the validation data as well as it fits the training data the size of this difference is likely to be large especially when the size of the training data set is small or when the number of parameters in the model is large crossvalidation is a way to estimate the size of this effect in linear regression we have real response values y1 yn and n pdimensional vector covariates x1 xn the components of the vectors xi are denoted xi1 xip if we use least squares to fit a function in the form of a hyperplane y a tx to the data xi yi1in we could then assess the fit using the mean squared error mse the mse for given estimated parameter values a and on the training set xi yi1in is 1 n i 1 n y i a t x i 2 1 n i 1 n y i a 1 x i 1 p x i p 2 displaystyle frac 1nsum i1nyiaboldsymbol beta tmathbf x i2frac 1nsum i1nyiabeta 1xi1dots beta pxip2 if the model is correctly specified it can be shown under mild assumptions that the expected value of the mse for the training set is np1np11 times the expected value of the mse for the validation set the expected value is taken over the distribution of training sets thus if we fit the model and compute the mse on the training set we will get an optimistically biased assessment of how well the model will fit an independent data set this biased estimate is called the insample estimate of the fit whereas the crossvalidation estimate is an outofsample estimate since in linear regression it is possible to directly compute the factor np1np1 by which the training mse underestimates the validation mse under the assumption that the model specification is valid crossvalidation can be used for checking whether the model has been overfitted in which case the mse in the validation set will substantially exceed its anticipated value crossvalidation in the context of linear regression is also useful in that it can be used to select an optimally regularized cost function in most other regression procedures eg logistic regression there is no simple formula to compute the expected outofsample fit crossvalidation is thus a generally applicable way to predict the performance of a model on unavailable data using numerical computation in place of theoretical analysis common types of crossvalidation two types of crossvalidation can be distinguished exhaustive and nonexhaustive crossvalidation exhaustive crossvalidation exhaustive crossvalidation methods are crossvalidation methods which learn and test on all possible ways to divide the original sample into a training and a validation set leavepout crossvalidation leavepout crossvalidation lpo cv involves using p observations as the validation set and the remaining observations as the training set this is repeated on all ways to cut the original sample on a validation set of p observations and a training set lpo crossvalidation requires training and validating the model c p n displaystyle cpn times where n is the number of observations in the original sample and where c p n displaystyle cpn is the binomial coefficient for p 1 and for even moderately large n lpo cv can become computationally infeasible for example with n 100 and p 30 30 percent of 100 as suggested above c 30 100 3 10 25 displaystyle c30100approx 3times 1025 leaveoneout crossvalidation leaveoneout crossvalidation loocv is a particular case of leavepout crossvalidation with p1 the process looks similar to jackknife however with crossvalidation one computes a statistic on the leftout samples while with jackknifing one computes a statistic from the kept samples only loo crossvalidation requires less computation time than lpo crossvalidation because there are only c 1 n n displaystyle c1nn passes rather than c k n displaystyle ckn however n displaystyle n passes may still require quite a large computation time in which case other approaches such as kfold cross validation may be more appropriate pseudocodealgorithm input x vector of length n with xvalues of data points y vector of length n with yvalues of data points output err estimate for the prediction error steps err 0 for i 1 n do define the crossvalidation subsets xin x yin y xout x yout interpolatexin yin xout yout err err y yout2 end for err errn nonexhaustive crossvalidation nonexhaustive cross validation methods do not compute all ways of splitting the original sample those methods are approximations of leavepout crossvalidation kfold crossvalidation in kfold crossvalidation the original sample is randomly partitioned into k equal sized subsamples of the k subsamples a single subsample is retained as the validation data for testing the model and the remaining k1 subsamples are used as training data the crossvalidation process is then repeated k times with each of the k subsamples used exactly once as the validation data the k results can then be averaged to produce a single estimation the advantage of this method over repeated random subsampling see below is that all observations are used for both training and validation and each observation is used for validation exactly once 10fold crossvalidation is commonly used but in general k remains an unfixed parameter for example setting k2 results in 2fold crossvalidation in 2fold crossvalidation we randomly shuffle the dataset into two sets d0 and d1 so that both sets are equal size this is usually implemented by shuffling the data array and then splitting it in two we then train on d0 and validate on d1 followed by training on d1 and validating ond0 when kn the number of observations the kfold crossvalidation is exactly the leaveoneout crossvalidation in stratified kfold crossvalidation the folds are selected so that the mean response value is approximately equal in all the folds in the case of binary classification this means that each fold contains roughly the same proportions of the two types of class labels holdout method in the holdout method we randomly assign data points to two sets d0 and d1 usually called the training set and the test set respectively the size of each of the sets is arbitrary although typically the test set is smaller than the training set we then train build a model on d0 and test evaluate its performance on d1 in typical crossvalidation results of multiple runs of modeltesting are averaged together in contrast the holdout method in isolation involves a single run it should be used with caution because without such averaging of multiple runs one may achieve highly misleading results ones indicator of predictive accuracy f as noted below will tend to be unstable since it will not be smoothed out by multiple iterations similarly indicators of the specific role played by various predictor variables eg values of regression coefficients will tend to be unstable while the holdout method can be framed as the simplest kind of crossvalidation repeated random subsampling validation this method also known as monte carlo crossvalidation randomly splits the dataset into training and validation data for each such split the model is fit to the training data and predictive accuracy is assessed using the validation data the results are then averaged over the splits the advantage of this method over kfold cross validation is that the proportion of the trainingvalidation split is not dependent on the number of iterations folds the disadvantage of this method is that some observations may never be selected in the validation subsample whereas others may be selected more than once in other words validation subsets may overlap this method also exhibits monte carlo variation meaning that the results will vary if the analysis is repeated with different random splits as the number of random splits approaches infinity the result of repeated random subsampling validation tends towards that of leavepout crossvalidation in a stratified variant of this approach the random samples are generated in such a way that the mean response value ie the dependent variable in the regression is equal in the training and testing sets this is particularly useful if the responses are dichotomous with an unbalanced representation of the two response values in the data measures of fit the goal of crossvalidation is to estimate the expected level of fit of a model to a data set that is independent of the data that were used to train the model it can be used to estimate any quantitative measure of fit that is appropriate for the data and model for example for binary classification problems each case in the validation set is either predicted correctly or incorrectly in this situation the misclassification error rate can be used to summarize the fit although other measures like positive predictive value could also be used when the value being predicted is continuously distributed the mean squared error root mean squared error or median absolute deviation could be used to summarize the errors statistical properties suppose we choose a measure of fit f and use crossvalidation to produce an estimate f of the expected fit ef of a model to an independent data set drawn from the same population as the training data if we imagine sampling multiple independent training sets following the same distribution the resulting values for f will vary the statistical properties of f result from this variation the crossvalidation estimator f is very nearly unbiased for ef the reason that it is slightly biased is that the training set in crossvalidation is slightly smaller than the actual data set eg for loocv the training set size is n1 when there are n observed cases in nearly all situations the effect of this bias will be conservative in that the estimated fit will be slightly biased in the direction suggesting a poorer fit in practice this bias is rarely a concern the variance of f can be large but this is considered a difficult problem computational issues most forms of crossvalidation are straightforward to implement as long as an implementation of the prediction method being studied is available in particular the prediction method can be a black box there is no need to have access to the internals of its implementation if the prediction method is expensive to train crossvalidation can be very slow since the training must be carried out repeatedly in some cases such as least squares and kernel regression crossvalidation can be sped up significantly by precomputing certain values that are needed repeatedly in the training or by using fast updating rules such as the shermanmorrison formula however one must be careful to preserve the total blinding of the validation set from the training procedure otherwise bias may result an extreme example of accelerating crossvalidation occurs in linear regression where the results of crossvalidation have a closedform expression known as the prediction residual error sum of squares press limitations and misuse crossvalidation only yields meaningful results if the validation set and training set are drawn from the same population and only if human biases are controlled in many applications of predictive modeling the structure of the system being studied evolves over time ie it is nonstationary both of these can introduce systematic differences between the training and validation sets for example if a model for predicting stock values is trained on data for a certain fiveyear period it is unrealistic to treat the subsequent fiveyear period as a draw from the same population as another example suppose a model is developed to predict an individuals risk for being diagnosed with a particular disease within the next year if the model is trained using data from a study involving only a specific population group eg young people or males but is then applied to the general population the crossvalidation results from the training set could differ greatly from the actual predictive performance in many applications models also may be incorrectly specified and vary as a function of modeler biases andor arbitrary choices when this occurs there may be an illusion that the system changes in external samples whereas the reason is that the model has missed a critical predictor andor included a confounded predictor new evidence is that crossvalidation by itself is not very predictive of external validity whereas a form of experimental validation known as swap sampling that does control for human bias can be much more predictive of external validity as defined by this large maqcii study across 30000 models swap sampling incorporates crossvalidation in the sense that predictions are tested across independent training and validation samples yet models are also developed across these independent samples and by modelers who are blinded to one another when there is a mismatch in these models developed across these swapped training and validation samples as happens quite frequently maqcii shows that this will be much more predictive of poor external predictive validity than traditional crossvalidation the reason for the success of the swapped sampling is a builtin control for human biases in model building in addition to placing too much faith in predictions that may vary across modelers and lead to poor external validity due to these confounding modeler effects these are some other ways that crossvalidation can be misused by performing an initial analysis to identify the most informative features using the entire data set if feature selection or model tuning is required by the modeling procedure this must be repeated on every training set otherwise predictions will certainly be upwardly biased by allowing some of the training data to also be included in the test set this can happen due to twinning in the data set whereby some exactly identical or nearly identical samples are present in the data set note that to some extent twinning always takes place even in perfectly independent training and validation samples this is because some of the training sample observations will have nearly identical values of predictors as validation sample observations and some of these will correlate with a target at better than chance levels in the same direction in both training and validation when they are actually driven by confounded predictors with poor external validity if such a crossvalidated model is selected from a kfold set human confirmation bias will be at work and determine that such a model has been validated this is why traditional crossvalidation needs to be supplemented with controls for human bias and confounded model specification like swap sampling and prospective studies cross validation for timeseries models since the order of the data is important crossvalidation might be problematic for timeseries models a more appropriate approach might be to use forward chaining applications crossvalidation can be used to compare the performances of different predictive modeling procedures for example suppose we are interested in optical character recognition and we are considering using either support vector machines svm or k nearest neighbors knn to predict the true character from an image of a handwritten character using crossvalidation we could objectively compare these two methods in terms of their respective fractions of misclassified characters if we simply compared the methods based on their insample error rates the knn method would likely appear to perform better since it is more flexible and hence more prone to overfitting compared to the svm method crossvalidation can also be used in variable selection suppose we are using the expression levels of 20 proteins to predict whether a cancer patient will respond to a drug a practical goal would be to determine which subset of the 20 features should be used to produce the best predictive model for most modeling procedures if we compare feature subsets using the insample error rates the best performance will occur when all 20 features are used however under crossvalidation the model with the best fit will generally include only a subset of the features that are deemed truly informative a recent development in medical statistics is its use in metaanalysis it forms the basis of the validation statistic vn which is used to test the statistical validity of metaanalysis summary estimates wikimedia commons has media related to crossvalidation statistics boosting machine learning bootstrap aggregating bagging bootstrapping statistics model selection resampling statistics stability learning theory validity statistics notes and references vtestatistics outline index descriptive statisticscontinuous datacenter mean arithmetic geometric harmonic median mode dispersion variance standard deviation coefficient of variation percentile range interquartile range shape central limit theorem moments skewness kurtosis lmoments count data index of dispersion summary tables grouped data frequency distribution contingency table dependence pearson productmoment correlation rank correlation spearmans rho kendalls tau partial correlation scatter plot graphics bar chart biplot box plot control chart correlogram fan chart forest plot histogram pie chart qq plot run chart scatter plot stemandleaf display radar chart data collectionstudy design population statistic effect size statistical power sample size determination missing data survey methodology sampling stratified cluster standard error opinion poll questionnaire controlled experiments design control optimal controlled trial randomized random assignment replication blocking interaction factorial experiment uncontrolled studies observational study natural experiment quasiexperiment statistical inferencestatistical theory population statistic probability distribution sampling distribution order statistic empirical distribution density estimation statistical model lp space parameter location scale shape parametric family likelihoodmonotone locationscale family exponential family completeness sufficiency statistical functional bootstrap u v optimal decision loss function efficiency statistical distance divergence asymptotics robustness frequentist inferencepoint estimation estimating equations maximum likelihood method of moments mestimator minimum distance unbiased estimators meanunbiased minimumvariance raoblackwellization lehmannscheff theorem median unbiased plugin interval estimation confidence interval pivot likelihood interval prediction interval tolerance interval resampling bootstrap jackknife testing hypotheses 1 2tails power uniformly most powerful test permutation test randomization test multiple comparisons parametric tests likelihoodratio wald score specific tests ztest normal students ttest ftest goodness of fit chisquared gtest kolmogorovsmirnov andersondarling lilliefors jarquebera normality shapirowilk likelihoodratio test model selection cross validation aic bic rank statistics sign sample median signed rank wilcoxon hodgeslehmann estimator rank sum mannwhitney nonparametric anova 1way kruskalwallis 2way friedman ordered alternative jonckheereterpstra bayesian inference bayesian probability prior posterior credible interval bayes factor bayesian estimator maximum posterior estimator correlationregression analysiscorrelation pearson productmoment partial correlation confounding variable coefficient of determination regression analysis errors and residuals regression model validation mixed effects models simultaneous equations models multivariate adaptive regression splines mars linear regression simple linear regression ordinary least squares general linear model bayesian regression nonstandard predictors nonlinear regression nonparametric semiparametric isotonic robust heteroscedasticity homoscedasticity generalized linear model exponential families logistic bernoulli binomial poisson regressions partition of variance analysis of variance anova anova analysis of covariance multivariate anova degrees of freedom categorical multivariate timeseries survival analysiscategorical cohens kappa contingency table graphical model loglinear model mcnemars test multivariate regression manova principal components canonical correlation discriminant analysis cluster analysis classification structural equation model factor analysis multivariate distributions elliptical distributions normal timeseriesgeneral decomposition trend stationarity seasonal adjustment exponential smoothing cointegration structural break granger causality specific tests dickeyfuller johansen qstatistic ljungbox durbinwatson breuschgodfrey time domain autocorrelation acf partial pacf crosscorrelation xcf arma model arima model boxjenkins autoregressive conditional heteroskedasticity arch vector autoregression var frequency domain spectral density estimation fourier analysis wavelet whittle likelihood survivalsurvival function kaplanmeier estimator product limit proportional hazards models accelerated failure time aft model first hitting time hazard function nelsonaalen estimator test logrank test applicationsbiostatistics bioinformatics clinical trials studies epidemiology medical statistics engineering statistics chemometrics methods engineering probabilistic design process quality control reliability system identification social statistics actuarial science census crime statistics demography econometrics national accounts official statistics population statistics psychometrics spatial statistics cartography environmental statistics geographic information system geostatistics kriging category portal commons wikiproject 