Training, validation, and test sets

training settraining datatest setvalidation setdatasetHoldouttraining datasettraining examplesavailable textcalibration set
In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data.wikipedia
97 Related Articles

Machine learning

machine-learninglearningstatistical learning
In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data.
Machine learning algorithms build a mathematical model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to perform the task.

Cross-validation (statistics)

cross-validationcross validationLeave-one-out cross-validation
If the data in the test dataset has never been used in training (for example in cross-validation), the test dataset is also called a holdout dataset.
In a prediction problem, a model is usually given a dataset of known data on which training is run (training dataset), and a dataset of unknown data (or first seen data) against which the model is tested (called the validation dataset or testing set).

Overfitting

overfitover-fitover-fitted
Validation datasets can be used for regularization by early stopping: stop training when the error on the validation dataset increases, as this is a sign of overfitting to the training dataset.
For example, a model might be selected by maximizing its performance on some set of training data, and yet its suitability might be determined by its ability to perform well on unseen data; then overfitting occurs when a model begins to "memorize" training data rather than "learning" to generalize from a trend.

Statistical classification

classificationclassifierclassifiers
A training dataset is a dataset of examples used for learning, that is to fit the parameters (e.g., weights) of, for example, a classifier.
In machine learning and statistics, classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known.

Early stopping

Validation datasets can be used for regularization by early stopping: stop training when the error on the validation dataset increases, as this is a sign of overfitting to the training dataset.
These early stopping rules work by splitting the original training set into a new training set and a validation set.

Algorithm

algorithmsalgorithm designcomputer algorithm
In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data.

Data

statistical datascientific datadatum
In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data.

Mathematical model

mathematical modelingmodelmathematical models
Such algorithms work by making data-driven predictions or decisions, through building a mathematical model from input data.

Data set

datasetdatasetsdata sets
A training dataset is a dataset of examples used for learning, that is to fit the parameters (e.g., weights) of, for example, a classifier. The data used to build the final model usually comes from multiple datasets. A test dataset is a dataset that is independent of the training dataset, but that follows the same probability distribution as the training dataset.

Artificial neural network

artificial neural networksneural networksneural network
The model is initially fit on a training dataset, that is a set of examples used to fit the parameters (e.g. weights of connections between neurons in artificial neural networks) of the model. The model (e.g. a neural net or a naive Bayes classifier) is trained on the training dataset using a supervised learning method (e.g. gradient descent or stochastic gradient descent).

Naive Bayes classifier

Naive Bayesnaive Bayes classificationNaïve Bayes
The model (e.g. a neural net or a naive Bayes classifier) is trained on the training dataset using a supervised learning method (e.g. gradient descent or stochastic gradient descent).

Supervised learning

supervisedsupervised machine learningsupervised classification
The model (e.g. a neural net or a naive Bayes classifier) is trained on the training dataset using a supervised learning method (e.g. gradient descent or stochastic gradient descent).

Gradient descent

steepest descentgradient ascentgradient
The model (e.g. a neural net or a naive Bayes classifier) is trained on the training dataset using a supervised learning method (e.g. gradient descent or stochastic gradient descent).

Stochastic gradient descent

AdaGradAdaptive Moment Estimationgradient-based learning methods
The model (e.g. a neural net or a naive Bayes classifier) is trained on the training dataset using a supervised learning method (e.g. gradient descent or stochastic gradient descent).

Array data structure

arrayarraysvector
In practice, the training dataset often consist of pairs of an input vector (or scalar) and the corresponding output vector (or scalar), which is commonly denoted as the target (or label).

Feature selection

variable selectionfeaturesselecting
The model fitting can include both variable selection and parameter estimation.

Estimation theory

parameter estimationestimationestimated
The model fitting can include both variable selection and parameter estimation.

Hyperparameter (machine learning)

hyperparametershyperparameterhyper-parameter
The validation dataset provides an unbiased evaluation of a model fit on the training dataset while tuning the model's hyperparameters (e.g. the number of hidden units in a neural network ).

Regularization (mathematics)

regularizationregularizedregularize
Validation datasets can be used for regularization by early stopping: stop training when the error on the validation dataset increases, as this is a sign of overfitting to the training dataset.

Independence (probability theory)

independentstatistically independentindependence
A test dataset is a dataset that is independent of the training dataset, but that follows the same probability distribution as the training dataset.

Probability distribution

distributioncontinuous probability distributiondiscrete probability distribution
A test dataset is a dataset that is independent of the training dataset, but that follows the same probability distribution as the training dataset.

Accuracy and precision

accuracyprecisionaccurate
For example, if the most suitable classifier for the problem is sought, the training dataset is used to train the candidate algorithms, the validation dataset is used to compare their performances and decide which one to take and, finally, the test dataset is used to obtain the performance characteristics such as accuracy, sensitivity, specificity, F-measure, and so on.

Sensitivity and specificity

sensitivityspecificitysensitive
For example, if the most suitable classifier for the problem is sought, the training dataset is used to train the candidate algorithms, the validation dataset is used to compare their performances and decide which one to take and, finally, the test dataset is used to obtain the performance characteristics such as accuracy, sensitivity, specificity, F-measure, and so on.

Precision and recall

precisionrecallF-measure
For example, if the most suitable classifier for the problem is sought, the training dataset is used to train the candidate algorithms, the validation dataset is used to compare their performances and decide which one to take and, finally, the test dataset is used to obtain the performance characteristics such as accuracy, sensitivity, specificity, F-measure, and so on.

Model selection

statistical model selectionselectingchoose a model
The basic process of using a validation dataset for model selection (as part of training dataset, validation dataset, and test dataset) is: