Training and testing

Training is the process of starting with a system that has default or random settings and gradually improving it.

Testing is used afterward to estimate how well the trained system performs on new, unseen data.

Training

In supervised learning:

• each training sample has a manually assigned label
• all labeled samples together form the training set

Training process:

• present each training sample to the classifier one at a time
• the classifier uses the sample's features to predict a class
• if the prediction is correct: the classifier moves to the next sample
• if the prediction is wrong: the classifier compares its output with the correct label and updates its internal parameters to improve future predictions

As this predict–compare–update loop runs, the classifier's internal variables are nudged toward values that improve its prediction accuracy.

Completing one full pass through the entire training set is called an epoch, and training usually involves many epochs.

Training continues as long as the system is still learning and improving its performance on the training data.

After training, the next step is to evaluate the classifier's accuracy.

Testing the performance

Training

Systems are trained on labeled training data, but strong performance on this data does not guarantee good real-world performance due to overfitting.

There is no formula that can guarantee how well a trained model will perform in the real world.

To estimate real-world performance, we must test systems through experiments on data that goes beyond the training set.

Test data

To estimate real-world performance, we must use unseen data called the test set (or test data).

A model is trained using training data, then evaluated once on the test data to estimate real-world performance.

Guidelines:

• poor performance on test data indicates a need to gather more data or retrain from scratch
• test data must never be used for learning:
  • the system can too easily rely on specific, non-general patterns (idiosyncrasies) in the test data
  • accidentally learning from test data is called data leakage (or data contamination) and must be avoided through good data hygiene

How to avoid leakage:

• keep test data separate from training
• use it only after training is complete

Test data is created by splitting the original dataset, commonly around 75% for training and 25% for testing.

Validation data

Up to this point:

• the system is trained and then evaluated using a test set
• if performance is poor, training is restarted from scratch

That strategy is slow.

We want a rough estimate of the system's performance as we go along.

To make this estimate, we split the input data into three sets:

1. 60 % training set
2. 20 % test set
3. 20 % validation set (chunk of data that's meant to be a good proxy of the real world data)

Updated workflow with validation:

1. train the system on the training set for one epoch
2. evaluate its performance on the validation set after each epoch
3. use this feedback to decide to stop training or to adjust hyperparameters (learning rate, model complexity, etc.)
4. evaluate the system once on the test set

Always reserve the test set for final evaluation to prevent overestimating the model due to subtle data contamination.

Cross-validation

Cross-validation (or rotation validation) is a technique used when datasets are small and rare (e.g., Pluto photos), so every sample is precious.

Instead of permanently splitting the dataset, the model is trained and tested multiple times on different temporary splits of the data.

The core idea is to run a loop:

• for each iteration of the loop:
  1. create a new, untrained version of the model
  2. split the dataset into a one-time training set and a one-time validation set
  3. train the model
  4. evaluate the model on the one-time validation set and record the score
• the splits are done differently each iteration, so all data is used for training at some point

After all iterations, all the recorded scores are averaged to get an overall estimate of the model's performance.

Estimates are less reliable than those from a dedicated test set, but worth it when data is scarce.

This algorithm avoids data leakage because each iteration trains a fresh model on one subset of data and evaluates it on a separate, unseen subset.

k-fold cross-validation

K-fold cross-validation is a variant of cross-validation where the data is split into k equal-sized groups, called folds.

One smaller group is allowed if the data can't be split evenly.

Each sample belongs to exactly one fold.

For example, in 5-fold cross-validation:

• the model is trained five times
• in each iteration, one fold is used for testing, and the remaining k−1 folds are used for training
• after all iterations, the performance scores are averaged to give an overall estimate

The process can be repeated or randomized for more robust results.