Learning Curve Theory

In machine learning, a learning curve (or training curve) plots the optimal value of a model’s loss function for a training set against this loss function evaluated on a validation data set with the same parameters as produced the optimal function. It is a tool to find out how much a machine model benefits from adding more training data and whether the estimator suffers more from a variance error or a bias error. If both the validation score and the training score converge to a value that is too low with increasing size of the training set, it will not benefit much from more training data.

The machine learning curve is useful for many purposes, including comparing different algorithms, choosing model parameters during design, adjusting optimization to improve convergence, and determining the amount of data used for training.

In the machine learning domain, there are two implications of learning curves differing in the \(x\)-axis of the curves, with the experience of the model graphed either as the number of training examples used for learning or the number of iterations used in training the model.

Mathematically, a learning curve can be interpreted from the below two types of graphs:

1. Training and validation loss as a function of the number of training examples.

2. Training and validation loss as a function of the number of iterations.

We will make this idea concrete in the following sections.

Table of Contents

• Concept
• Implementation
• Application: Overfitting and Underfitting
• Plotting Learning Curves and Checking Models’ Scalability

References and Further Readings

One can refer to the section on learning curve theory in Bias-Variance Tradeoff for an overview of the concept of learning curve theory.

• Learning Curve Theory by Marcus Hutter (DeepMind)

• Wikipedia: Learning Curve

• scikit-learn’s learning curve

• learning-curve-diagnostics

• machine learning master

• Ritchie Ng: Applying Machine Learning

• Ritchie Ng: Learning Curve