We introduced polynomial regression as a way of finding a curve that best fits the training examples. We combined it with logistic regression, in the Python and Julia programs, to create a curved boundary between examples in the class and those outside the class.
In practice, we are given a data set and asked to find a model that fits it. We detect overfitting by separating a testing set of data apart from the training set. During gradient descent, only the training set is used in the cost function. After the have been found that minimize the cost on the training set, we check that model’s cost on the testing set. If the testing set cost is significantly higher than the training set cost, then we know overfitting has occurred.
One way to avoid overfitting is to create a preference for simpler models, and one way to simplify a model is to eliminate the contribution of certain features. However, we may not know in advance which features are unimportant. Furthermore, it may be that their contribution should be diminished, but not eliminated. Therefore, instead of taking values out of the feature vector entirely, we try reducing . We “penalize” models having large values by adding a term to the logistic regression cost function:
or the linear regression cost function:
In the general case, where are are polynomial coefficients, we say that the resulting is made sparse because it contains more values that are zero (or close to zero), thereby eliminating (or diminishing) the contribution of a feature to a model. is a parameter that influences how simple the resulting model should become.