Feature Engineering & Regularization

Feature Engineering

Regularization

Overfitting

• Definition: The problem of overfitting is when the model captures the noise in the training data instead of the underlying structure.

Regularization

• Given objective function $J(\theta)$.
• Goal is to find: $\hat{\theta} = \argmin_{\theta} J(\theta) + \lambda r(\theta)$.
• Key idea: Define regularizer $r(\theta)$ s.t. we tradeoff between fitting the data and keeping the model simple.
• Choose form of $r(\theta)$ based on the model complexity.
  • Example: $q$-norm

$$\|\boldsymbol{\theta}\|_q = \left( \sum_{m=1}^M |\theta_m|^q\right)^{\frac{1}{q}}$$

评论