TB: Bias-Variance Tradeoff

In statistical modeling and machine learning: bias is systematic error from wrong assumptions (underfitting); variance is error from sensitivity to noise in training data (overfitting). Reducing one typically increases the other. The tradeoff defines the fundamental challenge of generalization: a model must be complex enough to capture signal but simple enough not to memorize noise.

The bias-variance tradeoff is the mathematical formalization of the precision vs. accuracy distinction, and generalizes far beyond machine learning. Any model of the world faces the same tradeoff: a simple model has high bias (wrong in a systematic direction) but low variance (consistent); a complex model has low bias (flexible) but high variance (sensitive to which data you happened to see). The practical lessons: (1) more data reduces variance without increasing bias, (2) regularization manages the tradeoff, (3) ensemble methods (averaging many models) reduce variance. The deep insight is that there is no free parameter — every modeling choice trades one kind of error for another.

• No reading examples yet — will populate as books are ingested.

• Local vs Global Optima — overfitting is a form of local optimization on the training set
• Survivorship Bias — in model evaluation, only testing on seen data is a form of survivorship bias
• Goodhart's Law — a model that is evaluated on training data has made training accuracy the target; it will overfit to it