Boosting (Ensemble Methods)

Boosting is an ensemble strategy that builds models one at a time, with each new model trained to fix the errors left by the ensemble so far. Unlike bagging (used in random forests), which trains models independently in parallel, boosting is inherently sequential — the order matters because each model learns from the failures of its predecessors.

The core insight is that many weak learners — models only slightly better than random guessing — can be combined into a single strong learner. Each iteration focuses the learning on the hardest examples: the data points that previous models got wrong receive higher weight, forcing the next model to pay more attention to them. The final prediction is a weighted combination of all the individual models.

Boosting has produced some of the most successful algorithms in applied machine learning. AdaBoost (1997) was the first widely adopted boosting algorithm and remains a useful teaching example. Gradient Boosted Decision Trees generalized the approach by framing boosting as gradient descent in function space. XGBoost (2016) added regularization and engineering optimizations that made it the dominant algorithm in Kaggle competitions for years. LightGBM (2017) introduced leaf-wise tree growth and histogram-based splitting for dramatically faster training on large datasets. CatBoost (2018) added native handling of categorical features.

Boosting methods consistently outperform other approaches on tabular data and remain the go-to choice for structured data problems in production — fraud detection, credit scoring, recommendation ranking, and ad click prediction all rely heavily on boosted trees.