LightGBM (Boosting)

LightGBM — Light Gradient Boosting Machine — was released by Microsoft Research in 2017 as a faster alternative to XGBoost. Its two main innovations are leaf-wise (best-first) tree growth and histogram-based feature bucketing, both of which reduce training time dramatically on large datasets.

Traditional gradient boosting grows trees level-wise — completing each depth level before moving to the next. LightGBM grows leaf-wise instead, always splitting the leaf with the highest loss reduction. This produces deeper, more asymmetric trees that converge faster with fewer total leaves. The risk is overfitting on small datasets, which is managed through a max-depth parameter.

Histogram-based splitting bins continuous features into discrete buckets before training, reducing the number of split candidates the algorithm needs to evaluate. This trades a small amount of precision for a large speedup and lower memory usage. On datasets with millions of rows, LightGBM can be 10 to 20x faster than XGBoost while producing models of comparable accuracy.

LightGBM also supports Gradient-based One-Side Sampling (GOSS), which keeps all instances with large gradients but randomly samples from those with small gradients, and Exclusive Feature Bundling (EFB), which bundles mutually exclusive sparse features to reduce dimensionality. These techniques make LightGBM particularly effective on high-dimensional, large-scale tabular data — the kind found in advertising, recommendation systems, and financial modeling.