Benchmarking Optimizers for Tabular Deep Learning: AdamW May Not Be the Best Default for MLPs

arXiv·medium signal

A systematic benchmark of multiple optimizers across numerous tabular datasets for training MLP-based models finds that the default choice of AdamW may not be optimal. Despite new optimizers showing promise in other domains (vision, NLP), optimizer selection for tabular deep learning has never been examined systematically — this paper fills that gap with practical recommendations for practitioners working with tabular data.