Despite their dominance in vision and language, deep neural networks often underperform relative to tree-based models on tabular data. To bridge this gap, we incorporate five key inductive biases into deep learning: robustness to irrelevant features, axis alignment, localized irregularities, feature heterogeneity, and training stability. We propose \emph{LassoFlexNet}, an architecture that evaluates the linear and nonlinear marginal contribution of each input via Per-Feature Embeddings, and sparsely selects relevant variables using a Tied Group Lasso mechanism. Because these components introduce optimization challenges that destabilize standard proximal methods, we develop a \emph{Sequential Hierarchical Proximal Adaptive Gradient optimizer with exponential moving averages (EMA)} to ensure stable convergence. Across 52 datasets from three benchmarks, LassoFlexNet matches or outperforms leading tree-based models, achieving up to a 10\% relative gain, while maintaining Lasso-like interpretability. We substantiate these empirical results with ablation studies and theoretical proofs confirming the architecture’s enhanced expressivity and structural breaking of undesired rotational invariance.
Bibtex
@misc{lui2026lassoflexnetflexibleneuralarchitecture,
title={LassoFlexNet: Flexible Neural Architecture for Tabular Data},
author={Kry Yik Chau Lui and Cheng Chi and Kishore Basu and Yanshuai Cao},
year={2026},
eprint={2603.20631},
archivePrefix={arXiv},
primaryClass={stat.ML},
url={https://arxiv.org/abs/2603.20631},
}
Related Research
-
Can LLMs Access their World Knowledge for Event Prediction?
Can LLMs Access their World Knowledge for Event Prediction?
Z. Wu, Z. Zhang, H. Hajimirsadeghi, N. Dvornik, and A. Pashevich. EMNLP
Publications
-
Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits
Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits
A. Bailey, X. Zhu, R. Aoki, Y. Cao, and K. Wilson. TMLR
Publications
-
Exploring Model Invariance with Discrete Search for Ultra-Low-Bit Quantization
Exploring Model Invariance with Discrete Search for Ultra-Low-Bit Quantization
Y. Wen, Y. Cao, and L. Mou. CIKM
Publications