On the Impact of Data Sparsification on Model Selection and Learning Performance
Abstract
Feature-wise sparsification expands each original feature into sparse interval-specific components, thereby changing the dimension and dependence structure of the design matrix. We investigate how this transformation affects model-selection outcomes and learning performance by comparing original and sparsified representations using Chinese housing-price data, controlled synthetic examples, an extended Monte Carlo study, and benchmark clustering datasets. The synthetic examples illustrate that sparsification can make the coefficient-entry paths of two Lasso implementations (lars and glmnet) and the related least-angle regression (LAR) procedure more similar. The controlled Monte Carlo study shows that this effect is conditional and metric-dependent. The clearest increases in parent-feature entry-order agreement occur in the evaluated high-dimensional correlated setting and remain positive under a denser and wider glmnet path, whereas some smaller entry-order effects in other settings change under that sensitivity specification. Active-set agreement is more stable to the path specification but is not uniformly improved. Under component-wise Lasso, support-recovery $F_{1}$ decreases and test RMSE increases across all 64 evaluated simulation configurations under both path specifications. A parent-group Lasso sensitivity analysis attenuates these differences but preserves the $F_{1}$ direction in all 64 configurations and the RMSE direction in 60 of 64. In clustering, sparsification improves some metrics for selected datasets, but the effects remain dataset- and metric-dependent. A secondary runtime comparison shows no uniform computational advantage. Overall, feature-wise sparsification should be viewed not as a universally beneficial preprocessing step, but as a change in feature organization whose effects depend on sample size, feature dimension, correlation structure, and representation design.