Skip to content
Open access

Beyond Feature Importance: Investigating Predictive Sensitivity in Machine Learning Models

Sep 2026 · International Journal For Multidisciplinary Research · Vol 8 · 0 citations · 14 references

TL;DR

It is suggested that feature importance and perturbation sensitivity capture related but distinct aspects of model behaviour, and that controlled feature perturbation analysis can provide a complementary perspective for interpreting feature relevance and evaluating the robustness of machine learning models.

Abstract

Feature importance is widely used in machine learning to assess the contribution of individual predictors to model performance and support the interpretation of model behaviour. However, it remains unclear whether features identified as highly important are also those to which a model is most sensitive when their values are disturbed. This study empirically investigates the relationship between feature importance and predictive sensitivity under controlled feature perturbations. Experiments are conducted on five classification datasets, namely Sonar, Ionosphere, Breast Cancer Wisconsin, Parkinson’s Disease, and Spambase, using Logistic Regression, Random Forest, and Gradient Boosting to represent different modelling characteristics. Permutation importance is used to estimate model-specific feature importance. Individual features in the test data are then progressively perturbed using Gaussian noise at multiple intensity levels, while the trained models remain unchanged. Predictive sensitivity is measured through the resulting degradation in F1 score, and Spearman rank correlation is used to examine the correspondence between feature importance and feature-wise sensitivity. The results show that higher feature importance is associated with greater perturbation sensitivity in several dataset and model combinations. However, the relationship is not consistent across all experimental conditions or classifiers. Both the magnitude and statistical significance of the observed associations vary across datasets, models, and perturbation intensities, with stronger associations observed at higher perturbation levels in several cases. These findings suggest that feature importance and perturbation sensitivity capture related but distinct aspects of model behaviour. Therefore, feature importance alone may not fully reflect how strongly model performance responds to disturbances in individual features. Controlled feature perturbation analysis can thus provide a complementary perspective for interpreting feature relevance and evaluating the robustness of machine learning models.

Read PDF

Similar papers

Open access Oct 2026

The Impact of Feature Selection on Machine Learning Model Performance

Feature selection can reduce input dimensionality and improve interpretability, but its effect on predictive performance depends on the data and learning algorithm. This study experimentally compares three classifiers Logistic Regression, Support Vector Machine (SVM) with a radial basis function kernel, and Random Fore...

J. Miharja · 0 citations
Open access Oct 2026

Equally accurate, differently explained: Stability of permutation feature importance across machine learning classifiers on clinical tabular data

Feature-importance explanations are increasingly used to justify machine learning (ML) models in clinical decision support, yet they are usually reported from a single fitted model without evidence that they would recur if the model were refitted. This study quantifies the stability of permutation feature importance ac...

Surjeet Dalal, Mohd Akbar · 0 citations
Open access Sep 2026

Explainable Machine Learning for Predictive Analytics: Examining the Influence of Feature Optimization and Model Interpretability

This study examined the application of explainable machine learning (XAI) in predictive analytics by investigating the influence of feature optimization and model interpretability on machine learning performance and transparency. A quantitative experimental and comparative research design was employed using a structure...

Noor Ul Ain Saba, Tasawar Nawaz, Muhammad Farman et al. · 0 citations
Open access Oct 2026

Interpretable Random Forest-Based Heart Disease Prediction with Clinical Feature Importance Analysis

Heart disease prediction from heterogeneous clinical attributes requires ML models capable of capturing nonlinear relationships among physiological and diagnostic variables. This research presents a Random Forest-based methodology for classification of binary heart disease data utilizing the publicly available Heart Di...

S. K., Mahavishnu, B. P. et al. · 0 citations
Open access Oct 2026

Empirical Evaluation of Feature Selection Strategies for Interpretable and Robust Healthcare Machine Learning

Feature Section (FS) is a fundamental step in developing accurate, robust, and interpretable Machine Learning (ML) models, especially when working with high-dimensional structured datasets commonly found in healthcare, finance, and other data-intensive domains. Selecting the most relevant subset of features not only im...

Choo-Yee Ting, M. Nishan, C. Pee et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.