SmartDS: AI-Powered Data Science Workflow

Abstract

As a result of the increasing complexity of machine learning processes, there is a need for intelligent approaches which are able to automate the entire process of data science life cycle. The purpose of this paper is to describe SmartDS, an end-to-end platform for automated data science based on the integration of Explainable Artificial Intelligence (XAI), Natural Language Processing (NLP), Automated Machine Learning (AutoML), adaptive preprocessing, and hyperparameter optimization. In contrast to traditional AutoML approaches, SmartDS utilizes a NLP module which is based on BERT architecture to automatically extract the task objective, detect target variables and categorize tasks in a zeroshot way without any human intervention. Moreover, Random Forest classifiers are used in order to select the best models, apply adaptive feature engineering, conduct Random Searchbased hyperparameter optimization with 5-fold cross-validation and profile datasets. The two benchmark data sets used for testing the proposed architecture were UCI Heart Disease Data Set (303 samples, 13 attributes) and UCI Adult Income Data Set (48,842 samples, 14 attributes). Compared to Auto-Sklearn and TPOT as the baselines for AutoML algorithms, the classification accuracy results were $\mathbf{8 9. 8 \%}$ and $\mathbf{9 1. 2 \%}$, respectively. Stable performance with standard deviations of 0.9 and 1.1, respectively, was obtained via cross-validation experiments. The explainability module effectively produced LIME explanations, SHAP summary charts, and SHAP force plots, offering comprehensible insights into model predictions. According to a comparative investigation, SmartDS improves workflow automation, repeatability, and interpretability while drastically reducing human labor. The suggested framework is an intelligent and scalable solution for automated data science systems of the future.