Deep Learning and Ensemble Methods for Forecasting Indian Large Capitalization Stock Index Prices
TL;DR
The Diebold-Mariano tests proved that Long Short-Term Memory (LSTM) predictions were more accurate compared to the individual forecasts of the ARIMA, GRU, and XGBoost models with a standard level of significance, and suggested the future incorporation of Transformer-based models to increase predictive power, like Informer or Crossformer.
Abstract
The present paper conducts an analysis of how different AI models perform in predicting the daily closing values of the National Stock Exchange (NSE) Indian Large Cap Stocks Index. The paper evaluates seven models, namely Autoregressive Integrated Moving Average (ARIMA), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Random Forest, Extreme Gradient Boosting (XGBoost), Support Vector Regression (SVR), and Prophet, based on 35-month historical Indian Large Cap Stocks data acquired from the NSE website. A strong evaluation method (walk-forward validation) is used to test the forecasting performance. The model was a deep learning model implemented using TensorFlow, and the other models were implemented using scikit-learn. Correlation (rho), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Coefficient of Determination (R²) were used to evaluate performance. The results of the Diebold-Mariano tests proved that Long Short-Term Memory (LSTM) predictions were more accurate compared to the individual forecasts of the ARIMA, GRU, and XGBoost models with a standard level of significance. The recurrent models and tree-based learners in a stacked ensemble with a linear meta-learner performed better than all individual models, reducing RMSE by 18.5% and MAE by 20.6% compared with the best single model. These changes happen because the ensemble combines the strengths of different modeling methods. It uses the ability of networks to remember what happened in the past and the ability of tree-based methods to make decisions in a more complex way. This results in accurate predictions when things are changing quickly. For people who trade and analyze the market, this shows that hybrid AI systems can be really useful for predicting what will happen in the future. If the systems are designed carefully, they can help in making decisions about managing risk and allocating resources. Besides, this work suggests the future incorporation of Transformer-based models to increase predictive power, like Informer or Crossformer.