An Empirical Investigation into the Failure Modes of Logistic Regression and Targeted Mitigation Strategies

Abstract

Despite the growing dominance of deep neural architectures, logistic regression continues to hold its ground as a go-to tool for binary classification — largely because of how easy it is to interpret and how cheaply it runs. That said, it is far from bulletproof. Certain data conditions can quietly erode its performance in serious ways. This paper takes a hands-on, experimentdriven approach to investigating four well-known but often under-examined failure scenarios: multicollinearity among input features, decision boundaries that curve rather than cut straight, the convergence breakdown that comes with complete separation in high-dimensional spaces, and the silent damage caused by heavily lopsided class distributions. We ran experiments on purpose-built synthetic datasets as well as two widely used real-world benchmarks from the UCI repository — the Wisconsin Breast Cancer diagnostic set and the Credit Card Fraud Detection collection — measuring outcomes through ROC-AUC and F1-score. On top of documenting where things fall apart, we also tested a suite of targeted fixes: L1 and L2 penalization, polynomial feature expansion, and the SMOTE oversampling method. The numbers paint a clear picture. Without any intervention, logistic regression barely beats a coin flip on non-linear data (ROC-AUC of 0.52) and essentially ignores the minority class under extreme skew (F1 below 0.10). But each of the tested remedies brought meaningful recovery — polynomial expansion pushed non-linear classification up to 0.97 ROC-AUC, while SMOTE lifted minority-class F1 to 0.82. We distill these findings into a straightforward decision guide that practitioners can use to quickly diagnose what is going wrong with their logistic regression model and choose the right corrective action.