Support Thresholds, Not Algorithms, Limit Rare-Association Recovery in Co-Purchase Networks
TL;DR
Five methods for co-purchase edge filtration on two grocery datasets are compared, including Apriori, Apriori + lift post-filtering, top-K ranking based on lift, and two methods based on networks, noise-corrected (NC) and disparity filter (DF).
Abstract
The support threshold of the Apriori algorithm involves a trade-off in conducting market basket analysis: the associations that occur frequently are noted with high threshold; however, the low ones lead to generating the large amount of rules. The paper compares five methods for co-purchase edge filtration on two grocery datasets: i.e., Instacart (3.2 million baskets) and Dunnhumby (208 thousand baskets), including Apriori, Apriori + lift post-filtering, top-K ranking based on lift, and two methods based on networks, noise-corrected (NC) and disparity filter (DF). The top-K method ensures the maximum average lift, while the NC achieves similar lift level by means of a single value of the significance parameter (α). These two methods recover substantially more rare high-lift associations than Apriori (80–100% against 22–28%). NC and top-K select meaningfully different edges (18–29% non-overlapping): NC retains statistically validated pairs, while top-K retains rare pairs with high lift but low statistical significance. A rolling-origin holdout evaluation shows that top-K edges recur at higher rates at every split, but NC edges are ∼12 pp more likely to remain statistically significant in the held-out network.
