Asymptotic bias of the plug-in Shannon entropy estimator under a regularly varying occupancy model

Abstract

Estimating the Shannon entropy of discrete distributions with countably infinite support is a challenging problem. In this paper, we investigate the bias of the plug-in estimator $\hat{H}_n$ for the Shannon entropy $H(\boldsymbol{p})$ under an occupancy model whose frequency sequence exhibits regular variation with tail index $\alpha\in(0,1)$. Using Poissonization and the theory of regular variation, we establish the asymptotic relation $|\mathsf{E}[\hat H_n] - H(\boldsymbol{p})| \sim n^{\alpha-1}L(n)C_\alpha$, where $L$ is a slowly varying function and $C_\alpha$ is an explicit constant depending only on $\alpha$ that admits an integral representation. Our result shows that the asymptotic behavior of the bias of the plug-in estimator under power-law frequency distributions is determined by the tail behavior of the underlying distribution.