Nikolaos Al.Papadopoulos

Information-Theoretic Measures in AI: A Practical Decision Framework

A practical decision framework for four foundational measures - Entropy, KL divergence/cross-entropy, Mutual Information, and Transfer Entropy is provided, organized around three prescriptive questions for each: what question does the measure answer and in which AI context; which estimator is appropriate for the data type and dimensionality; and what is the most dangerous misuse.