Restricted Boltzmann Machines, Bernoulli Mixtures and Sum-Product Networks: A Matched-Capacity Comparison on Binarized Image Data

Abstract

Generative probabilistic models differ in a fundamental way that is rarely measured directly: some permit exact inference, while others are more expressive but require their likelihood to be estimated. This study compares three model families on binarized Fashion-MNIST and MNIST under identical preprocessing, identical data splits, and matched parameter counts, evaluating every model by test log-likelihood on a common scale. Sum-product networks and mixtures of Bernoullis return exact likelihoods; the likelihood of a restricted Boltzmann machine is obtained by annealed importance sampling and reported with the effective sample size of the importance weights and a convergence study. Three results follow. First, the structure of a sum-product network matters more than its size: changing only which pixels are assigned to which leaf region, at a fixed parameter count of 706,800, is worth 38.7 nats on Fashion-MNIST and 41.2 nats on MNIST, whereas multiplying the capacity of a flat mixture eightfold yields approximately 13 nats. A network whose regions are misaligned with the data performs worse than a model with no hierarchy at all. Second, once the regions are aligned, the benefit of hierarchy depends on the data: the network exceeds the best flat mixture by 17.2 nats on Fashion-MNIST but falls 2.0 nats short on MNIST, where whole-image prototypes already suffice. Third, the restricted Boltzmann machine outperforms every tractable model tested at matched capacity, leading the best of them by 11.4 nats on Fashion-MNIST and 46.1 nats on MNIST, which quantifies the cost of guaranteeing exact inference. Exact inference nonetheless carries a practical benefit: for image completion, the sum-product network computes conditional marginals exactly and improves on max-product in every configuration tested. Training a fully connected Boltzmann machine to convergence proved infeasible on the available hardware, and the computational limitations are reported quantitatively.