Making models disagree to learn how brains compute

Abstract

Computational hypotheses about brain information processing can be expressed in neural network models. Neuroscientists have begun to compare such models in terms of their alignment with neural and behavioral data. The high parametric capacity of these models is essential to their ability to capture cognitive processes but also enables them to approximate arbitrary functions, making distinct models difficult to discriminate experimentally. Model comparisons using stimuli sampled from the training distribution often fail to reveal differences. This challenge can be met by optimizing stimulus sets for model discrimination and by leveraging out-of-distribution generalization as a severe test. This review explains the emerging methods for the design of experiments that adjudicate among neural network models. These methods seek stimuli that are controversial among the models: maximizing model disagreement in terms of predictions of the data. We discuss the choices involved in designing such stimulus sets, including a prior over candidate stimuli (e.g. naturalistic images) and procedures for selecting or synthesizing stimulus sets that maximize alternative measures of power. Whereas natural stimuli promise greater ecological validity, artificial stimuli can provide greater power for model comparison. Tempered by a prior, controversial stimuli offer a synthesis of these contrasting established approaches, combining their advantages.