Biological neural systems learn new behaviors while retaining earlier ones, whereas sequentially trained artificial networks often overwrite parameters and forget previously learned behaviors. We introduce the Context-modulated Synaptic-states Recurrent Neural Network (CoSyn-RNN), a new continual-learning model inspired by heterogeneous synaptic states and neuro-modulatory control. CoSyn-RNN combines sparse recurrent allocation with a discrete protection rule: after each task, recurrent neurons whose synaptic weights cross a fixed threshold form a task-specific neuron group, and their associated parameters enter a protected state during later learning. A neuron-specific gain and a task-cued mask over a fixed base readout control how recurrent activity contributes to behavior. Across various task sequences consisting of many cognitive tasks, training and validation losses and accuracies were maintained throughout learning. The protection rule produced modular, task-ordered recurrent structure compatible with forward reuse of earlier computations. Pairwise recruitment measurements further defined minimum- and maximum-cost curricula that produced markedly different final recurrent structures and different remaining capacity for future learning. Thus, CoSyn-RNN provides a biologically inspired model in which protected synaptic states, sparse allocation, and context-dependent modulation support continual learning.
Yuan Gao, S. Mihalas, Denis Turcu· bioRxiv· 0 citations
Biological vision has evolved to make efficient use of the limited information processing capability and tight energy budget of the brain by preferentially processing the most salient features of visual scenes. In contrast, modern deep vision models rely on expansive, high-dimensional representations. This may offer potential recognition gains but increases computing costs. As a consequence, the computer vision community has been exploring sparsity-enforcing techniques such as activation dropout and weight pruning. Beyond ameliorating the burden of computation, sparsity techniques such as random dropout have been shown to regularize model training, thus allowing for better generalization. Here, we pursue both dropout and weight pruning in tandem and adopt a biologically plausible saliency-informed dropout technique as an explainable alternative to the unstructured standard random dropout approach. Our approach involves a hierarchical “retinotopic" gating of convolutional feature maps, which promotes efficient deletion of “redundant" weights by iterative magnitude pruning. We explore the effectiveness of saliency-informed dropout based on different approaches to dropping based on the saliency map, dropout through all layers or only early layers, and by additionally applying dropout at inference. We compare throughout with standard random dropout. We present empirical results on regimes where such structured dynamic and static (weight) sparsities interact optimally to prune a ResNet model on the Imagenette dataset.
Shira Goldhaber-Gordon, A. Akwaboah, Aaron L. Sampson et al.· International Conference on...· 1 citation
In 1949, Donald Hebb proposed that neuronal assemblies with temporally specific patterns of activity form the building blocks of perception, cognition, and behavior. Finding the structural underpinning of such assemblies has been technically challenging due to a lack of large-scale structure-activity maps. Here, we combine in vivo optical physiology with postmortem electron microscopy (EM) in the same tissue volume. Using higher-order correlations in fluorescence traces, we extract neuronal assemblies. Physiologically, we show that these assemblies respond more reliably to repeated natural movies than size-matched control ensembles and decode such stimuli more accurately. Structurally, we find that over a quarter of the pyramidal neurons do not participate in any assembly and are significantly less integrated into the connectome than those that do. We do not observe a marked increase in the strength of monosynaptic excitatory connections between neurons sharing assembly assignment, but instead find significantly stronger indirect inhibitory connections targeting cells in other assemblies. These results show that assemblies can serve as functional units of perception and suggest they may be structurally delineated by mutual inhibition.
J. Wagner-Carena, Sai Kate, Trevor Riordan et al.· Cell Reports· 0 citations
A database of spiking activity across the mouse visual system-including the cortex, thalamus, and midbrain-while mice perform an image change detection task is presented, providing a valuable substrate for understanding sensorimotor computations in neural networks.
Corbett Bennett, Samuel D. Gale, Greggory Heller et al.· Cell· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
