Logo image
Back
Sparsity Aware Learning in Feedback-Driven Differential Recurrent Neural Networks
Conference proceeding   Peer reviewed

Sparsity Aware Learning in Feedback-Driven Differential Recurrent Neural Networks

Ankita Paul and Anup Das
ARTIFICIAL NEURAL NETWORKS AND MACHINE LEARNING-ICANN 2024, PT IV, v 15019, pp 47-57
01 Jan 2024
DOI: https://doi.org/10.1007/978-3-031-72341-4_4

Abstract

Computer Science, Artificial Intelligence Computer Science, Interdisciplinary Applications Computer Science, Theory & Methods Science & Technology Computer Science Robotics Technology
The avenue of training differential recurrent neural networks (d-RNNs) emerges as a promising route to address spatio-temporally evolving systems. The effective learning of variable information gain makes training d-RNNs important for their inherent derivative of states property. In addition to training readout weights, the optimization of the intrinsic recurrent connection of the d-RNNs prove significant for performance enhancement. We introduce sparsity aware learning for feedback-driven differential RNNs, tailored to adapt neuron-specific learnable thresholds. This enables neurons with lower sparsity thresholds to play a more significant role in decision-making processes, while simultaneously dampening the influence of neurons with higher thresholds. This learning paradigm is in addition to optimizing the recurrent connectivity matrix of the d-RNN for mastering tasks demanding complex spatio-temporal input-output mappings. Our learning approach yields networks capable of accomplishing classification and sequential learning tasks with fewer neurons while exhibiting heightened performance compared to existing differential recurrent network training least-squares methods. Sparse d-RNN improves on spatio-temporal learning tasks by a cumulative error rate reduction of 20 % in mean squared error for dynamic system mimicking tasks and 2.97 % increase in test accuracy for classification tasks compared to existing target based learning of recurrence in feedback driven d-RNNs.

Metrics

1 Record Views

Details

Logo image