Adapting to dissimilar tasks for continual learning via gradient norm regularisation

An essential challenge of continual learning(CL) is that the knowledge gained from old tasks might be erased as new tasks are learnt, which is called “catastrophic forgetting” (CF). Most CL methods assume tasks are drawn independently and identically distributed (i.i.d.); however, they ignore that as the number of tasks grows, some tasks inevitably share similarities, which could be leveraged for knowledge protection and transfer. This paper aims to enhance the adaptation of continual learning by leveraging task similarity. Specifically, we define task similarity as the change in empirical loss before training and show how it correlates with CF. As task similarity decreases, gradient norms increase, causing more severe forgetting. Building on this insight, we propose incorporating a Gradient Norm Regularisation(GNR) approach into the CL process. During training on a new task, we slow down the update magnitude for parameters deemed dissimilar. Meanwhile, gradient directions are adjusted to improve adaptability. Once training is complete, dissimilar parameters are consolidated to prevent them from being easily altered in future tasks. Experiments on four benchmark datasets show that our method outperforms other regularisation techniques and nearly matches the performance of multi-task learning. By plugging into existing regularisation and replay methods, GNR enhances the adaptation of continual learning. These results confirm that gradient norms can serve as an effective supervisory signal to balance knowledge protection and transfer. An extended experiment on a real-world pest classification dataset validates GNR’s generalisability and the soundness of the task-similarity assumption in practical applications. We make the code of GNR publicly available at https://github.com/wang-xulong/GNR.git.