Publications

Publications in reversed chronological order. Please check my Google Scholar for an up-to-date list.

2025

  1. How language models learn facts? Dynamics, curricula and hallucinations
    Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, and Soham De
    arXiv preprint arXiv:2503.21676, 2025
    PDF

2024

  1. Recurrent neural networks: vanishing and exploding gradients are not the end of the story
    Nicolas Zucchet, and Antonio Orvieto
    Advances in Neural Information Processing Systems, 2024
    PDF Thread Code Poster

2023

  1. Gated recurrent neural networks discover attention
    Nicolas Zucchet*, Seijin Kobayashi*, Yassir Akram*, Johannes Oswald, Maxime Larcher, Angelika Steger, and João Sacramento
    arXiv preprint arXiv:2309.01775, 2023
    PDF Thread
  2. Uncovering mesa-optimization algorithms in transformers
    Johannes Oswald*, Eyvind Niklasson*, Maximilian Schlegel*, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, and João Sacramento
    arXiv preprint arXiv:2309.05858, 2023
    PDF Thread
  3. Online learning of long-range dependencies
    Nicolas Zucchet*, Robert Meier*, Simon Schug*, Asier Mujika, and João Sacramento
    In Advances in Neural Information Processing Systems, 2023
    PDF Thread Code Poster

2022

  1. The least-control principle for local learning at equilibrium
    Alexander Meulemans*, Nicolas Zucchet*, Seijin Kobayashi*, Johannes Oswald, and João Sacramento
    In Advances in Neural Information Processing Systems, 2022
    Awarded PDF Thread Code Poster
  2. Random initialisations performing above chance and how to find them
    Frederik Benzing, Simon Schug, Robert Meier, Johannes Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger
    In Annual Workshop on Optimization for Machine Learning, 2022
    PDF Thread
  3. A contrastive rule for meta-learning
    Nicolas Zucchet*, Simon Schug*, Johannes Oswald*, Dominic Zhao, and João Sacramento
    In Advances in Neural Information Processing Systems, 2022
    PDF Thread Poster
  4. Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
    Nicolas Zucchet, and João Sacramento
    Neural Computation, 2022
    PDF

2021

  1. Learning where to learn: Gradient sparsity in meta and continual learning
    Johannes Oswald*, Dominic Zhao*, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento
    In Advances in Neural Information Processing Systems, 2021
    PDF