Dense Associative Memory: physical systems for novel AI architectures

Dense Associative Memories are recurrent neural networks with fixed-point attractor states that are described by an energy function. In contrast to conventional Hopfield Networks, which were popular in the 1980s, Dense Associative Memories have a very large information storage capacity, making them appealing tools for many problems in AI. In this talk, I will provide an intuitive understanding and mathematical framework for this class of models and give examples of problems in AI that can be tackled using these new ideas. Specifically, I will explore the relationship between Dense Associative Memories and transformers. I will present a neural network called the Energy Transformer, which unifies energy-based modeling, associative memories, and transformers in a single architecture. I will demonstrate how Energy Transformers can be used for challenging tasks in image processing, solve partial differential equations, and serve as computational modules for energy-based language modeling. I will also discuss an exciting possibility of mapping these models onto analog hardware accelerators, which could enable much more energy-efficient inference compared with GPUs.