Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention.

Bhattacharya N(1), Thomas N, Rao R, Dauparas J, Koo PK, Baker D, Song YS, Ovchinnikov S.

Author information:
(1)University of California, Berkeley, USA, nick_bhat@berkeley.edu.

Pac Symp Biocomput. 2022;27:34-45

© 2022 World Scientific
Open Access chapter published by World Scientific Publishing Company and distributed under the terms of the Creative Commons Attribution (CC BY) 4.0 License.

Abstract

The established approach to unsupervised protein contact prediction estimates coevolving positions using undirected graphical models. This approach trains a Potts model on a Multiple Sequence Alignment. Increasingly large Transformers are being pretrained on unlabeled, unaligned protein sequence databases and showing competitive performance on protein contact prediction. We argue that attention is a principled model of protein interactions, grounded in real properties of protein family data. We introduce an energy-based attention layer, factored attention, which, in a certain limit, recovers a Potts model, and use it to contrast Potts and Transformers. We show that the Transformer leverages hierarchical signal in protein family databases not captured by single-layer models. This raises the exciting possibility for the development of powerful structured models of protein family databases.