ESMC Interpretability and the ESM Atlas

Event Overview

ESMC is the latest in the ESM family of protein language models, establishing a new frontier in representation learning for protein biology. Trained on billions of evolutionary sequences, it learns representations that encode biochemical, structural, and functional principles across billions of parameters. Sparse autoencoders (SAEs) make this tractable by decomposing ESMC’s internal representations into individual, human interpretable features. In our preprint, we train SAEs on ESMC and find 16,384 features that organize protein biology across every level of complexity, from primary to secondary to tertiary structure and from biochemical to catalytic motifs. 

We then combined these features with ESMFold2 to build the ESM Atlas, the largest application of AI to protein biology to date, mapping 6.8 billion protein sequences and 1.1 billion predicted structures. The Atlas organizes proteins by similarity in the model’s learned feature space rather than by sequence alone.  This makes this space searchable, enabling the discovery of connections between proteins that have significantly diverged in sequence or independently arrived at shared characteristics through different evolutionary trajectories. Through the predicted structures and biologically relevant features, the Atlas provides a mechanism for interpreting these connections biologically through the ESMC world model.

In this one-hour webinar, researchers will show what SAEs reveal about ESMC’s internal representations and how those features compose into higher level biological concepts. You’ll see how to identify the specific features responsible for a protein’s function, use those features to find functional relatives of a protein sequence — even when there are no existing database annotations — and navigate the Atlas to surface structural and evolutionary connections that a sequence-based search would miss. The session closes with a demo of the Atlas and a tutorial of how to apply the ESMC SAEs directly to your own research.  

Speakers

Biohub

Alex Derry

Alex Derry is a research scientist at Biohub, formerly Evolutionary Scale. He is interested in mechanistic interpretability and scientific discovery using biological foundation models.

biohub

Ashton Trotman-Grant

Ashton Trotman-Grant works in AI research and product at Biohub, focusing on model interpretability and deployment. He holds a Ph.D. in Developmental Immunology and co-founded Notch Therapeutics. Partnering with lab teams, he leads the experimental validation and adoption of these models within the Biohub community.