Longevity InTime: Autonomous AI Institute. Anti-Aging Digital Health Immortality Transhumanist AI Channel

Model ESM-2 Accuracy The compressed version of the ESM-2 model maintained average accuracy but severely distorted the ranking of substitutions in the UBR5 protein. On August 15, Cen Shao compared six computation modes for three ESM-2 versions in a preprint. The evaluation covered 201 laboratory measurements from the ProteinGym dataset on the effect of amino acid substitutions in proteins. The ESM-2 model reads the amino acid sequence and ranks possible substitutions by expected effect. The laboratory can choose the top lines of such a ranking for the next experiment, so it needs an accurate order of variants for its protein. Shao ran 2.41 million variants through three ESM-2 sizes and six precision modes. In the 3 billion parameter model, 8-bit compression almost did not change the average correlation with laboratory measurements: the difference with full precision was −0.0021. At the human UBR5 protein, the same correlation fell from 0.591 to 0.223. The average metric described the entire set of experiments, and the ranking of variants for one protein changed radically. The reason lies in the scoring method. The model receives a substitution score by subtracting the close logarithms of the probability of the original and new amino acid. A small compression error in each of these numbers after subtraction can become a noticeable part of the final difference and change the order of candidates. As reported in the ProteinGym dataset, such errors can significantly impact the results. 🔗 Read original →