AI models predict how gene expression shifts when a gene is switched off or activated in single cells. A Nature Biotechnology paper published Oct. 1, 2026, found common scores can hide these changes, making models look no better than simple baselines.
Most genes remain unchanged in any one experiment. When a benchmark averages errors across all genes, they can dominate the score, making a predictor that ignores which gene was altered look competitive.
To test whether a metric could distinguish a useful prediction from an uninformative one, the authors compared two controls. The negative control was the mean expression response across training perturbations, so it ignored which gene was changed.
The positive control, called an “interpolated duplicate,” combined measured responses from one half of cells exposed to the same perturbation with the training-response mean. It gave more weight to the measured response for genes with stronger evidence of change and more weight to the mean for other genes.
The dynamic range fraction measured the score gap between the positive and negative controls relative to the gap between the negative control and a perfect score. A low fraction meant a metric had little room to register improvement over the baseline.
The calibration analysis covered 14 datasets and 18 metrics. Standard measures—mean squared error and control-referenced Pearson correlation, which compares predicted and observed expression shifts relative to untreated cells—were often poorly calibrated. Weighted and rank-based measures, including weighted mean squared error and normalized inverse rank, more consistently registered perturbation-specific changes.
The authors then tested nine models on genetic perturbations excluded from training. Under the better-calibrated measures, the models generally outperformed three internal baselines: the average response across training perturbations, average expression in untreated control cells, and a linear model.
The journal reports that five authors are Shift Bioscience employees and that Bo Wang is Chief Artificial Intelligence Scientist at Xaira Therapeutics.
The result changes how earlier benchmark scores should be read, but it is not evidence that the models work in new biological settings. The study did not test responses in unseen cell types, donors or conditions, and the authors call for evaluation on additional combination datasets.
Figure 1: Miller et al., Nature Biotechnology, licensed under CC BY 4.0.
Sources
Nature Biotechnology paper · Author preprint (CC BY 4.0) · Reproduction code