fig3

Figure 3. Evaluation-dependent performance of AI virtual-cell models and the evidence required for stronger biological claims. (A) In the original GEARS evaluation, the MSE for the top 20 differentially expressed genes was reduced by 54.2% and 17.4% relative to the next-best baseline on the Norman 2019 and Replogle 2020 K562 double-perturbation datasets, respectively. Baseline MSE was normalized to 100 for visualization. Adapted from reference^[40] under the terms of the CC BY 4.0 License; (B) AvgBIO of cell embeddings across five datasets. Darker cells indicate higher scores, outlined cells mark the best-performing method within each dataset, and the bottom row shows the median. Harmony and scVI were fitted to each dataset, whereas scGPT-human and Geneformer were evaluated as frozen embeddings. Adapted from reference^[42] under the terms of the CC BY 4.0 License; (C) Mean L2 prediction errors in an independent double-perturbation benchmark. The additive baseline achieved the lowest mean error, compared with GEARS, scGPT, scFoundation, and the no-change baseline. Adapted from reference^[11] under the terms of the CC BY 4.0 License; (D) Conceptual hierarchy linking increasingly strong biological claims to the corresponding evaluation evidence. AI: Artificial intelligence; GRARS: graph-enhanced gene activation and repression simulator; MSE: mean squared error; AvgBIO: average biological conservation; scVI: single-cell variational inference; scGPT: single-cell generative pretrained transformer.