AI Sequence Models in Regulatory Genomics: Hype, Reality and Where They Help

Sequence-to-function models have become genuinely useful for ranking and largely unreliable for explaining. That asymmetry — not the headline accuracy numbers — is what should shape how a lab uses them.

Three claims, each defended below:

1. Model performance is dominated by the label it was trained on, not by architecture.
2. Benchmark leakage inflates reported accuracy in ways that are hard to detect from a paper's methods section.
3. There are four distinct tasks in regulatory genomics, and models are useful for two of them.

A model trained to predict chromatin accessibility learns a different object from one trained to predict expression, and both differ from one trained to predict measured regulatory activity (as in an MPRA).

The practical consequence: "the model predicts this enhancer is active" is a statement about the training label, and different models will disagree because they are answering different questions with the same vocabulary.

A useful discipline when reading a paper: find the label before the architecture. A transformer trained on accessibility is not a better version of a CNN trained on MPRA activity; it is a different instrument.

Four leakage patterns recur, and each inflates reported performance:

• • Cell-type leakage. Random splits place the same cell type in training and test. Performance then reflects memorising cell-type-specific patterns rather than generalising to a new context.
• • Genomic-neighbour leakage. Nearby genomic regions share sequence and chromatin features; random splits place neighbours on both sides.
• • Variant leakage. Variants in strong linkage disequilibrium appear in both sets, so the model can recognise the haplotype rather than the mechanism.
• • Label leakage through features. Models that take accessibility as input while predicting expression are partly predicting from a correlated measurement.

Chromosome-held-out splits, cell-type-held-out splits and variant-disjoint splits are the honest tests. When a paper reports only random splits, the accuracy number should be read as an upper bound with unknown slack.

A second-order problem: benchmarks optimise for correlation across many loci, while a lab cares about the top few hundred predictions. Models that are excellent on average can be ordinary at the top of the list — the region that matters.

The fourth row is the one to hold firm on. Saliency maps and in silico mutagenesis describe model behaviour, not biology. They generate hypotheses about which bases matter, and those hypotheses are testable — by editing. A model's attribution is an input to an experiment, not a substitute for one.

The second row deserves nuance: variant effect prediction is genuinely useful for triage when the variant lies in a context represented in training data, and degrades sharply outside it. Rare cell types, poorly sampled tissues, and structural variants are all outside.

1. What is the label, exactly, and in which cells was it measured?
2. What was held out — chromosome, cell type, or variant? If the answer is "random", discount accordingly.
3. What effect size can it resolve? A model with good correlation may not distinguish a 20% change from a 30% change, which is the range most regulatory variants operate in.

If a model survives those three questions, it can legitimately shape which experiments you run. If it fails any of them, it is a source of hypotheses, and hypotheses should be cheap.

Prediction: within a few years, the field's most-cited sequence models will be judged on held-out cell types rather than held-out loci, and papers reporting only random splits will stop being competitive.

Refuted if: the community converges instead on larger multi-cell-type training sets with random splits, accepting that the model is a cell-type interpolator rather than an extrapolator — a legitimate but different product.

The distinction matters commercially as much as scientifically: an interpolator is useful inside a well-sampled tissue panel, and unhelpful for the rare disease context where nobody has data.

Models belong at the front of a project: ranking candidates, triaging variants, proposing motifs. They do not belong at the back, where a claim about mechanism is being made. The labs that use them well are the ones that can state, in one sentence, which decision the model informed and which experiment settled it.

Contact EDITGENE if you want candidate sets turned into perturbation experiments rather than further predictions.

This is an argument about how to interpret model outputs, not an evaluation of any specific model or benchmark. Reported accuracies depend on split strategy and training data, and should be read with both in mind.

Model predictions are hypotheses. Causal claims about regulatory elements require perturbation with appropriate controls.

• • Avsec Ž, et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods. 2021;18:1196–1203.
• • Zhou J, Troyanskaya OG. Predicting effects of noncoding variants with deep learning-based sequence model. Nature Methods. 2015;12:931–934.
• • Tewhey R, et al. Direct identification of hundreds of expression-modulating variants using a multiplexed reporter assay. Cell. 2016;165:1519–1529.
• • Whalen S, et al. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics. 2022;23:169–181.