Is cell-level train/validation/test splitting acceptable for Geneformer in silico perturbation?
Hi Dr. Theodoris,
Thank you very much for your previous explanations. I would like to clarify one point regarding data splitting for Geneformer in silico perturbation.
My dataset contains normal and tumor cells from multiple patient samples. My primary goal is not to develop a classifier for predicting completely unseen patients, but rather to use Geneformer to distinguish the two cellular states and subsequently perform in silico perturbation to identify genes associated with the transition between them.
I found that patient-level splitting resulted in substantially poorer classification performance. I therefore also tested a cell-level train/validation/test split, in which cells from all patient samples are represented and different cells from the same patient may appear in both the training and held-out sets.
For this specific purpose—learning a meaningful separation between normal and tumor states for subsequent in silico perturbation, rather than evaluating generalization to unseen patients—would you consider this cell-level splitting strategy methodologically acceptable?
For the subsequent perturbation analysis, I can restrict the analysis to held-out cells that were not used for model training.
Thank you very much for your guidance.
Thank you for your question. As you implied, it is generally recommended to hold out entire patients (or entire datasets if available) to confirm what the classifier learned to distinguish the states is generalizable to unseen patients. If there is minimal data though, removing patients can reduce the diversity of the samples seen by the model during training, potentially leading to less generalizability. If absolutely needed, holding out cells is acceptable, but it is thereby harder to rule out that the predictions are generalizable by confirming they extend to held out patients. That being said, if the project will ultimately experimentally validate the nominated targets, that verification will be performed at that stage regardless. Another option is to test whether the embeddings of normal vs. disease are already separable with the pretrained model before fine-tuning. Then one could use all patients for in silico perturbation directly without risk of learning confounding factors during fine-tuning in the case where held-out patients for testing generalizability are unavailable.