Project Description
Which AI Architecture Is Best? A Beginner-Friendly Benchmark of the Newest Single-Cell AI Tools
Faculty Mentor: [Soundararajan Ezekiel, MACS, IUP, sezekiel@iup.edu] IT Mentor: [to be paired via PA Science DMZ]
Duration: 2 semesters, Sep–Mar Modality: Fully remote; computing/storage via NCEMS Student: Domingo Benavidez
Project Description
Single-cell RNA sequencing measures gene activity in millions of individual cells, and labeling each cell's type (T cell, B cell, monocyte, etc.) is a routine bottleneck. New "foundation models" — transformer models pretrained on tens of millions of cells — promise automatic labeling, but whether they actually out-label simpler, faster tools remains an open, practical question. An undergraduate will test one champion tool from each of three AI generations — CellTypist (classic one-liner), scANVI [1] and UCE [2] and scGPT (foundation models, run zero-shot) — on the same public immune-cell data. The project needs only one intro Python course: tutorials exist for every tool, no model is built from scratch, and all heavy computing runs on NCEMS.
Problem Statement
Should a biology lab use a free model that runs on a laptop in minutes, or a large foundation model that needs a GPU? Fair, reproducible comparisons using the newest models and freshest data are still rare, so this student-led benchmark fills a practical gap.
Related Work: Why a New Benchmark Is Needed
Prior work supports the project's premise. A Genome Biology benchmark of six foundation models found they do not consistently beat simpler baselines [3]. An earlier zero-shot evaluation found scGPT and Geneformer embeddings sometimes barely beat a random baseline, occasionally matched by the simpler scVI. A 2026 Advanced Science survey of eleven foundation models found UCE among the strongest zero-shot performers, though cost and accessibility varied widely. This project's contribution: a head-to-head test spanning all three tool generations — including the newest 2026 UCE release — on continuously updated CZ CELLxGENE Census data, which the literature has not yet done.
Pilot Study
A two-week feasibility pilot has been run on two small, labeled datasets — the built-in PBMC 3k set and a ~5,000-cell subset streamed from CZ CELLxGENE Census — confirming the pipeline runs end-to-end. CellTypist and scANVI (seeded with CellTypist's confident predictions, since PBMC 3k lacks expert labels) were run on both. Results: 98.1% overall label agreement between the tools and 92.3% agreement on rare cell types (n=261 cells), with the full pipeline completing in under two minutes on a standard CPU — confirming the scoring harness and de-risking Step 2 before it begins.
Pilot dataset Size Purpose
PBMC 3k (scanpy built-in) ~2,700 cells Primary pilot set; matches Step 1 tutorial
Small labeled subset, CZ CELLxGENE Census ~5,000 cells Second source; checks pipeline isn't overfit to one dataset
Left: actual UMAP of PBMC 3k, colored by CellTypist zero-shot labels. Right: actual pilot scoring-harness output comparing CellTypist and scANVI. Real pilot results, not mockups.
The Plan: Three Steps
● Step 1 — Learn (Sep–Oct): Python/Git refresh; official scanpy tutorial (PBMC 3k); pull a large, expert-labeled immune-cell dataset from CZ CELLxGENE Census in one line of code.
● Step 2 — Compare (Nov–Jan): Run all three tool generations on the same data; score on accuracy, rare-cell accuracy, and time/cost. Test generalization on a second, held-out study. (Steps 1–2 = a complete one-semester project.)
● Step 3 — Stretch & share (Feb–Mar): Try tools on a slice of Tahoe-100M (Arc Institute, 2025) to distinguish drug-treated vs. untreated cells; publish code, a plain-language guide, and present a poster.
AI/ML Toolkit
Tool Role Why it’s easy
Python/Jupyter on NCEMS All analysis & figures, in-browser Nothing to install locally
scanpy + CELLxGENE Census API Load, clean, explore data Field standard; streams labeled data in one line
CellTypist (Gen 1) Instant cell-type labeling One line of code; regularly updated
scvi-tools / scANVI (Gen 2) Deep learning for scRNA-seq High-level API, no custom networks
UCE (2026) & scGPT, zero-shot (Gen 3) Pretrained foundation embeddings Checkpoints run as-is + simple classifier on top
Git/GitHub, Conda, AI coding assistants Reproducibility & debugging Industry-standard, mentor-guided use
Data (all open access)
Dataset Size Role
PBMC 3k (scanpy built-in) ~2,700 cells Week-one learning set
Labeled immune data, CZ CELLxGENE Census ~100k–200k cells Main benchmark
Second held-out Census study ~50,000 cells Generalization test
Tahoe-100M slice (Arc VCA, 2025) 1 cell-line subset Stretch: drug-treated vs. untreated
Expected Outcomes
● Public GitHub repo of rerunnable notebooks; a scoreboard rating three tool generations on accuracy, rare-cell performance, and cost.
● Short report/preprint and poster with a practical “which tool when” guide; a student trained in scanpy, scvi-tools, foundation models, Git, and GPU computing.
Timeline (Sep–Mar)
Activity Sep Oct Nov Dec Jan Feb Mar
Bootcamp & NCEMS setup ▲
Pilot study (PBMC 3k)
Stream Census data
CellTypist & scANVI ◆
Zero-shot UCE & scGPT
1Generalization test
Tahoe-100M stretch
Final report & poster ◆
▲ kick-off presentation ◆ milestone / payment
Deliverables & Team
● Kick-off/close-out presentations at PA Science DMZ Seminar Series; interim and final reports; public GitHub repo with NCEMS-archived results; poster for the NCEMS annual meeting.
Faculty mentor: frames questions, weekly meetings, mentors analysis/writing. Student: runs analyses, leads write-up. IT mentor: NCEMS accounts, environments, GPU scheduling, Science DMZ transfer.
Computing, Compliance, Budget & Travel
All data are open access and de-identified (no DUAs); work is expected to qualify as non-human-subjects research (IRB confirmation before start). Large datasets are streamed in subsets; computing/storage via NCEMS with GPU access for foundation-model work.
Budget: Two-semester award $6,000 ($4,000 faculty / $2,000 student), disbursed at mid-point and completion. One-semester option: $3,000 ($2,000/$1,000) at completion. Travel ($500 requested): Registration, lodging, and transportation to present NRRE-P2 results — first priority the NCEMS annual meeting (April), or ABRCMS if accepted there instead.
Reference:
[1] Xu, C., Lopez, R., Mehlman, E., Regier, J., Jordan, M. I. & Yosef, N. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular Systems Biology 17, e9620 (2021). doi:10.15252/msb.20209620 — introduces scANVI, the semi-supervised variant of scVI built to use existing cell state annotations. EMBO Press
[2] Rosen, Y., Roohani, Y., Agrawal, A., Samotorčan, L., Tabula Sapiens Consortium, Quake, S. R. & Leskovec, J. Universal cell embedding provides a foundation model for cell biology. Nature (2026). doi:10.1038/s41586-026-10689-z — your "UCE (Nature, 2026)" claim checks out; it was published 8 July 2026. UCE embeds 36 million cells spanning more than 1,000 named cell types across eight species.
[3] Kedzierska, K. Z., Crawford, L., Amini, A. P. & Lu, A. X. Zero-shot evaluation reveals limitations of single-cell foundation models. Genome Biology 26, 101 (2025). doi:10.1186/s13059-025-03574-x — your strongest citation. Zero-shot Geneformer and scGPT may face reliability challenges and can be outperformed by simpler methods.