Validate & register multi-modal data#
scRNA data has moved beyond just RNA and can also include the measurements of other modalities such as chromatin accessibility, surface proteins or adaptive immune receptors. ECCITE-seq is designed to enable interrogation of single-cell transcriptomes together with surface protein markers in the context of CRISPR screens.
Here, we’ll showcase how to curate and register ECCITE-seq data from Papalexi21 in the form of MuData objects.
Setup#
!lamin init --storage ./test-multimodal --schema bionty
Show code cell output
💡 creating schemas: core==0.47.5 bionty==0.30.4
✅ saved: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-09-06 17:09:12)
✅ saved: Storage(id='2B62xYuE', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-multimodal', type='local', updated_at=2023-09-06 17:09:12, created_by_id='DzTjkKse')
✅ loaded instance: testuser1/test-multimodal
💡 did not register local instance on hub (if you want, call `lamin register`)
import lamindb as ln
import lnschema_bionty as lb
lb.settings.species = "human"
ln.settings.verbosity = 3
✅ loaded instance: testuser1/test-multimodal (lamindb 0.52.2)
ln.track()
💡 notebook imports: lamindb==0.52.2 lnschema_bionty==0.30.4
✅ saved: Transform(id='yMWSFirS6qv2z8', name='Validate & register multi-modal data', short_name='multimodal', version='0', type=notebook, updated_at=2023-09-06 17:09:15, created_by_id='DzTjkKse')
✅ saved: Run(id='SCKFZElVIwtPznhejs0D', run_at=2023-09-06 17:09:15, transform_id='yMWSFirS6qv2z8', created_by_id='DzTjkKse')
Papalexi21#
Let’s use a MuData object:
Transform 
#
Show code cell content
mdata = ln.dev.datasets.mudata_papalexi21_subset()
mdata
MuData object with n_obs × n_vars = 200 × 300
var: 'name'
4 modalities
rna: 200 x 173
obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_HTO', 'nFeature_HTO', 'nCount_GDO', 'nCount_ADT', 'nFeature_ADT', 'percent.mito', 'MULTI_ID', 'HTO_classification', 'guide_ID', 'gene_target', 'NT', 'perturbation', 'replicate', 'S.Score', 'G2M.Score', 'Phase'
var: 'name'
adt: 200 x 4
obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_HTO', 'nFeature_HTO', 'nCount_GDO', 'nCount_ADT', 'nFeature_ADT', 'percent.mito', 'MULTI_ID', 'HTO_classification', 'guide_ID', 'gene_target', 'NT', 'perturbation', 'replicate', 'S.Score', 'G2M.Score', 'Phase'
var: 'name'
hto: 200 x 12
obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_HTO', 'nFeature_HTO', 'nCount_GDO', 'nCount_ADT', 'nFeature_ADT', 'percent.mito', 'MULTI_ID', 'HTO_classification', 'guide_ID', 'gene_target', 'NT', 'perturbation', 'replicate', 'S.Score', 'G2M.Score', 'Phase'
var: 'name'
gdo: 200 x 111
obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_HTO', 'nFeature_HTO', 'nCount_GDO', 'nCount_ADT', 'nFeature_ADT', 'percent.mito', 'MULTI_ID', 'HTO_classification', 'guide_ID', 'gene_target', 'NT', 'perturbation', 'replicate', 'S.Score', 'G2M.Score', 'Phase'
var: 'name'MuData objects build on top of AnnData objects to store and serialize multimodal data. More information can be found on the MuData documentation.
First we register the file:
file = ln.File(
"papalexi21_subset.h5mu", description="Sub-sampled MuData from Papalexi21"
)
file.save()
✅ storing file 'sAjL2jm6hR9u21U1Yydi' at '.lamindb/sAjL2jm6hR9u21U1Yydi.h5mu'
Now let’s validate and register the 3 feature sets this data contains:
RNA (gene expression)
ADT (antibody derived tags reflecting surface proteins)
obs (metadata)
For the two modalities rna and adt, we use bionty tables as the reference:
Validate 
#
mdata["rna"].var_names[:5]
Index(['RP5-827C21.6', 'XX-CR54.1', 'SH2D6', 'RP11-379B18.5', 'RP11-778D9.12'], dtype='object', name='index')
lb.Gene.validate(mdata["rna"].var_names, lb.Gene.symbol);
❗ 173 terms (100.00%) are not validated for symbol: RP5-827C21.6, XX-CR54.1, SH2D6, RP11-379B18.5, RP11-778D9.12, RP11-703G6.1, AC005150.1, RP11-717H13.1, CTC-498J12.1, CTC-467M3.1, ARHGAP26-AS1, GABRA1, HIST1H4K, HLA-DQB1-AS1, RP11-524H19.2, SPACA1, VNN1, AC006042.7, AC002066.1, AC073934.6, ...
genes = lb.Gene.from_values(mdata["rna"].var_names, lb.Gene.symbol)
ln.save(genes)
✅ created 77 Gene records from Bionty matching symbol: 'SH2D6', 'ARHGAP26-AS1', 'GABRA1', 'HLA-DQB1-AS1', 'SPACA1', 'VNN1', 'CTAGE15', 'PFKFB1', 'TRPC5', 'RBPMS-AS1', 'CA8', 'CSMD3', 'ZNF483', 'AK8', 'TMEM72-AS1', 'ARAP1-AS2', 'CRYAB', 'HOXC-AS2', 'LRRIQ1', 'TUBA3C', ...
✅ created 12 Gene records from Bionty matching synonyms: 'CTC-467M3.1', 'HIST1H4K', 'CASC1', 'LARGE', 'NBPF16', 'C1orf65', 'IBA57-AS1', 'KIAA1239', 'TMEM75', 'AP003419.16', 'FAM65C', 'C14orf177'
❗ ambiguous validation in Bionty for 6 records: 'HLA-DQB1-AS1', 'CTAGE15', 'CTRB2', 'LGALS9C', 'PCDHB11', 'TBC1D3G'
❗ did not create Gene records for 84 non-validated symbols: 'AC002066.1', 'AC004019.13', 'AC005150.1', 'AC006042.7', 'AC011558.5', 'AC026471.6', 'AC073934.6', 'AC091132.1', 'AC092295.4', 'AC092687.5', 'AE000662.93', 'AL132989.1', 'AP000442.4', 'CTA-373H7.7', 'CTB-134F13.1', 'CTB-31O20.9', 'CTC-498J12.1', 'CTD-2562J17.2', 'CTD-3012A18.1', 'CTD-3065B20.2', ...
mdata["rna"].var_names = lb.Gene.standardize(mdata["rna"].var_names, lb.Gene.symbol)
💡 standardized 89/173 terms
validated = lb.Gene.validate(mdata["rna"].var_names, lb.Gene.symbol)
✅ 89 terms (51.40%) are validated for symbol
❗ 84 terms (48.60%) are not validated for symbol: RP5-827C21.6, XX-CR54.1, RP11-379B18.5, RP11-778D9.12, RP11-703G6.1, AC005150.1, RP11-717H13.1, CTC-498J12.1, RP11-524H19.2, AC006042.7, AC002066.1, AC073934.6, RP11-268G12.1, U52111.14, RP11-235C23.5, RP11-12J10.3, RP11-324E6.9, RP11-187A9.3, RP11-365N19.2, RP11-346D14.1, ...
new_genes = [
lb.Gene(symbol=symbol, species=lb.settings.species)
for symbol in mdata["rna"].var_names[~validated]
]
ln.save(new_genes)
lb.Gene.validate(mdata["rna"].var_names, lb.Gene.symbol);
✅ 173 terms (100.00%) are validated for symbol
feature_set_rna = ln.FeatureSet.from_values(
mdata["rna"].var_names, field=lb.Gene.symbol
)
✅ 173 terms (100.00%) are validated for symbol
mdata["adt"].var_names
Index(['CD86', 'PDL1', 'PDL2', 'CD366'], dtype='object', name='index')
lb.CellMarker.validate(mdata["adt"].var_names, field=lb.CellMarker.name);
❗ 4 terms (100.00%) are not validated for name: CD86, PDL1, PDL2, CD366
markers = lb.CellMarker.from_values(mdata["adt"].var_names, field=lb.CellMarker.name)
ln.save(markers)
✅ created 4 CellMarker records from Bionty matching name: 'CD86', 'PDL1', 'PDL2', 'CD366'
lb.CellMarker.validate(mdata["adt"].var_names, field=lb.CellMarker.name);
✅ 4 terms (100.00%) are validated for name
Register 
#
feature_set_adt = ln.FeatureSet.from_values(
mdata["adt"].var_names, field=lb.CellMarker.name
)
✅ 4 terms (100.00%) are validated for name
Link them to file:
file.features.add_feature_set(feature_set_rna, slot="rna")
file.features.add_feature_set(feature_set_adt, slot="adt")
The 3rd feature set is the obs:
obs = mdata["rna"].obs
We’re only interested in a single metadata column:
ln.Feature(name="gene_target", type="category").save()
features = ln.Feature.from_df(obs)
ln.save(features)
feature_set_obs = ln.FeatureSet.from_df(obs)
✅ 19 terms (100.00%) are validated for name
file.features.add_feature_set(feature_set_obs, slot="obs")
gene_targets = lb.Gene.from_values(obs["gene_target"], lb.Gene.symbol)
ln.save(gene_targets)
features = ln.Feature.lookup()
file.add_labels(gene_targets, feature=features.gene_target)
✅ created 23 Gene records from Bionty matching symbol: 'IFNGR1', 'CAV1', 'IRF7', 'ATF2', 'NFKBIA', 'STAT1', 'SPI1', 'JAK2', 'STAT2', 'IFNGR2', 'CD86', 'STAT5A', 'SMAD4', 'ETV7', 'IRF1', 'UBE2L6', 'PDCD1LG2', 'BRD4', 'POU2F2', 'STAT3', ...
✅ created 1 Gene record from Bionty matching synonyms: 'MARCH8'
❗ ambiguous validation in Bionty for 4 records: 'MARCHF8', 'IRF7', 'IFNGR2', 'TNFRSF14'
❗ did not create Gene record for 1 non-validated symbol: 'NT'
✅ linked feature 'gene_target' to registry 'bionty.Gene'
nt = ln.Label(name="NT", description="Non-targeting control of perturbations")
nt.save()
file.add_labels(nt, feature=features.gene_target)
✅ linked feature 'gene_target' to registry 'core.Label'
for col in ["orig.ident", "perturbation", "replicate", "Phase", "guide_ID"]:
labels = [ln.Label(name=name) for name in obs[col].unique()]
ln.save(labels)
✅ loaded Label record with exact same name: 'NT'
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ records with similar names exist! did you mean to load one of them?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| NT | viJ2iltB | 90.0 |
❗ records with similar names exist! did you mean to load one of them?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
| NT | viJ2iltB | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ records with similar names exist! did you mean to load one of them?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ records with similar names exist! did you mean to load one of them?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
| NT | viJ2iltB | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ records with similar names exist! did you mean to load one of them?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| NT | viJ2iltB | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| NT | viJ2iltB | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| NT | viJ2iltB | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| G1 | 0gyRcpdm | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
❗ record with similar name exist! did you mean to load it?
| id | __ratio__ | |
|---|---|---|
| name | ||
| S | HO4Aos7A | 90.0 |
Because none of these labels seem like something we’d want to track in the registry or validate, we don’t link them to the file.
file.features
Features:
rna: FeatureSet(id='mHACrKcG85SzG7uDzKuU', n=184, type='number', registry='bionty.Gene', hash='Y8lsRtXCZKyPPberKAF0', updated_at=2023-09-06 17:09:23, created_by_id='DzTjkKse')
'MYO16', 'AP000442.4', 'RP11-524H19.2', 'ARAP1-AS2', 'SMCR5', 'MIA', 'HOXC-AS2', 'GRAPL', 'RP11-113K21.4', 'PXDNL', ...
adt: FeatureSet(id='avXDhT5hctS2bE1kt5sD', n=4, type='number', registry='bionty.CellMarker', hash='b-CtyjgPRO0WN27lTOqC', updated_at=2023-09-06 17:09:23, created_by_id='DzTjkKse')
'CD366', 'PDL1', 'CD86', 'PDL2'
obs: FeatureSet(id='vqTiNHhutMBgHqNMEWjD', n=19, registry='core.Feature', hash='Dzy-71bTbTKubcoOK9vi', updated_at=2023-09-06 17:09:24, created_by_id='DzTjkKse')
🔗 gene_target (bionty.Gene|core.Label)
🔗 gene_target (28, bionty.Gene): 'STAT3', 'MARCHF8', 'IFNGR2', 'ATF2', 'IFNGR2', 'CD86', 'ETV7', 'MARCHF8', 'IRF7', 'BRD4', ...
🔗 gene_target (1, core.Label): 'NT'
guide_ID (category)
nCount_RNA (number)
nCount_GDO (number)
orig.ident (category)
S.Score (number)
nCount_ADT (number)
MULTI_ID (category)
nFeature_ADT (number)
perturbation (category)
NT (category)
nCount_HTO (number)
nFeature_RNA (number)
G2M.Score (number)
Phase (category)
nFeature_HTO (number)
HTO_classification (category)
replicate (category)
percent.mito (number)
file.describe()
💡 File(id='sAjL2jm6hR9u21U1Yydi', suffix='.h5mu', accessor='MuData', description='Sub-sampled MuData from Papalexi21', size=606320, hash='RaivS3NesDOP-6kNIuaC3g', hash_type='md5', updated_at=2023-09-06 17:09:16)
Provenance:
🗃️ storage: Storage(id='2B62xYuE', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-multimodal', type='local', updated_at=2023-09-06 17:09:12, created_by_id='DzTjkKse')
💫 transform: Transform(id='yMWSFirS6qv2z8', name='Validate & register multi-modal data', short_name='multimodal', version='0', type=notebook, updated_at=2023-09-06 17:09:16, created_by_id='DzTjkKse')
👣 run: Run(id='SCKFZElVIwtPznhejs0D', run_at=2023-09-06 17:09:15, transform_id='yMWSFirS6qv2z8', created_by_id='DzTjkKse')
👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-09-06 17:09:12)
Features:
rna: FeatureSet(id='mHACrKcG85SzG7uDzKuU', n=184, type='number', registry='bionty.Gene', hash='Y8lsRtXCZKyPPberKAF0', updated_at=2023-09-06 17:09:23, created_by_id='DzTjkKse')
'MYO16', 'AP000442.4', 'RP11-524H19.2', 'ARAP1-AS2', 'SMCR5', 'MIA', 'HOXC-AS2', 'GRAPL', 'RP11-113K21.4', 'PXDNL', ...
adt: FeatureSet(id='avXDhT5hctS2bE1kt5sD', n=4, type='number', registry='bionty.CellMarker', hash='b-CtyjgPRO0WN27lTOqC', updated_at=2023-09-06 17:09:23, created_by_id='DzTjkKse')
'CD366', 'PDL1', 'CD86', 'PDL2'
obs: FeatureSet(id='vqTiNHhutMBgHqNMEWjD', n=19, registry='core.Feature', hash='Dzy-71bTbTKubcoOK9vi', updated_at=2023-09-06 17:09:24, created_by_id='DzTjkKse')
🔗 gene_target (bionty.Gene|core.Label)
🔗 gene_target (28, bionty.Gene): 'STAT3', 'MARCHF8', 'IFNGR2', 'ATF2', 'IFNGR2', 'CD86', 'ETV7', 'MARCHF8', 'IRF7', 'BRD4', ...
🔗 gene_target (1, core.Label): 'NT'
guide_ID (category)
nCount_RNA (number)
nCount_GDO (number)
orig.ident (category)
S.Score (number)
nCount_ADT (number)
MULTI_ID (category)
nFeature_ADT (number)
perturbation (category)
NT (category)
nCount_HTO (number)
nFeature_RNA (number)
G2M.Score (number)
Phase (category)
nFeature_HTO (number)
HTO_classification (category)
replicate (category)
percent.mito (number)
Labels:
🏷️ genes (28, bionty.Gene): 'STAT3', 'MARCHF8', 'IFNGR2', 'ATF2', 'IFNGR2', 'CD86', 'ETV7', 'MARCHF8', 'IRF7', 'BRD4', ...
🏷️ labels (1, core.Label): 'NT'
file.view_flow()
# clean up test instance
!lamin delete --force test-multimodal
!rm -r test-multimodal
Show code cell output
💡 deleting instance testuser1/test-multimodal
✅ deleted instance settings file: /home/runner/.lamin/instance--testuser1--test-multimodal.env
✅ instance cache deleted
✅ deleted '.lndb' sqlite file
❗ consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/test-multimodal