This page was generated from data_loading.ipynb. Interactive online version: .
Data loading and preparation¶
Here we walk through the necessary steps to get your data into ready for scvi-tools.
import sys #if branch is stable, will install via pypi, else will install from source branch = "stable" IN_COLAB = "google.colab" in sys.modules if IN_COLAB and branch == "stable": !pip install --quiet scvi-tools[tutorials] elif IN_COLAB and branch != "stable": !pip install --quiet --upgrade jsonschema !pip install --quiet git+https://github.com/yoseflab/scvi-tools@$branch#egg=scvi-tools[tutorials]
import scvi import scanpy as sc
Global seed set to 0
scvi-tools supports the AnnData data format, which also underlies Scanpy. AnnData is quite similar to other popular single cell objects like that of Seurat and SingleCellExperiment. In particular, it allows cell-level and feature-level metadata to coexist in the same data structure as the molecular counts.
It’s also now possible to automatically convert these R-based objects to AnnData within a Jupyter notebook. See the following tutorial for more information.
scvi-tools has a number of convenience methods for loading data from
.h5ad formats. To load outputs from Cell Ranger, please use Scanpy’s reading functionality.
Let us now download an AnnData object (
.h5ad format) and load it using scvi-tools.
--2021-10-05 16:18:13-- http://falexwolf.de/data/pbmc3k_raw.h5ad Resolving falexwolf.de (falexwolf.de)... 188.8.131.52 Connecting to falexwolf.de (falexwolf.de)|184.108.40.206|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 5855727 (5.6M) Saving to: ‘pbmc3k_raw.h5ad’ pbmc3k_raw.h5ad 100%[===================>] 5.58M 3.79MB/s in 1.5s 2021-10-05 16:18:16 (3.79 MB/s) - ‘pbmc3k_raw.h5ad’ saved [5855727/5855727]
pbmc3k = scvi.data.read_h5ad("pbmc3k_raw.h5ad")
AnnData object with n_obs × n_vars = 2700 × 32738 var: 'gene_ids'
This is a fairly simple object, it just contains the count data and the ENSEMBL ids for the genes.
As another example, let’s download a dataset from 10x Genomics. This data was obtained from a CITE-seq experiment, so it also contains protein count data.
--2021-10-05 16:18:48-- https://cf.10xgenomics.com/samples/cell-exp/3.0.2/5k_pbmc_protein_v3/5k_pbmc_protein_v3_filtered_feature_bc_matrix.h5 Resolving cf.10xgenomics.com (cf.10xgenomics.com)... 220.127.116.11, 18.104.22.168, 2606:4700::6812:1ad, ... Connecting to cf.10xgenomics.com (cf.10xgenomics.com)|22.214.171.124|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 17129253 (16M) [binary/octet-stream] Saving to: ‘5k_pbmc_protein_v3_filtered_feature_bc_matrix.h5’ 5k_pbmc_protein_v3_ 100%[===================>] 16.33M 48.8MB/s in 0.3s 2021-10-05 16:18:49 (48.8 MB/s) - ‘5k_pbmc_protein_v3_filtered_feature_bc_matrix.h5’ saved [17129253/17129253]
pbmc5k = sc.read_10x_h5( "5k_pbmc_protein_v3_filtered_feature_bc_matrix.h5", gex_only=False )
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
It’s often helpful to give the gene names unique names.
We can see that
adata.X contains the concatenated gene and protein expression data.
Index(['Antibody Capture', 'Gene Expression'], dtype='object')
We can use scvi-tools to organize this object, which places the protein expression in
AnnData object with n_obs × n_vars = 5247 × 33538 var: 'gene_ids', 'feature_types', 'genome' obsm: 'protein_expression'
Concatenate the datasets¶
adata = pbmc5k.concatenate(pbmc3k)
Notice that the resulting AnnData has a batch key in
Preprocessing the data¶
It is common to remove outliers, and even perform feature selection before model fitting. We prefer the Scanpy preprocessing module at this stage.
sc.pp.filter_genes(adata, min_counts=3) sc.pp.filter_cells(adata, min_counts=3)
As it is popular to normalize the data for many methods, we can use Scanpy for this; however, it’s important to keep the count information intact for scvi-tools models.
adata.layers["counts"] = adata.X.copy()
Now we can proceed with common normalization methods.
sc.pp.normalize_total(adata, target_sum=1e4) sc.pp.log1p(adata)
We can store the normalized values in
.raw to keep them safe in the event the anndata gets subsetted feature-wise.
adata.raw = adata
Register the data with scvi-tools¶
Now that we have an AnnData object, we need to alert scvi-tools of all the interesting data in our object. For example, now that we have batches in our AnnData, we can alert the models that we’d like to perform batch correction. Also, because we have the count data in a layer, we can use the
Normally, we set up the data right before using a model, thus we would call the
setup_anndata method specific to that model. However, we are not using any particular model here since we are just demonstrating data usage and handling in this tutorial. We will use the SCVI model’s
setup_anndata method here and in what follows for sake of example.
scvi.model.SCVI.setup_anndata(adata, layer="counts", batch_key="batch")
INFO Using batches from adata.obs["batch"] INFO No label_key inputted, assuming all cells have same label INFO Using data from adata.layers["counts"] INFO Successfully registered anndata object containing 7947 cells, 14309 vars, 2 batches, 1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra continuous covariates. INFO Please do not further modify adata until model is trained.
Notice the info messages notify us that batches were detected in the data. Just to demonstrate what happens if we don’t include this option:
INFO No batch_key inputted, assuming all cells are same batch INFO No label_key inputted, assuming all cells have same label INFO Using data from adata.layers["counts"] INFO Successfully registered anndata object containing 7947 cells, 14309 vars, 1 batches, 1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra continuous covariates. INFO Please do not further modify adata until model is trained.
Now integration-based tasks can no longer be performed in subsequent models because there is no knowledge of such information.
As PBMC5k is a CITE-seq dataset, we can use scvi-tools to register the protein expression. Note that totalVI is the only current model that uses the protein expression. The usage of registered items is model specific. As another example, registering the labels in the AnnData object will not affect totalVI or scVI, but is necessary to run scANVI.
We have not preprocessed the
pbmc5k object, which we do recommend. We show how to run
setup_anndata in this case for illustrative purposes.
INFO No batch_key inputted, assuming all cells are same batch INFO No label_key inputted, assuming all cells have same label INFO Using data from adata.X INFO Using protein expression from adata.obsm['protein_expression'] INFO Using protein names from columns of adata.obsm['protein_expression'] INFO Successfully registered anndata object containing 5247 cells, 33538 vars, 1 batches, 1 labels, and 32 proteins. Also registered 0 extra categorical covariates and 0 extra continuous covariates. INFO Please do not further modify adata until model is trained.
setup_anndata has been run, the adata object should not be modified. In other words, the very next step in the workflow is to initialize and train the model of interest (e.g., scVI, totalVI). If you do modify the adata, it’s ok, just run
setup_anndata again – and then reinitialize the model.
Viewing the scvi-tools data setup¶
Anndata setup with scvi-tools version 0.0.0.
Data Summary ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓ ┃ Data ┃ Count ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩ │ Cells │ 5247 │ │ Vars │ 33538 │ │ Labels │ 1 │ │ Batches │ 1 │ │ Proteins │ 32 │ │ Extra Categorical Covariates │ 0 │ │ Extra Continuous Covariates │ 0 │ └──────────────────────────────┴───────┘
SCVI Data Registry ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Data ┃ scvi-tools Location ┃ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ X │ adata.X │ │ batch_indices │ adata.obs['_scvi_batch'] │ │ labels │ adata.obs['_scvi_labels'] │ │ protein_expression │ adata.obsm['protein_expression'] │ └────────────────────┴──────────────────────────────────┘
Label Categories ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓ ┃ Source Location ┃ Categories ┃ scvi-tools Encoding ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩ │ adata.obs['_scvi_labels'] │ 0 │ 0 │ └───────────────────────────┴────────────┴─────────────────────┘
Batch Categories ┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓ ┃ Source Location ┃ Categories ┃ scvi-tools Encoding ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩ │ adata.obs['_scvi_batch'] │ 0 │ 0 │ └──────────────────────────┴────────────┴─────────────────────┘
Transferring the anndata setup¶
In this section we will transfer the setup from one AnnData object to another AnnData object.
adata1 = scvi.data.synthetic_iid() print(adata1)
INFO Using batches from adata.obs["batch"] INFO Using labels from adata.obs["labels"] INFO Using data from adata.X INFO Using protein expression from adata.obsm['protein_expression'] INFO Using protein names from adata.uns['protein_names'] INFO Successfully registered anndata object containing 400 cells, 100 vars, 2 batches, 3 labels, and 100 proteins. Also registered 0 extra categorical covariates and 0 extra continuous covariates. INFO Please do not further modify adata until model is trained. AnnData object with n_obs × n_vars = 400 × 100 obs: 'batch', 'labels', '_scvi_batch', '_scvi_labels' uns: 'protein_names', '_scvi' obsm: 'protein_expression'
adata2 = scvi.data.synthetic_iid(run_setup_anndata=False) print(adata2)
AnnData object with n_obs × n_vars = 400 × 100 obs: 'batch', 'labels' uns: 'protein_names' obsm: 'protein_expression'
transfer_anndata_setup() will use the fields used to setup adata1 to register the data in adata2.
INFO Using data from adata.X INFO Registered keys:['X', 'batch_indices', 'labels', 'protein_expression'] INFO Successfully registered anndata object containing 400 cells, 100 vars, 2 batches, 3 labels, and 100 proteins. Also registered 0 extra categorical covariates and 0 extra continuous covariates.