Using scvi-hub to upload pretrained scvi-tools models#
In this tutorial, we will see how to use scvi-tools to upload pretrained models onto Hugging Face. We will also see how to handle large training datasets.
If you have not already, make sure to refer to our scvi_hub_into_and_download tutorial, which is a pre-requisite to this one. It introduces Hugging Face (HF) and the scvi-hub, and describes how to use them for downloading pre-trained models from the HF Model Hub.
Note
Running the following cell will install tutorial dependencies on Google Colab only. It will have no effect on environments other than Google Colab.
!pip install --quiet scvi-colab
from scvi_colab import install
install()
import os
import tempfile
import anndata
import scanpy as sc
import scvi
import seaborn as sns
import torch
from scvi.hub import HubMetadata, HubModel, HubModelCardHelper
scvi.settings.seed = 0
print("Last run with scvi-tools version:", scvi.__version__)
Last run with scvi-tools version: 1.4.2
Note
You can modify save_dir below to change where the data files for this tutorial are saved.
sc.set_figure_params(figsize=(6, 6), frameon=False)
sns.set_theme()
torch.set_float32_matmul_precision("high")
save_dir = tempfile.TemporaryDirectory()
%config InlineBackend.print_figure_kwargs={"facecolor": "w"}
%config InlineBackend.figure_format="retina"
Imports#
Let’s start by adding the Python imports we need.
Pretrain a demo model#
Let’s pretrain a model on some synthetic data which we’ll use to upload to the scvi-hub later.
model_path = os.path.join(save_dir.name, "scvi_hub_upload")
adata = scvi.data.synthetic_iid()
scvi.model.SCVI.setup_anndata(adata)
model = scvi.model.SCVI(adata)
model.train(1)
model.save(model_path, save_anndata=True, overwrite=True)
Model Card and Metadata#
To upload pretrained models, you’ll need to create an instance of the HubModel class and then simply call its push_to_huggingface_hub method.
As you can see from the API reference, the HubModel init function requires metadata and a Model Card. There are a few ways you can provide these:
The metadata can be either an instance of the
HubMetadataclass that contains the required metadata for this model, or a path to a JSON file on disk where this metadata can be read from.The Model Card can be an instance of the
HubModelCardHelperclass created for this model, or an instance of the HF Model Card object, or a path to a Markdown file on disk where the model card can be read from.You can also use the
HubModelCardHelperclass to create a Model Card from the scvi-tools template, then save it on disk and change it as you wish before passing its path into theHubModelclass.
Here we’ll see how to create the HubMetadata and a Model Card from the data on disk.
hm = HubMetadata.from_dir(model_path, anndata_version=anndata.__version__)
hmch = HubModelCardHelper.from_dir(
model_path,
license_info="cc-by-4.0",
anndata_version=anndata.__version__,
data_modalities=["rna", "protein"],
data_is_annotated=False,
description="This is a demo model used during upload demo.",
references="None.",
)
INFO File /tmp/tmpu0d6wpcd/scvi_hub_upload/model.pt already downloaded
INFO File /tmp/tmpu0d6wpcd/scvi_hub_upload/model.pt already downloaded
print(hmch.model_card.content)
---
library_name: scvi-tools
license: cc-by-4.0
tags:
- biology
- genomics
- single-cell
- model_cls_name:SCVI
- scvi_version:1.4.2
- anndata_version:0.12.7
- modality:rna
- modality:protein
- annotated:False
---
ScVI is a variational inference model for single-cell RNA-seq data that can learn an underlying
latent space, integrate technical batches and impute dropouts.
The learned low-dimensional latent representation of the data can be used for visualization and
clustering.
scVI takes as input a scRNA-seq gene expression matrix with cells and genes.
We provide an extensive [user guide](https://docs.scvi-tools.org/en/stable/user_guide/models/scvi.html).
- See our original manuscript for further details of the model:
[scVI manuscript](https://www.nature.com/articles/s41592-018-0229-2).
- See our manuscript on [scvi-hub](https://www.biorxiv.org/content/10.1101/2024.03.01.582887v2) how
to leverage pre-trained models.
This model can be used for fine tuning on new data using our Arches framework:
[Arches tutorial](https://docs.scvi-tools.org/en/stable/tutorials/notebooks/scrna/scarches_scvi_tools.html).
# Model Description
This is a demo model used during upload demo.
# Metrics
We provide here key performance metrics for the uploaded model, if provided by the data uploader.
<details>
<summary><strong>Coefficient of variation</strong></summary>
The cell-wise coefficient of variation summarizes how well variation between different cells is
preserved by the generated model expression. Below a squared Pearson correlation coefficient of 0.4
, we would recommend not to use generated data for downstream analysis, while the generated latent
space might still be useful for analysis.
**Cell-wise Coefficient of Variation**:
Not provided by uploader
The gene-wise coefficient of variation summarizes how well variation between different genes is
preserved by the generated model expression. This value is usually quite high.
**Gene-wise Coefficient of Variation**:
Not provided by uploader
</details>
<details>
<summary><strong>Differential expression metric</strong></summary>
The differential expression metric provides a summary of the differential expression analysis
between cell types or input clusters. We provide here the F1-score, Pearson Correlation
Coefficient of Log-Foldchanges, Spearman Correlation Coefficient, and Area Under the Precision
Recall Curve (AUPRC) for the differential expression analysis using Wilcoxon Rank Sum test for each
cell-type.
**Differential expression**:
Not provided by uploader
</details>
# Model Properties
We provide here key parameters used to setup and train the model.
<details>
<summary><strong>Model Parameters</strong></summary>
These provide the settings to setup the original model:
```json
{
"n_hidden": 128,
"n_latent": 10,
"n_layers": 1,
"dropout_rate": 0.1,
"dispersion": "gene",
"gene_likelihood": "zinb",
"use_observed_lib_size": true,
"latent_distribution": "normal"
}
```
</details>
<details>
<summary><strong>Setup Data Arguments</strong></summary>
Arguments passed to setup_anndata of the original model:
```json
{
"layer": null,
"batch_key": null,
"labels_key": null,
"size_factor_key": null,
"categorical_covariate_keys": null,
"continuous_covariate_keys": null
}
```
</details>
<details>
<summary><strong>Data Registry</strong></summary>
Registry elements for AnnData manager:
| Registry Key | scvi-tools Location |
|--------------------------|--------------------------------------|
| X | adata.X |
| batch | adata.obs['_scvi_batch'] |
| labels | adata.obs['_scvi_labels'] |
- **Data is Minified**: False
</details>
<details>
<summary><strong>Summary Statistics</strong></summary>
| Summary Stat Key | Value |
|--------------------------|-------|
| n_batch | 1 |
| n_cells | 400 |
| n_extra_categorical_covs | 0 |
| n_extra_continuous_covs | 0 |
| n_labels | 1 |
| n_vars | 100 |
</details>
<details>
<summary><strong>Training</strong></summary>
<!-- If your model is not uploaded with any data (e.g., minified data) on the Model Hub, then make
sure to provide this field if you want users to be able to access your training data. See the
scvi-tools documentation for details. -->
**Training data url**: Not provided by uploader
If provided by the original uploader, for those interested in understanding or replicating the
training process, the code is available at the link below.
**Training Code URL**: Not provided by uploader
</details>
# References
None.
Note: Suppose I wanted to change the content a little bit. To do that, I’d save the card to disk, change it manually as I wish, and then pass its path to HubModel.
hmch.model_card.save(
"local/my_model_card.md"
) # then change the markdown file on disk...
Create a HubModel and upload it#
Now we can create the HubModel and push it to the HF Model Hub:
hmo = HubModel(model_path, metadata=hm, model_card=hmch)
hmo
HubModel with:
local_dir: /tmp/tmpu0d6wpcd/scvi_hub_upload
model loaded? No
adata loaded? No
large_training_adata loaded? No
metadata:
HubMetadata(scvi_version='1.4.2', anndata_version='0.12.7', model_cls_name='SCVI', training_data_url=None, model_parent_module='scvi.model')
model_card:
------------------------------------------------------------------------------------------------------------------- library_name: scvi-tools license: cc-by-4.0 tags: • biology • genomics • single-cell • model_cls_name:SCVI • scvi_version:1.4.2 • anndata_version:0.12.7 • modality:rna • modality:protein • annotated:False ------------------------------------------------------------------------------------------------------------------- ScVI is a variational inference model for single-cell RNA-seq data that can learn an underlying latent space, integrate technical batches and impute dropouts. The learned low-dimensional latent representation of the data can be used for visualization and clustering. scVI takes as input a scRNA-seq gene expression matrix with cells and genes. We provide an extensive user guide. • See our original manuscript for further details of the model: scVI manuscript. • See our manuscript on scvi-hub how to leverage pre-trained models. This model can be used for fine tuning on new data using our Arches framework: Arches tutorial. Model Description This is a demo model used during upload demo. Metrics We provide here key performance metrics for the uploaded model, if provided by the data uploader. The cell-wise coefficient of variation summarizes how well variation between different cells is preserved by the generated model expression. Below a squared Pearson correlation coefficient of 0.4 , we would recommend not to use generated data for downstream analysis, while the generated latent space might still be useful for analysis. Cell-wise Coefficient of Variation: Not provided by uploader The gene-wise coefficient of variation summarizes how well variation between different genes is preserved by the generated model expression. This value is usually quite high. Gene-wise Coefficient of Variation: Not provided by uploader The differential expression metric provides a summary of the differential expression analysis between cell types or input clusters. We provide here the F1-score, Pearson Correlation Coefficient of Log-Foldchanges, Spearman Correlation Coefficient, and Area Under the Precision Recall Curve (AUPRC) for the differential expression analysis using Wilcoxon Rank Sum test for each cell-type. Differential expression: Not provided by uploader Model Properties We provide here key parameters used to setup and train the model. These provide the settings to setup the original model: { "n_hidden": 128, "n_latent": 10, "n_layers": 1, "dropout_rate": 0.1, "dispersion": "gene", "gene_likelihood": "zinb", "use_observed_lib_size": true, "latent_distribution": "normal" } Arguments passed to setup_anndata of the original model: { "layer": null, "batch_key": null, "labels_key": null, "size_factor_key": null, "categorical_covariate_keys": null, "continuous_covariate_keys": null } Registry elements for AnnData manager: | Registry Key | scvi-tools Location | |--------------------------|--------------------------------------| | X | adata.X | | batch | adata.obs['_scvi_batch'] | | labels | adata.obs['_scvi_labels'] | • Data is Minified: False | Summary Stat Key | Value | |--------------------------|-------| | n_batch | 1 | | n_cells | 400 | | n_extra_categorical_covs | 0 | | n_extra_continuous_covs | 0 | | n_labels | 1 | | n_vars | 100 | Training data url: Not provided by uploader If provided by the original uploader, for those interested in understanding or replicating the training process, the code is available at the link below. Training Code URL: Not provided by uploader References None.
To upload, you need to call:
hmo.push_to_huggingface_hub(
repo_name=repo_name, repo_token=repo_token, repo_create=True
)
We won’t do it here but will explain the parameters you need to pass:
repo_name: The name/id of your repo.repo_token: The token you need to authenticate yourself to the HF Model Hub. It must have “write” permissions. You can get this from your HF account page. Read this article to find out how.The token can either be passed in as plain text or as the full path to a file on disk where the token is stored.
repo_create: Whether you want scvi-tools to create the repo for you. If you want to create the repo yourself on the HF Model Hub or if it already exists, you can set this to False.
Large training data#
So far, all models we’ve seen have contained the dataset in the HF Hub Model object. However, in some cases, this is not possible — or desirable — if the training data is too large. For all files large than 5GB, you are prompted to store your training data on a separate storage and provide its URL to the HubModel. This will alert scvi-tools as to where to pull the data from when loading it (or the model) into memory.
There are four possible scenarios. Here we’re assuming that the minified data is <5GB which is very likely to not be the case.
Your training data is <5GB and it is not minified.
👉 In this case, both your model and data will be uploaded to the same HF Model.Your training data is <5GB and it is minified.
👉 In this case, both your model and minified data will be uploaded to the same HF Model. Optionally, you can provide a link to your full (i.e., non-minified) training data.Your training data is >=5GB and it is not minified.
👉 In this case, only your model will be uploaded to the HF Model. If you want to use your training data, then it is required to provide a link to it (this must be in the required metadata file, and can be present in the model card as well). When needed, scvi-tools will automatically download your training data from the link you registered.Your training data is >=5GB and it is minified.
👉 In this case, both your model and minified data will be uploaded to the same HF Model. Optionally, you can provide a link to your full (i.e., non-minified) training data.
It is highly recommended to try to minify your data if possible. Please refer to our Minification tutorial for how to do that.
Note
It is always possible to use another dataset than your training data. You can set model.adata prior to saving. However, the convention with scvi-hub is to provide access to the training data (full or minified form), so that users can reproduce the results of the model and perform their own analyses on the same data.
Model evaluation#
We recommend that you include some evaluation results in your Model Card. One way to do this is by using our scvi-criticism Python package. It provides a simple API to evaluate the goodness of fit of your model and generate various visualizations. Read more about it in the scvi-criticism documentation.