Models¶
Mlp Classification and Regression¶
We have implemented a Mlp Classifier and Mlp Regressor with an interface similar to the one in scikit-learn. Our implementation is much faster than the one in scikit-learn since it is based on pytorch and runs on GPUs. Moreover, we have reimplemented the method described in the Unsupervised Label Noise Modeling and Loss Correction for classification with noisy labels. The use-case for the MlpRegressor and MlpClassifier is to check whether the extracted features can be used to recapitulate biological annotations. For example, can the features be used to classify the the tissue-condition (“wild-type” vs “disease”)? Or, can the features be used to regress the Moran’s I score?
- class BaseEstimator(hidden_dims=None, hidden_activation='relu', batch_size=256, solver='adam', alpha=0.99, momentum=0.9, betas=(0.9, 0.999), warm_up_epochs=0, warm_down_epochs=0, max_epochs=200, min_learning_rate=0.0001, max_learning_rate=0.001, min_weight_decay=0.0001, max_weight_decay=0.0001, **kargs)[source]¶
Abstract Base Class which implements an interface similar to the MLP classifier/regressor in scikit-learn. The classes
PlRegressor
andPlClassifier
inherit from this class.
- class MlpRegressor(output_activation=torch.nn.Identity, **kargs)[source]¶
Bases:
tissue_purifier.models.classifier_regressor.scikit_learn_interface.BaseEstimator
Mlp regressor with interface similar to scikit-learn but able to run on GPUs.
- fit(X, y)[source]¶
Fit the model.
- Parameters
X – independent variable of shape \((n, *)\)
y – dependent variable of shape \((n)\)
- Return type
None
- property is_classifier¶
Returns False. For compatibility with scikit-learn interface.
- property is_regressor¶
Returns True. For compatibility with scikit-learn interface.
- predict(X)¶
Run the model forward to obtain the predictions, i.e. \(y_\text{pred} = \text{model}(X)\).
- Parameters
X – independent variable of shape \((n, *)\)
- Returns
y – the predicted values of shape \((n)\)
- Return type
ndarray
- score(X, y)¶
Compute the predictions, i.e. \(y_\text{pred} = \text{model}(X)\), and score them against the true values y.
- Parameters
X – independent variable of shape \((n, *)\)
y – dependent variable of shape \((n)\)
- Returns
score – R^2 (coefficient of determination) between \(y_\text{pred}\) and y.
- Return type
float
- class MlpClassifier(noisy_labels=False, bootstrap_epoch_start=100, lambda_reg=1.0, hard_bootstrapping=False, **kargs)[source]¶
Bases:
tissue_purifier.models.classifier_regressor.scikit_learn_interface.BaseEstimator
Mlp classifier with interface similar to scikit-learn but able to run on GPUs.
It can performs classification with noisy labels following the method described in Unsupervised Label Noise Modeling and Loss Correction According to this method, the labels are dynamically corrected according to the formula:
\(l_\text{new} = (1.0-w) \times l_\text{old} + w \times p_\text{net}\)
where \(l_\text{old}\) are the noisy (and one-hot) original labels, \(p_\text{net}\) are the probabilities computed by the neural network and w is the probability of label being incorrect. w is computed by solving the assignment problem for a 2-component Mixture Model. This is based on the idea that correct (incorrect) labels will lead to small (large) losses. Therefore correct labels will belong to the low-loss component and incorrect label will belong to the high-loss component.
- fit(X, y)[source]¶
Fit the model.
- Parameters
X – independent variable of shape \((n, *)\)
y – dependent variable of shape \((n)\)
- property is_classifier¶
Returns True. For compatibility with scikit-learn interface.
- property is_regressor¶
Returns False. For compatibility with scikit-learn interface.
- predict(X)¶
Run the model forward to obtain the predictions, i.e. \(y_\text{pred} = \text{model}(X)\).
- Parameters
X – independent variable of shape \((n, *)\)
- Returns
y – the predicted values of shape \((n)\)
- Return type
ndarray
- predict_log_proba(X)¶
Compute the log_probabilities for all the classes.
- Parameters
X – independent variable of shape \((n, *)\)
- Returns
log_p – Log_Probability of all the classes of shape \((n, C)\) where C is the number of classes.
- Return type
ndarray
- predict_proba(X)¶
Compute the probabilities for all the classes.
- Parameters
X – independent variable of shape \((n, *)\)
- Returns
prob – Probability of all the classes of shape \((n, C)\) where C is the number of classes.
- Return type
ndarray
- score(X, y)¶
Compute the predictions, i.e. \(y_\text{pred} = \text{model}(X)\), and score them against the true values y.
- Parameters
X – independent variable of shape \((n, *)\)
y – dependent variable of shape \((n)\)
- Returns
accuracy – Accuracy classification score
- Return type
float
Self Supervised Models¶
We have implemented multiple self-supervised learning (ssl) models.
All these models ingest image patches. The data augmentation strategy and loss function depends on
the ssl framework chosen. After training, these models can be used to compute features for new image patches.
All ssl models inherit from the base class tissue_purifier.models.ssl_models.SslModelBase
which
is responsible for the validation (which is common to all ssl models) and logging.
- class SslModelBase(val_iomin_threshold)[source]¶
Base class for the self-supervised learning (ssl) models (Vae, Dino, Barlow, Simclr). This base class is responsible for the validation (which is common to all ssl models) and some logging. The child classes need to implement
head_and_backbone_embeddings_step()
,forward()
andtraining_step()
.
- class BarlowModel(backbone_type, image_in_ch, head_hidden_chs, head_out_ch, lambda_off_diagonal, optimizer_type, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, val_iomin_threshold=0.0, **kwargs)[source]¶
Barlow self supervised learning model. Inspired by the BarlowTwin official implementation and this Barlow pytorch lightning reimplementation.
- classmethod add_specific_args(parent_parser)[source]¶
Utility functions which add parameters to argparse to simplify setting up a CLI
Example
>>> import sys >>> import argparse >>> parser = argparse.ArgumentParser(add_help=False, conflict_handler='resolve') >>> parser = BarlowModel.add_specific_args(parser) >>> args = parser.parse_args(sys.argv[1:])
- class DinoModel(image_in_ch, backbone_type, head_hidden_chs, head_use_bn, head_out_ch, center_momentum, optimizer_type, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, val_iomin_threshold=0.0, set_temperature_using_ipr_init=False, ipr_teacher_init=40.0, ipr_student_init=80.0, temperature_teacher_init=0.04, temperature_student_init=0.1, param_momentum_init=0.996, param_momentum_final=0.996, param_momentum_epochs_end=1000, **kwargs)[source]¶
DINO self supervised learning model. Inspired by the DINO official implementation and this DINO pytorch-lightning reimplementation
- classmethod add_specific_args(parent_parser)[source]¶
Utility functions which add parameters to argparse to simplify setting up a CLI
Example
>>> import sys >>> import argparse >>> parser = argparse.ArgumentParser(add_help=False, conflict_handler='resolve') >>> parser = DinoModel.add_specific_args(parser) >>> args = parser.parse_args(sys.argv[1:])
- class SimclrModel(backbone_type, image_in_ch, head_hidden_chs, head_out_ch, optimizer_type, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, val_iomin_threshold=0.0, **kwargs)[source]¶
Simclr self supervised learning model. Inspired by the Simclr official implementation and this Simclr pytorch lightning reimplementation
- classmethod add_specific_args(parent_parser)[source]¶
Utility functions which add parameters to argparse to simplify setting up a CLI
Example
>>> import sys >>> import argparse >>> parser = argparse.ArgumentParser(add_help=False, conflict_handler='resolve') >>> parser = SimclrModel.add_specific_args(parser) >>> args = parser.parse_args(sys.argv[1:])
- class VaeModel(backbone_type, global_size, image_in_ch, latent_dim, encoder_hidden_dims, decoder_output_activation, optimizer_type, beta_vae_init, momentum_beta_vae, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, gradient_clip_val=0.0, gradient_clip_algorithm='value', val_iomin_threshold=0.0, **kwargs)[source]¶
Convolutional Variational Auto Encoders (VAE) with dynamically adjusted hyper-parameter \(\beta\).
The loss function is a weighted sum of the reconstruction (MSE) and regularization (KL):
\(\text{loss} = \beta \times \text{KL} + (1-\beta) \times \text{MSE}\)
We view this problem as a Multi-Objective Optimization (minimizing MSE and KL) and we dynamically adjust \(\beta \in (.0, 1.0)\) taking inspiration from the ideas in Multi-Task Learning as Multi-Objective Optimization
Note
Depending on the data pre-processing step, the input images might be mostly zeros with few “spots” in them. In this case, the VAE might collapse to a local minimum in the loss function corresponding to identically zero reconstruction. You might try to solve this collapse by using a larger latent dimension or by implementing a KL-free variation of the VAE, such as VQ-VAE. However, spotty images are inherently difficult for (unstructured) VAE which are required to retain (in their latent embedding) precise information about the location of each individual spots.
- classmethod add_specific_args(parent_parser)[source]¶
Utility functions which add parameters to argparse to simplify setting up a CLI
Example
>>> import sys >>> import argparse >>> parser = argparse.ArgumentParser(add_help=False, conflict_handler='resolve') >>> parser = VaeModel.add_specific_args(parser) >>> args = parser.parse_args(sys.argv[1:])
Patch Analyzers¶
We have implemented two classes tissue_purifier.models.patch_analyzer.patch_analyzer.Composition
and
tissue_purifier.models.patch_analyzer.patch_analyzer.SpatialAutocorrelation
which can be used to extract
annotations from image patches. Together with the other models described in Self Supervised Models
and Mlp Classification and Regression
these allow to answer interesting questions such as: “Can the patch embedding be used to predict the
cellular-composition of a patch?” or “Can the patch embedding be used to predict the Moran’s I score of a patch?”.
- class Composition(return_fraction=True)[source]¶
Counts the number of elements in every channel and return their raw values or their normalized frequencies.
- __call__(data, windows=None)[source]¶
Count the intensity for each channel in a 2D window.
- Parameters
data (
Union
[Tensor
,Tensor
,SparseImage
,List
[Tensor
],List
[SparseImage
]]) – torch.Tensor or torch.sparse.Tensor or SparseImage (or list thereof) corresponding to a spatial data of shape \((C, W, H)\)windows (
Optional
[Tuple
[float
,float
,float
,float
]]) – tuple with (min_row, min_col, max_row, max_col). If None (default) the entire image is considered.
- Returns
composition – A vector of size C with the count for each channel (or list thereof).
- Return type
Tensor
- class SpatialAutocorrelation(modality='moran', n_neighbours=None, radius=None, neigh_correct=True)[source]¶
Compute the Moran’s I or Geary’s C score of a sparse torch tensor. If the sparse tensor has ch channels it will produce ch scores, each indicating how each channel is dispersed in all the others.
Note
The results of a Moran’s I and Geary’s C tests depend on the choice of the weight matrix (see Moran and Geary for details). The input parameters determine the strategy used for the construction of the weight matrix.
- __call__(data)¶
- Parameters
data (
Union
[SparseImage
,List
[SparseImage
],Tensor
,List
[Tensor
]]) – A (list of) sparse tensor with C channels- Returns
score – A (list of) torch.tensor of size C with the score (either moran or gready) indicating how each channel is dispersed in all the others.