# Models¶

## Mlp Classification and Regression¶

We have implemented a Mlp Classifier and Mlp Regressor with an interface similar to the one in scikit-learn. Our implementation is much faster than the one in scikit-learn since it is based on pytorch and runs on GPUs. Moreover, we have reimplemented the method described in the Unsupervised Label Noise Modeling and Loss Correction for classification with noisy labels. The use-case for the MlpRegressor and MlpClassifier is to check whether the extracted features can be used to recapitulate biological annotations. For example, can the features be used to classify the the tissue-condition (“wild-type” vs “disease”)? Or, can the features be used to regress the Moran’s I score?

class BaseEstimator(hidden_dims=None, hidden_activation='relu', batch_size=256, solver='adam', alpha=0.99, momentum=0.9, betas=(0.9, 0.999), warm_up_epochs=0, warm_down_epochs=0, max_epochs=200, min_learning_rate=0.0001, max_learning_rate=0.001, min_weight_decay=0.0001, max_weight_decay=0.0001, **kargs)[source]

Abstract Base Class which implements an interface similar to the MLP classifier/regressor in scikit-learn. The classes PlRegressor and PlClassifier inherit from this class.

class MlpRegressor(output_activation=torch.nn.Identity, **kargs)[source]

Mlp regressor with interface similar to scikit-learn but able to run on GPUs.

fit(X, y)[source]

Fit the model.

Parameters
• X – independent variable of shape $$(n, *)$$

• y – dependent variable of shape $$(n)$$

Return type

None

property is_classifier

Returns False. For compatibility with scikit-learn interface.

property is_regressor

Returns True. For compatibility with scikit-learn interface.

predict(X)

Run the model forward to obtain the predictions, i.e. $$y_\text{pred} = \text{model}(X)$$.

Parameters

X – independent variable of shape $$(n, *)$$

Returns

y – the predicted values of shape $$(n)$$

Return type

ndarray

score(X, y)

Compute the predictions, i.e. $$y_\text{pred} = \text{model}(X)$$, and score them against the true values y.

Parameters
• X – independent variable of shape $$(n, *)$$

• y – dependent variable of shape $$(n)$$

Returns

score – R^2 (coefficient of determination) between $$y_\text{pred}$$ and y.

Return type

float

class MlpClassifier(noisy_labels=False, bootstrap_epoch_start=100, lambda_reg=1.0, hard_bootstrapping=False, **kargs)[source]

Mlp classifier with interface similar to scikit-learn but able to run on GPUs.

It can performs classification with noisy labels following the method described in Unsupervised Label Noise Modeling and Loss Correction According to this method, the labels are dynamically corrected according to the formula:

$$l_\text{new} = (1.0-w) \times l_\text{old} + w \times p_\text{net}$$

where $$l_\text{old}$$ are the noisy (and one-hot) original labels, $$p_\text{net}$$ are the probabilities computed by the neural network and w is the probability of label being incorrect. w is computed by solving the assignment problem for a 2-component Mixture Model. This is based on the idea that correct (incorrect) labels will lead to small (large) losses. Therefore correct labels will belong to the low-loss component and incorrect label will belong to the high-loss component.

fit(X, y)[source]

Fit the model.

Parameters
• X – independent variable of shape $$(n, *)$$

• y – dependent variable of shape $$(n)$$

property is_classifier

Returns True. For compatibility with scikit-learn interface.

property is_regressor

Returns False. For compatibility with scikit-learn interface.

predict(X)

Run the model forward to obtain the predictions, i.e. $$y_\text{pred} = \text{model}(X)$$.

Parameters

X – independent variable of shape $$(n, *)$$

Returns

y – the predicted values of shape $$(n)$$

Return type

ndarray

predict_log_proba(X)

Compute the log_probabilities for all the classes.

Parameters

X – independent variable of shape $$(n, *)$$

Returns

log_p – Log_Probability of all the classes of shape $$(n, C)$$ where C is the number of classes.

Return type

ndarray

predict_proba(X)

Compute the probabilities for all the classes.

Parameters

X – independent variable of shape $$(n, *)$$

Returns

prob – Probability of all the classes of shape $$(n, C)$$ where C is the number of classes.

Return type

ndarray

score(X, y)

Compute the predictions, i.e. $$y_\text{pred} = \text{model}(X)$$, and score them against the true values y.

Parameters
• X – independent variable of shape $$(n, *)$$

• y – dependent variable of shape $$(n)$$

Returns

accuracy – Accuracy classification score

Return type

float

## Self Supervised Models¶

We have implemented multiple self-supervised learning (ssl) models. All these models ingest image patches. The data augmentation strategy and loss function depends on the ssl framework chosen. After training, these models can be used to compute features for new image patches. All ssl models inherit from the base class tissue_purifier.models.ssl_models.SslModelBase which is responsible for the validation (which is common to all ssl models) and logging.

class SslModelBase(val_iomin_threshold)[source]

Base class for the self-supervised learning (ssl) models (Vae, Dino, Barlow, Simclr). This base class is responsible for the validation (which is common to all ssl models) and some logging. The child classes need to implement head_and_backbone_embeddings_step(), forward() and training_step().

class BarlowModel(backbone_type, image_in_ch, head_hidden_chs, head_out_ch, lambda_off_diagonal, optimizer_type, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, val_iomin_threshold=0.0, **kwargs)[source]

Barlow self supervised learning model. Inspired by the BarlowTwin official implementation and this Barlow pytorch lightning reimplementation.

Utility functions which add parameters to argparse to simplify setting up a CLI

Example

>>> import sys
>>> import argparse
>>> args = parser.parse_args(sys.argv[1:])
classmethod get_default_params()[source]

Get the default configuration parameters for this model

Example

>>> config = BarlowModel.get_default_params()
>>> my_barlow = BarlowModel(**config)
Return type

dict

class DinoModel(image_in_ch, backbone_type, head_hidden_chs, head_use_bn, head_out_ch, center_momentum, optimizer_type, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, val_iomin_threshold=0.0, set_temperature_using_ipr_init=False, ipr_teacher_init=40.0, ipr_student_init=80.0, temperature_teacher_init=0.04, temperature_student_init=0.1, param_momentum_init=0.996, param_momentum_final=0.996, param_momentum_epochs_end=1000, **kwargs)[source]

DINO self supervised learning model. Inspired by the DINO official implementation and this DINO pytorch-lightning reimplementation

Utility functions which add parameters to argparse to simplify setting up a CLI

Example

>>> import sys
>>> import argparse
>>> args = parser.parse_args(sys.argv[1:])
classmethod get_default_params()[source]

Get the default configuration parameters for this model

Example

>>> config = DinoModel.get_default_params()
>>> my_barlow = DinoModel(**config)
Return type

dict

class SimclrModel(backbone_type, image_in_ch, head_hidden_chs, head_out_ch, optimizer_type, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, val_iomin_threshold=0.0, **kwargs)[source]

Simclr self supervised learning model. Inspired by the Simclr official implementation and this Simclr pytorch lightning reimplementation

Utility functions which add parameters to argparse to simplify setting up a CLI

Example

>>> import sys
>>> import argparse
>>> args = parser.parse_args(sys.argv[1:])
classmethod get_default_params()[source]

Get the default configuration parameters for this model

Example

>>> config = SimclrModel.get_default_params()
>>> my_barlow = SimclrModel(**config)
Return type

dict

class VaeModel(backbone_type, global_size, image_in_ch, latent_dim, encoder_hidden_dims, decoder_output_activation, optimizer_type, beta_vae_init, momentum_beta_vae, warm_up_epochs, warm_down_epochs, max_epochs, min_learning_rate, max_learning_rate, min_weight_decay, max_weight_decay, gradient_clip_val=0.0, gradient_clip_algorithm='value', val_iomin_threshold=0.0, **kwargs)[source]

Convolutional Variational Auto Encoders (VAE) with dynamically adjusted hyper-parameter $$\beta$$.

The loss function is a weighted sum of the reconstruction (MSE) and regularization (KL):

$$\text{loss} = \beta \times \text{KL} + (1-\beta) \times \text{MSE}$$

We view this problem as a Multi-Objective Optimization (minimizing MSE and KL) and we dynamically adjust $$\beta \in (.0, 1.0)$$ taking inspiration from the ideas in Multi-Task Learning as Multi-Objective Optimization

Note

Depending on the data pre-processing step, the input images might be mostly zeros with few “spots” in them. In this case, the VAE might collapse to a local minimum in the loss function corresponding to identically zero reconstruction. You might try to solve this collapse by using a larger latent dimension or by implementing a KL-free variation of the VAE, such as VQ-VAE. However, spotty images are inherently difficult for (unstructured) VAE which are required to retain (in their latent embedding) precise information about the location of each individual spots.

Utility functions which add parameters to argparse to simplify setting up a CLI

Example

>>> import sys
>>> import argparse
>>> args = parser.parse_args(sys.argv[1:])
classmethod get_default_params()[source]

Get the default configuration parameters for this model

Example

>>> config = VaeModel.get_default_params()
>>> my_barlow = VaeModel(**config)
Return type

dict

## Patch Analyzers¶

We have implemented two classes tissue_purifier.models.patch_analyzer.patch_analyzer.Composition and tissue_purifier.models.patch_analyzer.patch_analyzer.SpatialAutocorrelation which can be used to extract annotations from image patches. Together with the other models described in Self Supervised Models and Mlp Classification and Regression these allow to answer interesting questions such as: “Can the patch embedding be used to predict the cellular-composition of a patch?” or “Can the patch embedding be used to predict the Moran’s I score of a patch?”.

class Composition(return_fraction=True)[source]

Counts the number of elements in every channel and return their raw values or their normalized frequencies.

__call__(data, windows=None)[source]

Count the intensity for each channel in a 2D window.

Parameters
• data (Union[Tensor, Tensor, SparseImage, List[Tensor], List[SparseImage]]) – torch.Tensor or torch.sparse.Tensor or SparseImage (or list thereof) corresponding to a spatial data of shape $$(C, W, H)$$

• windows (Optional[Tuple[float, float, float, float]]) – tuple with (min_row, min_col, max_row, max_col). If None (default) the entire image is considered.

Returns

composition – A vector of size C with the count for each channel (or list thereof).

Return type

Tensor

Compute the Moran’s I or Geary’s C score of a sparse torch tensor. If the sparse tensor has ch channels it will produce ch scores, each indicating how each channel is dispersed in all the others.

Note

The results of a Moran’s I and Geary’s C tests depend on the choice of the weight matrix (see Moran and Geary for details). The input parameters determine the strategy used for the construction of the weight matrix.

__call__(data)
Parameters

data (Union[SparseImage, List[SparseImage], Tensor, List[Tensor]]) – A (list of) sparse tensor with C channels

Returns

score – A (list of) torch.tensor of size C with the score (either moran or gready) indicating how each channel is dispersed in all the others.