solvers

quinn

Module for base QUiNN class.

class quinn.solvers.quinn.QUiNNBase(nnmodel)

Bases: object

Base QUiNN class.

nens

Number of samples requested, M.

Type:

int

nnmodel

Underlying PyTorch NN model.

Type:

torch.nn.Module

__init__(nnmodel)

Initialization.

Parameters:

nnmodel (torch.nn.Module) – Underlying PyTorch NN model.

print_params(names_only=False)

Print model parameter names and optionally, values.

Parameters:

names_only (bool, optional) – Print names only. Default is False.

predict_sample(x)

Produce a single sample prediction.

Parameters:

x (np.ndarray) – (N,d) input array.

Raises:

NotImplementedError – Not implemented in the base class.

predict_ens(x, nens=None)

Produce an ensemble of predictions.

Parameters:

• x (np.ndarray) – (N,d) input array.

• nens (int, optional) – Number of samples requested, M.

Returns:

Array of size (M, N, o), i.e. M random samples of (N,o) outputs

Return type:

np.ndarray

predict(x)

predict_mom_sample(x, msc=0, nsam=1000)

Predict function, given input \(x\).

Parameters:

• x (np.ndarray) – A 2d array of inputs of size \((N,d)\) at which bases are evaluated.

• msc (int, optional) – Prediction mode: 0 (mean-only), 1 (mean and variance), or 2 (mean, variance and covariance). Defaults to 0.

Returns:

triple of Mean (array of size (N, o)), Variance (array of size (N, o) or None), Covariance (array of size (N, N, o) or None).

Return type:

tuple(np.ndarray, np.ndarray, np.ndarray)

predict_plot(xx_list, yy_list, nmc=100, plot_qt=False, labels=None, colors=None, iouts=None, msize=14, figname=None)

Plots the diagonal comparison figures.

Parameters:

• xx_list (list[np.ndarray]) – List of (N,d) inputs (e.g., training, validation, testing).

• yy_list (list[np.ndarray]) – List of (N,o) outputs.

• nmc (int, optional) – Requested number of samples for computing statistics, M.

• plot_qt (bool, optional) – Whether to plot quantiles or mean/st.dev.

• labels (list[str], optional) – List of labels. If None, set label internally.

• colors (list[str], optional) – List of colors. If None, sets colors internally.

• iouts (list[int], optional) – List of outputs to plot. If None, plot all.

• msize (int, optional) – Markersize. Defaults to 14.

• figname (str, optional) – Name of the figure to be saved.

Note

There is a similar function for deterministic NN in :class:..nns.nnbase.

plot_1d_fits(xx_list, yy_list, domain=None, ngr=111, plot_qt=False, nmc=100, true_model=None, labels=None, colors=None, name_postfix='')

Plotting one-dimensional slices, with the other dimensions at the nominal, of the fit.

Parameters:

• xx_list (list[np.ndarray]) – List of (N,d) inputs (e.g., training, validation, testing).

• yy_list (list[np.ndarray]) – List of (N,o) outputs.

• domain (np.ndarray, optional) – Domain of the function, (d,2) array. If None, sets it automatically based on data.

• ngr (int, optional) – Number of grid points in the 1d plot.

• plot_qt (bool, optional) – Whether to plot quantiles or mean/st.dev.

• nmc (int, optional) – Requested number of samples for computing statistics, M.

• true_model (callable, optional) – Optionally, plot a function

• labels (list[str], optional) – List of labels. If None, set label internally.

• colors (list[str], optional) – List of colors. If None, sets colors internally.

• name_postfix (str, optional) – Postfix of the filename of the saved fig.

Note

There is a similar function for deterministic NN in :class:..nns.nnbase.

nn_ens

Module for Ensemble NN wrapper.

class quinn.solvers.nn_ens.NN_Ens(nnmodel, nens=1, dfrac=1.0, verbose=False)

Bases: QUiNNBase

Deep Ensemble NN Wrapper.

dfrac

Fraction of data each learner sees.

Type:

float

learners

List of learners.

Type:

list[Learner]

nens

Number of ensemble members.

Type:

int

verbose

Verbose or not.

Type:

bool

__init__(nnmodel, nens=1, dfrac=1.0, verbose=False)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – PyTorch NN model.

• nens (int, optional) – Number of ensemble members. Defaults to 1.

• dfrac (float, optional) – Fraction of data for each learner. Defaults to 1.0.

• verbose (bool, optional) – Verbose or not.

print_params(names_only=False)

Print model parameter names and optionally, values.

Parameters:

names_only (bool, optional) – Print names only. Default is False.

fit(xtrn, ytrn, **kwargs)

Fitting function for each ensemble member.

Parameters:

• xtrn (np.ndarray) – Input array of size (N,d).

• ytrn (np.ndarray) – Output array of size (N,o).

• **kwargs (dict) – Any keyword argument that nns.nnfit.nnfit() takes.

predict_sample(x)

Predict a single, randomly selected sample.

Parameters:

x (np.ndarray) – Input array of size (N,d).

Returns:

Output array of size (N,o).

Return type:

np.ndarray

predict_ens(x, nens=None)

Predict from all ensemble members.

Parameters:

x (np.ndarray) – (N,d) input array.

Returns:

List of M arrays of size (N, o), i.e. M random samples of (N,o) outputs.

Return type:

list[np.ndarray]

Note

This overloads QUiNN’s base predict_ens function.

predict_ens_fromsamples(x, nens=1)

Predict ensemble in a loop using individual predict_sample() calls.

Parameters:

• x (np.ndarray) – (N,d) input array.

• nens (int, optional) – Number of samples requested.

Returns:

List of M arrays of size (N, o), i.e. M random samples of (N,o) outputs.

Return type:

list[np.ndarray]

nn_mcmc

Module for MCMC NN wrapper.

class quinn.solvers.nn_mcmc.NN_MCMC(nnmodel, verbose=True)

Bases: QUiNNBase

MCMC NN wrapper class.

cmode

MAP values of all parameters, size M.

Type:

np.ndarray

lpinfo

Dictionary that holds likelihood computation necessary information.

Type:

dict

pdim

Dimensonality d of chain.

Type:

int

samples

MCMC samples of all parameters, size (M,d).

Type:

np.ndarray

verbose

Whether to be verbose or not.

Type:

bool

__init__(nnmodel, verbose=True)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – PyTorch NN model.

• verbose (bool, optional) – Verbose or not.

logpost(modelpars, lpinfo)

Function that computes log-posterior given model parameters.

Parameters:

• modelpars (np.ndarray) – Log-posterior input parameters.

• lpinfo (dict) – Dictionary of arguments needed for likelihood computation.

Returns:

log-posterior value.

Return type:

float

logpostgrad(modelpars, lpinfo)

Function that computes log-posterior given model parameters.

Parameters:

• modelpars (np.ndarray) – Log-posterior input parameters.

• lpinfo (dict) – Dictionary of arguments needed for likelihood computation.

Returns:

log-posterior gradient array.

Return type:

np.ndarray

fit(xtrn, ytrn, zflag=True, datanoise=0.05, nmcmc=6000, param_ini=None, sampler='amcmc', sampler_params=None)

Fit function that perfoms MCMC on NN parameters.

Parameters:

• xtrn (np.ndarray) – Input data array x of size (N,d).

• ytrn (np.ndarray) – Output data array y of size (N,o).

• zflag (bool, optional) – Whether to precede MCMC with a LBFGS optimization. Default is True.

• datanoise (float, optional) – Datanoise size. Defaults to 0.05.

• nmcmc (int, optional) – Number of MCMC steps, M.

• param_ini (None, optional) – Initial parameter array of size p. Default samples randomly.

• sampler (str, optional) – Sampler method (‘amcmc’, ‘hmc’, ‘mala’). Defaults to ‘amcmc’.

• sampler_params (dict, optional) – Sampler parameter dictionary.

get_best_model(param)

Creates a PyTorch NN module with parameters set with a given flattened parameter array.

Parameters:

param (np.ndarray) – A flattened weight parameter vector.

Returns:

PyTorch NN module with the given parameters.

Return type:

torch.nn.Module

predict_MAP(x)

Predict with the max a posteriori (MAP) parameter setting.

Parameters:

x (np.ndarray) – Input array of size (N,d).

Returns:

Outpur array of size (N,o).

Return type:

np.ndarray

predict_sample(x, param)

Predict with a given parameter array.

Parameters:

• x (np.ndarray) – Input array of size (N,d).

• param (np.ndarray) – Flattened weight parameter array.

Returns:

Outpur array of size (N,o).

Return type:

np.ndarray

predict_ens(x, nens=10, nburn=1000)

Predict an ensemble of results.

Parameters:

• x (np.ndarray) – (N,d) input array.

• nens (int, optional) – Number of ensemble members requested, M. Defaults to 10.

• nburn (int, optional) – Burn-in for the MCMC chain. Defaults to 1000.

Returns:

Array of size (M, N, o), i.e. M random samples of (N,o) outputs.

Return type:

np.ndarray

Note

This overloads QUiNN’s base predict_ens functions

nn_vi

Module for the Variational Inference (VI) NN wrapper.

class quinn.solvers.nn_vi.NN_VI(nnmodel, verbose=False, pi=0.5, sigma1=1.0, sigma2=1.0, mu_init_lower=-0.2, mu_init_upper=0.2, rho_init_lower=-5.0, rho_init_upper=-4.0)

Bases: QUiNNBase

VI wrapper class. This implements the Bayes-by-backprop method. For details of the method, see Blundell et al. [1].

best_model

The best PyTorch NN model found during training.

Type:

torch.nn.Module

bmodel

The underlying Bayesian model.

Type:

BNet

device

Device on which the computations are done.

Type:

str

trained

Whether the model is trained or not.

Type:

bool

verbose

Whether to be verbose or not.

Type:

bool

__init__(nnmodel, verbose=False, pi=0.5, sigma1=1.0, sigma2=1.0, mu_init_lower=-0.2, mu_init_upper=0.2, rho_init_lower=-5.0, rho_init_upper=-4.0)

Instantiate a VI wrapper object.

Parameters:

• nnmodel (torch.nn.Module) – The underlying PyTorch NN model.

• verbose (bool, optional) – Whether to print out model details or not.

• pi (float) – Weight of the first gaussian. The second weight is 1-pi.

• sigma1 (float) – Standard deviation of the first gaussian. Can also be a scalar torch.Tensor.

• sigma2 (float) – Standard deviation of the second gaussian. Can also be a scalar torch.Tensor.

• mu_init_lower (float) – Initialization of mu lower value

• mu_init_upper (float) – Initialization of mu upper value

• rho_init_lower (float) – Initialization of rho lower value

• rho_init_upper (float) – Initialization of rho upper value

fit(xtrn, ytrn, val=None, nepochs=600, lrate=0.01, batch_size=None, freq_out=100, freq_plot=1000, wd=0, cooldown=100, factor=0.95, nsam=1, scheduler_lr=None, datanoise=0.05)

Fit function to train the network.

Parameters:

• xtrn (np.ndarray) – Training input array of size (N,d).

• ytrn (np.ndarray) – Training output array of size (N,o).

• val (tuple, optional) – x,y tuple of validation points. Default uses the training set for validation.

• nepochs (int, optional) – Number of epochs.

• lrate (float, optional) – Learning rate or learning rate schedule factor. Default is 0.01.

• batch_size (int, optional) – Batch size. Default is None, i.e. single batch.

• freq_out (int, optional) – Frequency, in epochs, of screen output. Defaults to 100.

• freq_plot (int, optional) – Frequency, in epoch, of plotting the loss.

• wd (float, optional) – Optional weight decay (L2 regularization) parameter.

• cooldown (int, optional) – cooldown in ReduceLROnPlateau

• factor (float, optional) – factor in ReduceLROnPlateau

• nsam (int, optional) – Number of samples for ELBO computation. Defaults to 1.

• scheduler_lr (None, optional) – Scheduler of learning rate. See the corresponding argument in nns.nnfit.nnfit().

• datanoise (float, optional) – Datanoise for ELBO computation. Defaults to 0.05.

predict_sample(x)

Predict a single sample.

Parameters:

x (np.ndarray) – Input array x of size (N,d).

Returns:

Output array x of size (N,o).

Return type:

np.ndarray

Note

predict_ens() from the parent class will use this to sample an ensemble.

nn_laplace

Module for Laplace NN wrapper.

class quinn.solvers.nn_laplace.NN_Laplace(nnmodel, la_type='full', cov_scale=1.0, datanoise=0.1, priorsigma=1.0, **kwargs)

Bases: NN_Ens

Wrapper class for the Laplace method.

cov_mats

List of covariance matrices.

Type:

list

cov_scale

Covariance scaling factor for prediction.

Type:

TYPE

datanoise

Data noise standard deviation.

Type:

float

la_type

Laplace approximation type (‘full’ or ‘diag’).

Type:

str

means

List of MAP centers.

Type:

list

nparams

Number of parameters in the model.

Type:

int

priorsigma

Gaussian prior standard deviation.

Type:

float

__init__(nnmodel, la_type='full', cov_scale=1.0, datanoise=0.1, priorsigma=1.0, **kwargs)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – NNWrapper class.

• la_type (str, optional) – Laplace approximation type (‘full’ or ‘diag’). Dedaults to ‘full’.

• cov_scale (float, optional) – Covariance scaling factor for prediction. Defaults to 1.0.

• datanoise (float, optional) – Data noise standard deviation. Defaults to 0.1.

• priorsigma (float, optional) – Gaussian prior standard deviation. Defaults to 1.0.

• **kwargs – Any keyword argument that nns.nnfit.nnfit() takes.

fit(xtrn, ytrn, **kwargs)

Fitting function for each ensemble member. :param xtrn: Input array of size (N,d). :type xtrn: np.ndarray :param ytrn: Output array of size (N,o). :type ytrn: np.ndarray :param **kwargs: Any keyword argument that nns.nnfit.nnfit() takes. :type **kwargs: dict

la_calc(learner, xtrn, ytrn, batch_size=None)

Given alearner, this method stores in the corresponding lists the vectors and matrices defining the posterior according to the laplace approximation. :param learner: Instance of the Learner class including the model :type learner: Learner :param torch.nn.Module being used.: :param xtrn: input part of the training data. :type xtrn: np.ndarray :param ytrn: target part of the training data. :type ytrn: np.ndarray :param batch_size: batch size used in the hessian estimation. :type batch_size: int :param Defaults to None: :param i.e. single batch.:

predict_sample(x)

Predict a single sample.

Parameters:

x (np.ndarray) – Input array x of size (N,d).

Returns:

Output array x of size (N,o).

Return type:

np.ndarray

predict_ens(x, nens=1)

Predict an ensemble of results.

Parameters:

x (np.ndarray) – (N,d) input array.

Returns:

List of M arrays of size (N, o), i.e. M random samples of (N,o) outputs.

Return type:

list[np.ndarray]

Note

This overloads NN_Ens’s and QUiNN’s base predict_ens function.

nn_swag

Module for SWAG NN wrapper.

class quinn.solvers.nn_swag.NN_SWAG(nnmodel, k=10, n_steps=12, c=1, cov_type='lowrank', lr_swag=0.1, datanoise=0.1, priorsigma=1.0, **kwargs)

Bases: NN_Ens

SWAG NN Wrapper class.

c

Frequency of the moment update.

Type:

int

cov_diags

List of diagonal covariances.

Type:

list

cov_type

Covariance type.

Type:

str

d_mats

List of D-matrices.

Type:

list

datanoise

Data noise standard deviation.

Type:

float

k

k-parameter of the method

Type:

int

lr_swag

Learning rate.

Type:

float

means

List of mean values of the history.

Type:

list

n_steps

Number of steps in SWAG algorithm.

Type:

int

nparams

Number of underlying NN module parameters.

Type:

int

priorsigma

Standard deviation of the prior.

Type:

float

__init__(nnmodel, k=10, n_steps=12, c=1, cov_type='lowrank', lr_swag=0.1, datanoise=0.1, priorsigma=1.0, **kwargs)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – NNWrapper class.

• k (int, optional) – k-parameter of the method. Defaults to 10.

• n_steps (int, optional) – Number of steps. Defaults to 12.

• c (int, optional) – Frequency of moment update. Defaults to 1.

• cov_type (str, optional) – Covariance type. Defaults to ‘lowrank’, anything else ignores low-rank approximation.

• lr_swag (float, optional) – Learning rate. Defaults to 0.1.

• datanoise (float, optional) – Data noise standard deviation. Defaults to 0.1.

• priorsigma (float, optional) – Standard deviation of the prior. Defaults to 1.0.

• **kwargs – Any other keyword argument that nns.nnfit.nnfit() takes.

fit(xtrn, ytrn, **kwargs)

Fitting function for each ensemble member.

Parameters:

• xtrn (np.ndarray) – Input array of size (N,d).

• ytrn (np.ndarray) – Output array of size (N,o).

• **kwargs (dict) – Any keyword argument that nns.nnfit.nnfit() takes.

swag_calc(learner, xtrn, ytrn)

Given a learner, this method stores in the corresponding lists the vectors and matrices defining the posterior according to the laplace approximation.

Parameters:

• learner (Learner) – Instance of the Learner class including the model

• used. (torch.nn.Module being)

• xtrn (np.ndarray) – input part of the training data.

• ytrn (np.ndarray) – target part of the training data.

predict_sample(x)

Predict a single sample.

Parameters:

x (np.ndarray) – Input array x of size (N,d).

Returns:

Output array x of size (N,o).

Return type:

np.ndarray

predict_ens(x, nens=1)

Predict an ensemble of results.

Parameters:

x (np.ndarray) – (N,d) input array.

Returns:

List of M arrays of size (N, o), i.e. M random samples of (N,o) outputs.

Return type:

list[np.ndarray]

Note

This overloads NN_Ens’s and QUiNN’s base predict_ens function.

nn_rms

Module for RMS NN wrapper.

class quinn.solvers.nn_rms.NN_RMS(nnmodel, datanoise=0.1, priorsigma=1.0, **kwargs)

Bases: NN_Ens

RMS Ensemble NN Wrapper. For details of the method, see Pearce et al. [2].

datanoise

Data noise standard deviation.

Type:

float

nparams

Number of model parameters.

Type:

int

priorsigma

Prior standard deviation.

Type:

float

__init__(nnmodel, datanoise=0.1, priorsigma=1.0, **kwargs)

Initialization.

Parameters:

• nnmodel (torch.nn.Module) – NNWrapper class.

• datanoise (float, optional) – Data noise standard deviation. Defaults to 0.1.

• priorsigma (float, optional) – Gaussian prior standard deviation. Defaults to 1.0.

• **kwargs – Any keyword argument that nns.nnfit.nnfit() takes.

fit(xtrn, ytrn, **kwargs)

Fitting function for each ensemble member.

Parameters:

• xtrn (np.ndarray) – Input array of size (N,d).

• ytrn (np.ndarray) – Output array of size (N,o).

• **kwargs (dict) – Any keyword argument that nns.nnfit.nnfit() takes.